we have a large job coming up and want to stress test

Question

< Yee> we have a large job coming up and want to stress test what our Flyte deployment Flyte scheduler can manage we were considering doing a stress test with an increasing amount of very simple tasks Starting somewhere around 10k pods and increasing on subsequent runs to roughly 200k pods All will be spun up in a short amount of time and we want to see if it kills our deployment or what the performance impacts are Any ideas you have on better being able to stress test or things to look out for that could help us better manage this massive amount of parallelism

Nicholas Roberson · Answer

Initial thoughts are even at 10k to create them in batches to not DDoS the flyte API But past that once they all exist we want to see how the Scheduler manages under this load

Nicholas Roberson · Answer

We also want to see what the difference may be when just creating them in a for loop vs using `map task`

Rahul Mehta · Answer

Very interested in the for loops ie in `dynamic` vs map task We have some high fan out `dynamic` workflows right now that will spin up ~1k pods but nothing close to that scale

Yee · Answer

cc < Dan Rammer hamersaw > and < Eduardo Apolinario eapolinario >

Yee · Answer

we ve definitely run stress tests in the past i don t quite remember the numbers though

Yee · Answer

< Ketan kumare3 > may have a better idea

Ketan (kumare3) · Answer

< Nicholas Roberson> so map tasks are more optimal as compared to dynamic or other types of pods

Ketan (kumare3) · Answer

also propeller will try to protect itself and kubeapi aggressively so you may not get very high through put as there is max parallelism you should tweak it

Ketan (kumare3) · Answer

there are ways of increasing performance spec offloading increasing kube client limits and running sharded propeller

Nicholas Roberson · Answer

< Ketan kumare3 > Interesting and yeah we suspected that the `map task` would be the ideal performance we can limit the concurrency there to protect us a bit no need to spin up that many at once but that many over 1hr is still reasonable

Nicholas Roberson · Answer

gt there are ways of increasing performance spec offloading increasing kube client limits and running sharded propeller edited Id be interested to know more about this if you have any docs or examples you can point me to

Dan Rammer (hamersaw) · Answer

< Nicholas Roberson> very interested in discussing the results of this There are a number of knobs in flyte mostly between propeller and admin than can be tuned for higher throughput As suggested in executions of this scale we typically see the kube apiserver as the bottleneck

Dan Rammer (hamersaw) · Answer

also per <https kubernetes io docs setup best practices cluster large |k8s docs> `No more than 150 000 total pods` might be something to note

Nicholas Roberson · Answer

< Dan Rammer hamersaw > Good to know so in reality we would hit the limits of the Kubernetes API first before we max out Flyte

Nicholas Roberson · Answer

We will update to max at 100k then probably but I assume we will bring something down or Flyte slows us down to protect itself

Dan Rammer (hamersaw) · Answer

docs for <https docs flyte org en latest deployment configuration performance html offloading static workflow information from crd|offloading static workflow information to CRD>

Nicholas Roberson · Answer

In reality also we can have N pods process M jobs each not each has to be its own job so I would imagine we would probably not set M = 1 if what I am reading from you all is true

Dan Rammer (hamersaw) · Answer

gt we would hit the limits of the Kubernetes API first before we max out Flyte That s what we expect If your seeing anything different it would be great to dive into

Nicholas Roberson · Answer

Solid ill add this to the stress test doc I have internally and share with infra team as well as other stakeholders

Dan Rammer (hamersaw) · Answer

and then here s the <https github com flyteorg flyteadmin pull 491|PR for adding kubeclient config to admin> in admin this just covers creating new FlyteWorkflow CRs so if you re using maptasks this shouldn t be an issue however similar configuration exists in flytepropeller don t have the docs on hand which will affect creating of Pods etc at larger scales this will probably require some tweaking

Ketan (kumare3) · Answer

I think we should just have this in the docs

Nicholas Roberson · Answer

Ok thanks all I appreciate the feedback Typically what would be the best metrics to look at during running this We have DataDog + kube metrics scraped by Prometheus and ETLed to a reporting DB

Nicholas Roberson · Answer

I mean 1 is always did we crash anything but past that

Nicholas Roberson · Answer

Its likely we are not going to try to run all 100k at the same time but will sumbit them in a short time we will increase the parallelism to a certain point and then will watch how the remaining 90k lets say stress the system Essentially these jobs are going to come in really fast in real life however they don t need to all be processed immediately and can sit waiting for multiple days if we cant get to them We are largely trying to see how things behave under this scenario High throughput large queue for the scheduler propellor to manage

Nicholas Roberson · Answer

Can someone help me interpret this error ``` 1 1 currentAttempt done Last Error USER 1 1 currentAttempt done Last Error USER containers with unready status nicholas 0vr23dznsximza1mggnq1a n0 0 dn313 0 |Back off pulling image <http 772228263286 dkr ecr us west 2 amazonaws com flyte run parallel workflow b48da3d087bede34e4097373a549e5ffdad4f4e988156d1f4b667246c1863f5a|772228263286 dkr ecr us west 2 amazonaws com flyte run parallel workflow b48da3d087bede34e4097373a549e5ffdad4f4e988156d1f4b667246c1863f5a> ``` I ran a dynamic workflow with 1000 tasks and `max parallelism` set to 100 Very bare bones

Yee · Answer

that error message is probably correct you re probably actually hitting the rate limit on ecr assuming the image exists

Nicholas Roberson · Answer

Hm yeah seems like it for individual repositories in looks like the image pull is rate limited on top of ECR limits for all images is there any way to cache the image in a workflow so we dont have to consistently pull it past the first time

Nicholas Roberson · Answer

We may be hitting this limit <https repost aws knowledge center ecs pull container error rate limit>

Nicholas Roberson · Answer

The overall limit on image pulls is really high spinning up 1k pods should not hit it Using `map task` to handle this helps however if we have 10 workflows with 50 concurrency lets say we will still blow past the image pull rate if the post I sent is accurate

Nicholas Roberson · Answer

I was able to reproduce the error in Flyte ```containers with unready status anjmtfxbkkzndb4sm9kj n0 0 dn810 0 |Back off pulling image ``` and ran this a bunch of times calling `docker rmi ` to remove inbetween ```docker pull ``` which successfully downloaded the image to my local I wonder if this is the limits on the propellor we are hitting and the pods cant spin up gt increasing our `burst` setting may help here

Rahul Mehta · Answer

gt Essentially these jobs are going to come in really fast in real life however they don t need to all be processed immediately and can sit waiting for multiple days if we cant get to them We are largely trying to see how things behave under this scenario High throughput large queue for the scheduler propellor to manage Have you thought about using SQS or Kafka to manage the queuing and then have a scheduled workflow that pops batches of messages off the queue and launches them Seems like relying on the workflow controller ie propeller as your queue may not be precisely what you want here you ll have a greater degree of control over the concurrency of the tasks

Ketan (kumare3) · Answer

This is not propeller it s getting throttles pulling images

Ketan (kumare3) · Answer

But need to understand more ideally you should not hit this as it will reuse a downloaded image

Nicholas Roberson · Answer

Yeah the current setup we have now is doing something similar to that in the name of simplification of some of our tools we wanted to try to move to a Flyte only implementation That being said we could use a new redis queue to do this

Nicholas Roberson · Answer

1 On job submit push 10k messages to queue + spin up workflow to process that queue 2 Workflow loops with a delay and creates async tasks to process messages on the queue 3 Once all messages are processed and queue is empty workflow ends

Rahul Mehta · Answer

Is there a reason you prefer a single workflow execution to handle the entire set of messages Rather than a workflow itself handling the loop could a cron schedule handle that

Rahul Mehta · Answer

Ie workflow on a cron schedule pops k messages off the queue and dispatches tasks to process those and ends when processing that batch is finished

Nicholas Roberson · Answer

We are pushing for better traceability at the moment so tracking logs cost etc for large jobs if we can do it in one workflow that would be ideal however if there are too many complications as a result we can move to a hybrid solution

Nicholas Roberson · Answer

Having all work under a single umbrella workflow makes things easy to track for devs as well as users wondering about the status of their job since they can just go to one place in the console to track

Rahul Mehta · Answer

I see in general we ve found that smaller units of work make reasoning about retries etc easier But understood on tracking workflow execution state

Nicholas Roberson · Answer

Yeah I agree with you 100 were trying to balance that with this implementation

Nicholas Roberson · Answer

Let me do some noodling but at the moment the only barrier seems to be some rate limit we are hitting in ECR and if we want to have thousands of pods pulling the same image we will need to figure out how to have our setup insulate from hitting that as best as we can

Rahul Mehta · Answer

That sounds like you need to tune image pull policies amp ensure that tasks pack onto the same nodes so you can re use the image depending on your cluster autoscaler ie karpenter autopilot etc this is fairly simple

Rahul Mehta · Answer

We hit something similar when our autoscaler was launching a larger number of small nodes by configuring it to prefer larger instances we could pack more tasks onto a single node and avoid the unnecessary pulls

Ketan (kumare3) · Answer

So < Nicholas Roberson> < Rahul Mehta> one of the completely undocumented feature in Flyte is inbuilt resource control system that uses redis

Ketan (kumare3) · Answer

Also you are right Rahul at lyft we have massive machines with ssds and at scale we did not even spin many of them down

Ketan (kumare3) · Answer

Also happy to jump on a call sometime to talk more about things

Ketan (kumare3) · Answer

< Nicholas Roberson> not a sales thing but union cloud has a different architecture it has queues built in that can store workflows and distribute them

Nicholas Roberson · Answer

That would be really helpful sometime during the week let me get back and see if infra would want to join and we can chat

Ketan (kumare3) · Answer

Also great to hear that you are centralizing on Flyte

Nicholas Roberson · Answer

100 We looked at a few options however considering that we have a team of scientists who code but aren t full developers this was the best tool to enable them to write remote workflows with minimal load on the engineering team Help them move faster while not requiring us to do a ton of work for production ize their code

Ketan (kumare3) · Answer

We would love to call it a platform haha

Nicholas Roberson · Answer

It is a full fledged platform hahaha don t take my wording to heart

Ketan (kumare3) · Answer

I am joking gotta have a funny bone when in the community

Nicholas Roberson · Answer

Amen to that