Hey there! Let's say I have a team that wants to trigger 100k instances of a workflow to run, but we only want 1000 of them to run at the same time. 1. Would Flyte "enqueue" all 100k of them and enable us to set some sort of max concurrency parameter? The goal is for them to "fire and forget" so that they don't have to wait for trigger such a large number of jobs 2. If all of those workflows are extremely similar, is it better to run those as MapTasks?

Maybe max_parallelism would work for this?

You can set different max_parallelism per workflow/execution, so I don’t think that’s it From the page you linked:

The worst case for FlytePropeller is workflows that have an extremely large fan-out. This is because FlytePropeller implements a greedy traversal algorithm, that tries to evaluate the entire unblocked nodes within a workflow in every round. A solution for this is to limit the maximum number of nodes that can be evaluated. This can be done by setting max-parallelism for an execution. This can done in multiple ways

It seems to be a setting for limiting the amount of nodes that can run concurrently within an execution

You can launch those 100k as subworkflows, in which case this

max_parallelism

would apply

That’s probably a bad idea for the reason I quoted above

The worst case for FlytePropeller is workflows that have an extremely large fan-out.

I’ve also had issues with Flyte console being unable to display large execution graphs sometimes (https://github.com/flyteorg/flyte/issues/3803)

About rendering; I don’t think that having 100k perfectly rendered workflows would be easier to follow; with such amount of tasks, I would not rely heavily on the console. About the performance; it’s true, it will take longer to update the workflows. Let’s wait and see if some else has a better idea then or this would be the way to go

About the queue handling 100,000 executions, I see there’s an option to configure queue parameters in the flyte propeller configmap (or from the helm values.yaml):

queue:
        batch-size: -1
        batching-interval: 2s
        queue:
          base-delay: 5s
          capacity: 1000
          max-delay: 120s
          rate: 100
          type: maxof
        sub-queue:
          capacity: 1000
          rate: 100
          type: bucket

So you should be able to configure a higher capacity for the queue to keep your 100,000 executions in the queue But this does not limit the amount of executions that can run at the same time

Thanks a lot for your suggestions!

But this would impact other workflows, right? So if other workflow is launched it will count for that limit and might not scale to 100k. Or am I wrong? And how would the autoscaling at a node-* level work, given that depending on resources usage one node might have n pods ?

Because these 100,000 tasks executions are all similar, you can check how many nodes run concurrently in a single execution and set the autoscaling limit to 1000 * that amount. Other tasks will still compete for the nodes quota, I think it is a good idea to create a separate Flyte cluster for these 100,000. It’s still a workaround, and it would be great if we had a feature to limit the executions per project/workflow natively from Flyte, that would also count non-kubernetes tasks By the way, this amount of executions comes close to the recommended limits for a single kubernetes cluster: https://kubernetes.io/docs/setup/best-practices/cluster-large/

• No more than 110 pods per node

• No more than 5,000 nodes

• No more than 150,000 total pods

• No more than 300,000 total containers

So this should probably run on a multi-cluster setup: https://docs.flyte.org/en/latest/deployment/deployment/multicluster.html

Thanks, yes we're setting this up as multi-cluster 🙂 Do you think the new ArrayNode map tasks could be a good fit as well? Maybe they would fit within the `max_parallelism`limit (as opposed to regular map tasks)? https://flyte.org/blog/flyte-1-9-arraynode-execution-tags-new-navigation-and-more I was wondering if making a workflow that triggers 1000 map tasks of 1000 tasks each could help, with a max parallelism of 1 (since it seems like map task nodes count for 1 unit of parallelism)

I would write a parent workflow that can launch your launchplans