Is there a way to configure Flyte to use different...
# ask-the-communityg
Gopal Vashishtha
10/21/2023, 2:18 AMIs there a way to configure Flyte to use different sets of resources for different kinds of workloads? We have a situation where user A submits long-running tasks and user B's short tasks don't get scheduled until user B's tasks finish. I see executionQueues mentioned in the docs but am not really understanding how to use them
y
Yee
10/22/2023, 11:46 PM
execution queues are for working with things like aws batch. i don’t think they’ll help in this scenario.
Yee
10/22/2023, 11:47 PM
you can split them into multiple projects.
Yee
10/22/2023, 11:47 PM
and then assign different namespace level k8s resourcequotas for those namespaces.
Yee
10/22/2023, 11:48 PM
but that will permanently cap user A.
Yee
10/22/2023, 11:48 PM
you don’t want to forcibly evict user a, that’s the worst outcome.
Yee
10/22/2023, 11:49 PM
as user As tasks finish however you should be seeing user B’s tasks kick off. if you’re seeing that the user A workflow needs to run to completion before any of user b’s tasks start then something feels off.
g
Gopal Vashishtha
10/23/2023, 1:06 AMUser A’s task is like a 35K map task. Could that explain this? I think different projects might be the solution for us
y
Yee
10/23/2023, 1:16 AM
i think you should try a different project.
Yee
10/23/2023, 1:16 AM
easy enough to try, just re-register
Yee
10/23/2023, 1:17 AM
map tasks shouldn’t inherently introduce issues in multi-tenancy beyond just the sheer number
2 Views