Is there a way to configure Flyte to use different...
# flyte-supportc
calm-zoo-68637
10/21/2023, 2:18 AMIs there a way to configure Flyte to use different sets of resources for different kinds of workloads? We have a situation where user A submits long-running tasks and user B's short tasks don't get scheduled until user B's tasks finish. I see executionQueues mentioned in the docs but am not really understanding how to use them
t
thankful-minister-83577
10/22/2023, 11:46 PM
execution queues are for working with things like aws batch. i don’t think they’ll help in this scenario.
thankful-minister-83577
10/22/2023, 11:47 PM
you can split them into multiple projects.
thankful-minister-83577
10/22/2023, 11:47 PM
and then assign different namespace level k8s resourcequotas for those namespaces.
thankful-minister-83577
10/22/2023, 11:48 PM
but that will permanently cap user A.
thankful-minister-83577
10/22/2023, 11:48 PM
you don’t want to forcibly evict user a, that’s the worst outcome.
thankful-minister-83577
10/22/2023, 11:49 PM
as user As tasks finish however you should be seeing user B’s tasks kick off. if you’re seeing that the user A workflow needs to run to completion before any of user b’s tasks start then something feels off.
c
calm-zoo-68637
10/23/2023, 1:06 AMUser A’s task is like a 35K map task. Could that explain this? I think different projects might be the solution for us
t
thankful-minister-83577
10/23/2023, 1:16 AM
i think you should try a different project.
thankful-minister-83577
10/23/2023, 1:16 AM
easy enough to try, just re-register
thankful-minister-83577
10/23/2023, 1:17 AM
map tasks shouldn’t inherently introduce issues in multi-tenancy beyond just the sheer number
6 Views