https://flyte.org logo
#contribute
Title
# contribute
e

Eli Bixby

11/30/2023, 12:02 PM
Is anybody looking into integration points that might exist between https://github.com/kubernetes-sigs/kueue and Flyte? GKE at least seems to have plan to integrate with this directly for job queuing (we're in contact with them about capacity issues)
Specifically, does anyone know what Propeller's reaction to a pod having
suspend: True
specified is? as long as it doesn't cause killing of the workflow, I think they should be able to work fine together
d

Dan Rammer (hamersaw)

11/30/2023, 3:29 PM
I'm not aware of any potential integrations. We have broadly explored similar queueing solutions in the past though, without any real strong adoption effort. Are you envisioning that propeller, rather than creating a k8s pod to execute a task, it writes the task to kueue and waits for it to be picked up? We might need a specific backend plugin for this to correctly identify task phases for jobs in different states of kueue execution. With a maybe naive view of this, it sounds like it wouldn't be terribly difficult.
e

Eli Bixby

11/30/2023, 3:32 PM
We're going to try just a user-side integration (essentially we use pod-template for relevant pods since kueue just requires an annotation to know that it should be managing a pod). The only concern there is if propeller does not know how to handle a pod with
suspend: True
. Longer term I could see having job queues be a little bit more natively integrated as a way of providing priority/resource management, but don't have a clear sense of what that would look like.
d

Dan Rammer (hamersaw)

11/30/2023, 3:36 PM
Oh interesting, so basically you just annotate the pod and that tells kueue that it should be managing it. I assumed (again naievely based on other jobqueue implementations) that it was more complex - ie. write a
job
to kueue, wait for it to create a pod, etc. Propeller bases task status off pod / container status. So if kueue just leaves the Pod in
pending
until it decides to execute there should be no issue. Very interested to see how this goes and happy to help if you run into issues!
e

Eli Bixby

11/30/2023, 3:42 PM
Yeah, AFAICT Kueue just watches pods with the annotation, and puts them in a queue, then when they receive resources, it just sets
suspend: False
Ahh unfortunately it appears that only Job resources have the
suspend
field... So Flyte would need to change tasks to run as jobs and not pods.... no idea how involved that would be. Correction: Looks like it will work just fine, just leave the pod in pending, provided you enable managing bare pods: https://kueue.sigs.k8s.io/docs/tasks/run_plain_pods/. We'll do a little write-up if we get it working
d

Dan Rammer (hamersaw)

12/01/2023, 3:34 PM
Sounds good! Looking forward to hearing back on this.
k

Ketan (kumare3)

12/02/2023, 1:11 PM
Ya jobs are terrible - prefer pods please
d

David Espejo (he/him)

12/06/2023, 3:35 PM
We'll do a little write-up if we get it working
hey @Eli Bixby, let us know, we can collaborate on this and post it on Flyte's blog too