Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

hey a while back there was an RFC on ray integration that included something about support for persisting cluster resources across tasks, is that something still in progress? can someone point me to the docs?

so we spoke with <@U03F38WVDHN> about the persistent cluster feature.

to clarify, you mean that if one workflow runs two ray tasks, they both end up using the same cluster right?

to save boot time yes.  serialization still happens iiuc

<https://github.com/flyteorg/flyteplugins/tree/master/go/tasks/plugins/k8s/ray> is all the code right kevin?

do you have any more info on the persistent clusters? we could potentially use it to speed up our end to end workflows by quite a bit :slightly_smiling_face:

we don’t support it right now. we can support that, need to update backend plugin. I’ll work on it next week, and get back to you once it’s done.

haha that's great! but i'm mostly looking to understand the mechanics and i remember there being an RFC discussing them

we don't have an urgent timeline and are looking to plan some work

<@USU6W5ATA> let’s wait on this. <@U01E1L78CLA> what we understood from <@U03F38WVDHN> was that there is potential corruption that happens when a cluster is reused in ray

or just needs to be thought through more?

it can be done on flyte side, we dont know if ray is ready yet

actually wait, "it can be done on flyte side" does this mean the infra for reusing resources exists?

There is a `ClusterSelector` In ray job <https://github.com/ray-project/kuberay/blob/master/ray-operator/apis/ray/v1alpha1/rayjob_types.go#L53|CRD>, so basically we should be able to use it to run the ray job on the existing cluster. The propeller need to save the rayCluster id generated by first ray task, and the second ray task should reuse the same ray cluster by passing the cluster selector. lastly, propeller shut down the ray cluster at the end node.

<@UNZB4NW3S> We will need this feature as well.  With more complex Flyte workflows, users should be able to share Ray cluster among different Flyte tasks.

I know you will, remember we were adding this but you had issues