hey a while back there was an RFC on ray integration that in

Question

hey a while back there was an RFC on ray integration that included something about support for persisting cluster resources across tasks is that something still in progress can someone point me to the docs

Yee · Answer

< Kevin Su>

Yee · Answer

so we spoke with < Keshi Dai> about the persistent cluster feature

Yee · Answer

to clarify you mean that if one workflow runs two ray tasks they both end up using the same cluster right

Dylan Wilder · Answer

yea to save the boot time right

Dylan Wilder · Answer

and possibly skip serialization

Yee · Answer

to save boot time yes serialization still happens iiuc

Yee · Answer

<https github com flyteorg flyteplugins tree master go tasks plugins k8s ray> is all the code right kevin

Kevin Su · Answer

yup

Dylan Wilder · Answer

do you have any more info on the persistent clusters we could potentially use it to speed up our end to end workflows by quite a bit slightly smiling face

Kevin Su · Answer

we don t support it right now we can support that need to update backend plugin I ll work on it next week and get back to you once it s done

Dylan Wilder · Answer

haha that s great but i m mostly looking to understand the mechanics and i remember there being an RFC discussing them

Dylan Wilder · Answer

we don t have an urgent timeline and are looking to plan some work

Ketan (kumare3) · Answer

< Kevin Su> let s wait on this < Dylan Wilder> what we understood from < Keshi Dai> was that there is potential corruption that happens when a cluster is reused in ray

Dylan Wilder · Answer

does that mean it s off the roadmap

Dylan Wilder · Answer

or just needs to be thought through more

Ketan (kumare3) · Answer

its not off the roadmap

Ketan (kumare3) · Answer

it can be done on flyte side we dont know if ray is ready yet

Ketan (kumare3) · Answer

but de prioritized

Dylan Wilder · Answer

got it thanks for the context pray

Dylan Wilder · Answer

actually wait it can be done on flyte side does this mean the infra for reusing resources exists

Kevin Su · Answer

There is a `ClusterSelector` In ray job <https github com ray project kuberay blob master ray operator apis ray v1alpha1 rayjob types go L53|CRD> so basically we should be able to use it to run the ray job on the existing cluster The propeller need to save the rayCluster id generated by first ray task and the second ray task should reuse the same ray cluster by passing the cluster selector lastly propeller shut down the ray cluster at the end node

Keshi Dai · Answer

< Ketan kumare3 > We will need this feature as well With more complex Flyte workflows users should be able to share Ray cluster among different Flyte tasks

Ketan (kumare3) · Answer

I know you will remember we were adding this but you had issues