Hello! We noticed that the ray plugin, when used to create a RayCluster and RayJob on another, separate, GKE cluster, it will default to try to create those resources in a namespace (in the separate GKE cluster) with the same name as the one running the flyte workflow. Have some questions here: • Is there any way to override this behaviour? ◦ If there isn’t any, is this something that could be added?

you can’t override for now, but you can add a new config (namespace?) in ray plugin, and override rayjob’s namespace

thanks for the quick reply?

and override rayjob’s namespace

how would I do that? I was wondering if we could potentially add an extra parameter to https://github.com/flyteorg/flyteidl/blob/master/protos/flyteidl/plugins/ray.proto#L17 that defines the namespace. Then we could take that new field in the plugin and override the namespace in the rayjob

name is critical to flyte

to be clear, we don’t want to change the namespace where the flyte workflow is running, just where the rayjob and raycluster will be created (we are already using another gke cluster for ray than flyte) you are saying that more than just the ray flyte plugin depends on the namespace to be named the same?

the question is - how do we add namespace. pod-templates seem to be the right thing to do this. But, this can lead to a lot of problems, about how we track the status etc

@Ketan (kumare3) @Kevin Su I would like to follow up on this issue. When we were trying out the Ray plugin, we had the following issues related to our Ray/Flyte internal setup: 1. Namespace to launch a Ray job (Flyte uses the same namespace where Flyte job runs for the Ray job) 2. Fixed service account (Flyte doesn’t specify a service account in Ray job, which picks up

default

k8s SA) 3. Doesn’t allow different resource configurations for head and workers (Currently head and workers use the same resource configurations) 4. Node pool selector for head/worker nodes (There’s no way we can specify node pool in head/worker spec)

No 3/4 seem very fixable problems

@Ketan (kumare3), regarding:

For 2 for every execution you can specify a separate service account

can you share an example on how to specify the SA for the rayjob (which runs in a different GKE cluster than flyte)

Also please feel free to propose contributions

Thanks Ketan! (I heard you are on vacation - sorry that my question interrupted your vacation, this is not urgent at all)

Namespaces should be configured at Flyte level and I am sorry this is non negotiable

Do you mind expanding it a bit? I think there might be a misunderstanding. In our internal setup, Flyte and Ray run on different GKE clusters (see the graph below). Here we are asking to make the namespace parameter for the RayJob configurable, so we can control the namespace of a RayJob k8s resource object that Flyte plugin submits to our Ray infrastructure. We do not want to change the namespace where Flyte job runs. Let us know if there’re any constraints that we are not aware of.