https://flyte.org logo
#ray-integration
Title
# ray-integration
b

Bosco Raju

08/28/2023, 5:34 PM
Hi Flyte folks, In our organization, we have a strict requirement to label all of our Kubernetes resources. Using Flytectl, I've successfully labeled the head and worker pods for Ray. However, these labels are not being propagated to the Kubernetes Jobs managed by the RayCluster. Could you provide guidance on how to label these Jobs through Flyte? Thank you https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L144C2-L144C9
k

Ketan (kumare3)

08/28/2023, 6:48 PM
this seems like a kuberay issue
b

Bosco Raju

08/28/2023, 7:41 PM
Hi @Ketan (kumare3), The Head and worker pod labels are set using the https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L210-L212 using the
TaskExecutionMetadata
but the Rayjob labels are set using the
ObjectMeta
from PodSpec https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L58C52-L58C66. https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L152-L158 Can you please confirm this behaviour?
y

Yicheng Lu

08/29/2023, 1:32 AM
Based on the code, I think your description is correct.
k

Ketan (kumare3)

08/29/2023, 1:36 AM
Object Meta should be a superset
b

Bosco Raju

08/29/2023, 11:45 AM
Hi @Ketan (kumare3), I've attempted to set labels from a PodTemplate in Kubernetes but the Kubernetes Job resource is not picking it up. As a Flyte user, I expect that labels set at the project level or via the PodTemplate should be applied to both Pods and Job resources. Could you confirm whether this behaviour can be addressed within Flyte before I reach out to the Ray community for further assistance? Thanks.
k

Ketan (kumare3)

08/29/2023, 1:45 PM
Yes, we will tal
h

Haytham Abuelfutuh

08/29/2023, 5:36 PM
Hi @Bosco Raju, can you confirm that you can see the labels appear in the RayJobSpec k8s object?
Labels of (Pod Template + TaskExecMetadata) should appear in the RayJobSpec, KubeRay then, supposedly creates the actual K8s Pod based on that spec. We can debug where things are broken by noting where do labels get dropped...
b

Bosco Raju

08/29/2023, 5:46 PM
Hi @Haytham Abuelfutuh thanks for looking into this issue. Yes, I can confirm the labels are present in RayJob.
h

Haytham Abuelfutuh

08/29/2023, 5:48 PM
I think you are looking at the Labels of the RayJob itself (which is great) but can you also look into the spec part of the RayJob?
where it says HeadSpec or something?
If you are ok sharing the RayJob spec would be great...
Copy code
kubectl get RayJob -n mlops-platform-sample-ml-project-sandbox fee0cc..... -o yaml
b

Bosco Raju

08/29/2023, 5:53 PM
Here you go
h

Haytham Abuelfutuh

08/29/2023, 5:55 PM
Are these the labels you are expecting?
Copy code
sennder.dora4.service/ba: senn-ai
            sennder.dora4.service/component: sample-project
            sennder.dora4.service/environment: playground
            sennder.dora4.service/namespace: sennder
            sennder.dora4.service/project: sample-project
            sennder.dora4.service/team: mlops-platform
b

Bosco Raju

08/29/2023, 5:55 PM
Yes
The labels appear on both Ray Head and Worker pod.
The issue is with the pod controlled by the job.
k

Kevin Su

08/29/2023, 6:00 PM
is this pod created by kuberay? IIRC, kuberay only create the pods for worker and head nodes?
b

Bosco Raju

08/29/2023, 6:01 PM
Yes, this pod is controlled by a Job which is controlled by RayCluster.
k

Kevin Su

08/29/2023, 6:01 PM
Which version of kuberay you are using?
b

Bosco Raju

08/29/2023, 6:02 PM
I am using 0.6.0
k

Kevin Su

08/29/2023, 6:12 PM
sorry, could you ask kuberay community? In flyte, we only create ray job custom resource, and we do set the label for the pod spec. Kuberay is responsible to create the pods. seems like labels are not added to the pod.
b

Bosco Raju

08/29/2023, 6:13 PM
Thanks, @Kevin Su I will bring it up with the Kuberay community. Appreciate you guys looking into it.
h

Haytham Abuelfutuh

08/29/2023, 6:14 PM
🙏
y

Yicheng Lu

08/29/2023, 6:37 PM
There have been recent changes on the Kuberay side. Previously, Kuberay would "submit jobs" directly. Now, Kuberay initiates a k8s job first. The problem you encounter may because Kuberay doesn't pass labels to that k8s job. Flyte only create Rayjob CR. So, not relate to Flyte.
While labels cannot be set on that k8s job itself, the pods created by that job can have labels set. See this newly added field SubmitterPodTemplate: https://github.com/ray-project/kuberay/blob/55b1d39efb62aaaae74b4113ec8a227cdf0d17d6/ray-operator/config/samples/ray_v1alpha1_rayjob.yaml#L95
b

Bosco Raju

08/29/2023, 7:52 PM
Thanks for the added information, guys appreciate the input!
3 Views