Channels
#ray-integration
Title
# ray-integration
b
Bosco Raju
08/28/2023, 5:34 PMHi Flyte folks,
In our organization, we have a strict requirement to label all of our Kubernetes resources.
Using Flytectl, I've successfully labeled the head and worker pods for Ray. However, these labels are not being propagated to the Kubernetes Jobs managed by the RayCluster. Could you provide guidance on how to label these Jobs through Flyte? Thank you
https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L144C2-L144C9
k
Ketan (kumare3)
08/28/2023, 6:48 PM
this seems like a kuberay issue
b
Bosco Raju
08/28/2023, 7:41 PMHi @Ketan (kumare3), The Head and worker pod labels are set using the
https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L210-L212
using the
TaskExecutionMetadataObjectMetay
Yicheng Lu
08/29/2023, 1:32 AMBased on the code, I think your description is correct.
k
Ketan (kumare3)
08/29/2023, 1:36 AM
Object Meta should be a superset
b
Bosco Raju
08/29/2023, 11:45 AMHi @Ketan (kumare3),
I've attempted to set labels from a PodTemplate in Kubernetes but the Kubernetes Job resource is not picking it up. As a Flyte user, I expect that labels set at the project level or via the PodTemplate should be applied to both Pods and Job resources. Could you confirm whether this behaviour can be addressed within Flyte before I reach out to the Ray community for further assistance? Thanks.
k
Ketan (kumare3)
08/29/2023, 1:45 PM
Yes, we will tal
h
Haytham Abuelfutuh
08/29/2023, 5:36 PM
Hi @Bosco Raju, can you confirm that you can see the labels appear in the RayJobSpec k8s object?
Labels of (Pod Template + TaskExecMetadata) should appear in the RayJobSpec, KubeRay then, supposedly creates the actual K8s Pod based on that spec. We can debug where things are broken by noting where do labels get dropped...
b
Bosco Raju
08/29/2023, 5:46 PMHi @Haytham Abuelfutuh thanks for looking into this issue. Yes, I can confirm the labels are present in RayJob.
h
Haytham Abuelfutuh
08/29/2023, 5:48 PM
I think you are looking at the Labels of the RayJob itself (which is great) but can you also look into the spec part of the RayJob?
where it says HeadSpec or something?
If you are ok sharing the RayJob spec would be great...
Copy code
kubectl get RayJob -n mlops-platform-sample-ml-project-sandbox fee0cc..... -o yamlb
Bosco Raju
08/29/2023, 5:53 PMHere you go
h
Haytham Abuelfutuh
08/29/2023, 5:55 PM
Are these the labels you are expecting?
Copy code
sennder.dora4.service/ba: senn-ai
sennder.dora4.service/component: sample-project
sennder.dora4.service/environment: playground
sennder.dora4.service/namespace: sennder
sennder.dora4.service/project: sample-project
sennder.dora4.service/team: mlops-platformb
Bosco Raju
08/29/2023, 5:55 PMYes
The labels appear on both Ray Head and Worker pod.
The issue is with the pod controlled by the job.
k
Kevin Su
08/29/2023, 6:00 PMis this pod created by kuberay? IIRC, kuberay only create the pods for worker and head nodes?
b
Bosco Raju
08/29/2023, 6:01 PMYes, this pod is controlled by a Job which is controlled by RayCluster.
k
Kevin Su
08/29/2023, 6:01 PMWhich version of kuberay you are using?
b
Bosco Raju
08/29/2023, 6:02 PMI am using 0.6.0
k
Kevin Su
08/29/2023, 6:12 PMsorry, could you ask kuberay community? In flyte, we only create ray job custom resource, and we do set the label for the pod spec. Kuberay is responsible to create the pods. seems like labels are not added to the pod.
b
Bosco Raju
08/29/2023, 6:13 PMThanks, @Kevin Su I will bring it up with the Kuberay community. Appreciate you guys looking into it.
h
Haytham Abuelfutuh
08/29/2023, 6:14 PM
🙏
I found this on the project: https://github.com/ray-project/kuberay/pull/1070
y
Yicheng Lu
08/29/2023, 6:37 PMThere have been recent changes on the Kuberay side. Previously, Kuberay would "submit jobs" directly. Now, Kuberay initiates a k8s job first. The problem you encounter may because Kuberay doesn't pass labels to that k8s job. Flyte only create Rayjob CR. So, not relate to Flyte.
Here's how Kuberay creates that k8s job. I don't see any labels being passed:https://github.com/ray-project/kuberay/pull/1177/files#diff-f10777dd022e918221782f3ea3f88c70293404f9e62ded13cbad66b68c6d19efR377
While labels cannot be set on that k8s job itself, the pods created by that job can have labels set. See this newly added field SubmitterPodTemplate: https://github.com/ray-project/kuberay/blob/55b1d39efb62aaaae74b4113ec8a227cdf0d17d6/ray-operator/config/samples/ray_v1alpha1_rayjob.yaml#L95
b
Bosco Raju
08/29/2023, 7:52 PMThanks for the added information, guys appreciate the input!
3 Views