Hi Flyte folks, In our organization, we have a strict requirement to label all of our Kubernetes resources. Using Flytectl, I've successfully labeled the head and worker pods for Ray. However, these labels are not being propagated to the Kubernetes Jobs managed by the RayCluster. Could you provide guidance on how to label these Jobs through Flyte? Thank you https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L144C2-L144C9

this seems like a kuberay issue

Hi @Ketan (kumare3), The Head and worker pod labels are set using the https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L210-L212 using the

TaskExecutionMetadata

but the Rayjob labels are set using the

ObjectMeta

from PodSpec https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L58C52-L58C66. https://github.com/flyteorg/flyteplugins/blob/master/go/tasks/plugins/k8s/ray/ray.go#L152-L158 Can you please confirm this behaviour?

Based on the code, I think your description is correct.

Object Meta should be a superset

Hi @Ketan (kumare3), I've attempted to set labels from a PodTemplate in Kubernetes but the Kubernetes Job resource is not picking it up. As a Flyte user, I expect that labels set at the project level or via the PodTemplate should be applied to both Pods and Job resources. Could you confirm whether this behaviour can be addressed within Flyte before I reach out to the Ray community for further assistance? Thanks.

Yes, we will tal

Hi @Bosco Raju, can you confirm that you can see the labels appear in the RayJobSpec k8s object?

Hi @Haytham Abuelfutuh thanks for looking into this issue. Yes, I can confirm the labels are present in RayJob.

I think you are looking at the Labels of the RayJob itself (which is great) but can you also look into the spec part of the RayJob?

Here you go

Are these the labels you are expecting?

sennder.dora4.service/ba: senn-ai
            sennder.dora4.service/component: sample-project
            sennder.dora4.service/environment: playground
            sennder.dora4.service/namespace: sennder
            sennder.dora4.service/project: sample-project
            sennder.dora4.service/team: mlops-platform

Yes

is this pod created by kuberay? IIRC, kuberay only create the pods for worker and head nodes?

Yes, this pod is controlled by a Job which is controlled by RayCluster.

Which version of kuberay you are using?

I am using 0.6.0

sorry, could you ask kuberay community? In flyte, we only create ray job custom resource, and we do set the label for the pod spec. Kuberay is responsible to create the pods. seems like labels are not added to the pod.

Thanks, @Kevin Su I will bring it up with the Kuberay community. Appreciate you guys looking into it.

🙏

There have been recent changes on the Kuberay side. Previously, Kuberay would "submit jobs" directly. Now, Kuberay initiates a k8s job first. The problem you encounter may because Kuberay doesn't pass labels to that k8s job. Flyte only create Rayjob CR. So, not relate to Flyte.

Thanks for the added information, guys appreciate the input!