Hi. I'm reading the documentation on enabling gang scheduling here: https://docs.flyte.org/en/latest/deployment/plugins/k8s/index.html I have question around enabling Yunikorn scheduler. In particular:

template:
  metadata:
    annotations:
      <http://yunikorn.apache.org/task-group-name|yunikorn.apache.org/task-group-name>: ""
      <http://yunikorn.apache.org/task-groups|yunikorn.apache.org/task-groups>: ""
      <http://yunikorn.apache.org/schedulingPolicyParameters|yunikorn.apache.org/schedulingPolicyParameters>: ""

Should the task-group-name be unique value shared for all the pods that belong to the same Flyte execution and task group? Or is there a different unique identifier that Yubikorn will pick to identify pods that belong to the same Flyte execution and task-group-name needs to be unique only within the specific Flyte execution. Cc @cool-lifeguard-49380

Can’t unswer for yubicorn unfortunately, only know about Kubernetes scheduler plugins with co-scheduling.

Thanks. That link is broken btw (same link in the Flyte docs).

> Thanks. That link is broken btw (same link in the Flyte docs). Thanks, will adapt in the docs.

Hi @swift-scooter-38934 Saw that you are from Yunikorn. I am wondering how I can set applicationId label to make my pods that are part of the same flyte execution to be considered part of the same application by Yunikorn.

@fierce-oil-47448 you could use the pod template argument of @task to add a label which generates a uuid at registration time.

What if the same registered workflow is run multiple times?

Ok then this approach won’t work, only if all worker pods would need the same value, regardless of what it is.

My understanding is that all worker pods that are part of the same PyTorchJob need a shared unique value for their applicationId.

If it’s any shared unique value, write a function that creates a pod template for the task which generates a random uuid for this label everytime it’s called maybe. Worth a try.

cc @dazzling-zebra-38345

Hi @fierce-oil-47448 Task group name, it must be unique within the yunikorn application. But i don't know how to set labels and annotions to template.metadata by flytectl. Expected tf yaml would be like.

apiVersion: kubeflow.org/v1
kind: TFJob
metadata:
  name: dist-mnist-for-e2e-test
  namespace: kubeflow
  labels:
    applicationId: app-01
spec:
  tfReplicaSpecs:
    PS:
      replicas: 1
      restartPolicy: Never
      template:
        metadata:
          labels:
            applicationId: "tf-job-001"
            queue: root.sandbox
          annotations:
            yunikorn.apache.org/task-group-name: task-group-example
            yunikorn.apache.org/task-groups: |-
              [{
                "name": "task-group-example",
                "minMember": 3,
                "minResource": {
                  "cpu": "100m",
                  "memory": "50M"
                },
                "nodeSelector": {},
                "tolerations": [],
                "affinity": {},
                "topologySpreadConstraints": []
              }]
        spec:
          schedulerName: yunikorn
          containers:
            - name: tensorflow
              image: kubeflow/tf-dist-mnist-test:1.0
              imagePullPolicy: IfNotPresent
    Worker:
      replicas: 2
      restartPolicy: Never
      template:
        metadata:
          labels:
            applicationId: "tf-job-001"
            queue: root.sandbox
          annotations:
            yunikorn.apache.org/task-group-name: task-group-example
            yunikorn.apache.org/task-groups: |-
              [{
                "name": "task-group-example",
                "minMember": 3,
                "minResource": {
                  "cpu": "100m",
                  "memory": "50M"
                },
                "nodeSelector": {},
                "tolerations": [],
                "affinity": {},
                "topologySpreadConstraints": []
              }]
        spec:
          schedulerName: yunikorn
          containers:
            - name: tensorflow
              image: kubeflow/tf-dist-mnist-test:1.0
              imagePullPolicy: IfNotPresent

Thanks @swift-scooter-38934. Is the applicationId at the TFJob required? From the documentation, the applicationId under the template should be sufficient.

Yes, yunikorn can generate a application itself. https://github.com/apache/yunikorn-k8shim/blob/master/pkg/common/utils/utils.go#L185 For example, i submitted the tf job after removing the applicationId. Result was following.

ok, but for gang scheduling purposes, I need to have a unique metadata.labels.applicationId, right?

Yes. Yunikorn receives pod level information from k8s, and add the pod to the specific application based on applicationId label. So consistency of the applicationId label in pods require. For example, tf PS and worker will belong to different applications in yunikorn if i only give worker pods with a unique applicationId. Necessary annotations to activate Yunikorn gang scheduling are following

<http://yunikorn.apache.org/task-group-name|yunikorn.apache.org/task-group-name>

<http://yunikorn.apache.org/task-groups|yunikorn.apache.org/task-groups>

Ok, so I can skip top level applicationId completely, as long as I provide unique id for the pod spec’s applicationId.

Cc @faint-activity-87590