Hi. I'm reading the documentation on enabling gang...
# ask-the-community
b
Hi. I'm reading the documentation on enabling gang scheduling here: https://docs.flyte.org/en/latest/deployment/plugins/k8s/index.html I have question around enabling Yunikorn scheduler. In particular:
Copy code
template:
  metadata:
    annotations:
      <http://yunikorn.apache.org/task-group-name|yunikorn.apache.org/task-group-name>: ""
      <http://yunikorn.apache.org/task-groups|yunikorn.apache.org/task-groups>: ""
      <http://yunikorn.apache.org/schedulingPolicyParameters|yunikorn.apache.org/schedulingPolicyParameters>: ""
Should the task-group-name be unique value shared for all the pods that belong to the same Flyte execution and task group? Or is there a different unique identifier that Yubikorn will pick to identify pods that belong to the same Flyte execution and task-group-name needs to be unique only within the specific Flyte execution. Cc @Fabio Grätz
f
Can’t unswer for yubicorn unfortunately, only know about Kubernetes scheduler plugins with co-scheduling.
b
Thanks. That link is broken btw (same link in the Flyte docs).
I think
applicationId
label is needed for identifying pods belonging to the same application.
f
> Thanks. That link is broken btw (same link in the Flyte docs). Thanks, will adapt in the docs.
b
Hi @陳Chen宥騰Yuteng Saw that you are from Yunikorn. I am wondering how I can set applicationId label to make my pods that are part of the same flyte execution to be considered part of the same application by Yunikorn.
Flyte pods have a unique execution-id label, but I don't see how we can add an additional applicationId.
f
@Buğra Gedik you could use the pod template argument of @task to add a label which generates a uuid at registration time.
Don’t know if there is a more native way.
b
What if the same registered workflow is run multiple times?
It looks like I need label.applicationId to have the same value as label.execution-id.
f
Ok then this approach won’t work, only if all worker pods would need the same value, regardless of what it is.
b
My understanding is that all worker pods that are part of the same PyTorchJob need a shared unique value for their applicationId.
f
If it’s any shared unique value, write a function that creates a pod template for the task which generates a random uuid for this label everytime it’s called maybe. Worth a try.
k
cc @Ryan Lo
u
Hi @Buğra Gedik Task group name, it must be unique within the yunikorn application. But i don't know how to set labels and annotions to template.metadata by flytectl. Expected tf yaml would be like.
Copy code
apiVersion: kubeflow.org/v1
kind: TFJob
metadata:
  name: dist-mnist-for-e2e-test
  namespace: kubeflow
  labels:
    applicationId: app-01
spec:
  tfReplicaSpecs:
    PS:
      replicas: 1
      restartPolicy: Never
      template:
        metadata:
          labels:
            applicationId: "tf-job-001"
            queue: root.sandbox
          annotations:
            yunikorn.apache.org/task-group-name: task-group-example
            yunikorn.apache.org/task-groups: |-
              [{
                "name": "task-group-example",
                "minMember": 3,
                "minResource": {
                  "cpu": "100m",
                  "memory": "50M"
                },
                "nodeSelector": {},
                "tolerations": [],
                "affinity": {},
                "topologySpreadConstraints": []
              }]
        spec:
          schedulerName: yunikorn
          containers:
            - name: tensorflow
              image: kubeflow/tf-dist-mnist-test:1.0
              imagePullPolicy: IfNotPresent
    Worker:
      replicas: 2
      restartPolicy: Never
      template:
        metadata:
          labels:
            applicationId: "tf-job-001"
            queue: root.sandbox
          annotations:
            yunikorn.apache.org/task-group-name: task-group-example
            yunikorn.apache.org/task-groups: |-
              [{
                "name": "task-group-example",
                "minMember": 3,
                "minResource": {
                  "cpu": "100m",
                  "memory": "50M"
                },
                "nodeSelector": {},
                "tolerations": [],
                "affinity": {},
                "topologySpreadConstraints": []
              }]
        spec:
          schedulerName: yunikorn
          containers:
            - name: tensorflow
              image: kubeflow/tf-dist-mnist-test:1.0
              imagePullPolicy: IfNotPresent
b
Thanks @陳Chen宥騰Yuteng. Is the applicationId at the TFJob required? From the documentation, the applicationId under the template should be sufficient.
u
Yes, yunikorn can generate a application itself. https://github.com/apache/yunikorn-k8shim/blob/master/pkg/common/utils/utils.go#L185 For example, i submitted the tf job after removing the applicationId. Result was following.
b
ok, but for gang scheduling purposes, I need to have a unique metadata.labels.applicationId, right?
u
Yes. Yunikorn receives pod level information from k8s, and add the pod to the specific application based on applicationId label. So consistency of the applicationId label in pods require. For example, tf PS and worker will belong to different applications in yunikorn if i only give worker pods with a unique applicationId. Necessary annotations to activate Yunikorn gang scheduling are following
<http://yunikorn.apache.org/task-group-name|yunikorn.apache.org/task-group-name>
<http://yunikorn.apache.org/task-groups|yunikorn.apache.org/task-groups>
b
Ok, so I can skip top level applicationId completely, as long as I provide unique id for the pod spec’s applicationId.
k
Cc @Jan Fiedler
Cc @Haytham Abuelfutuh
Cc @John Votta