Hi All, We are currently using

default-node-selector:

in the k8s plugin config to select nodes on our MNG for flyte workers pods. This allows us to use either spot nodes of on-demand nodes on AWS. We want to set up nodeAffinity selectors with weights to try to use spot first and if not available, go to on-demand nodes. We are using flyte-binary chart and the

deployment

section in values.yaml has a

extraPodSpec

which I believe only applies to the flyte-binary pods. Are pod templates the only way to apply nodeAffinity selection rules to worker pods? Or is there any way to apply these via the values.yaml for flyte-binary?

so is your ultimate goal to set up Flyte to run on spot instances? have you looked at the docs? https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/spot_instances.html#using-spot-preemptible-instances does this help you in any way?

Ideally yes, we want the flyte worker pods to be launched on spot instances. We had this setup earlier, but we were not getting spot instances for the configuration we were requesting, so we switched to ondemand instances exclusively, however these are expensive, so we want to try to use node affinity selectors with weights instead of node selectors so k8s tries to get spot instances first, and only if not available provisions ondemand instances. I think we can achieve something by setting the nodeAffinity like this -

nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: node.myorg/lifecycle
            operator: In
            values:
            - flyte-spot
      - weight: 50
        preference:
          matchExpressions:
          - key: node.myorg/lifecycle
            operator: In
            values:
            - flyte

flyte-spot

is the spot MNG and

flyte

is the on demand MNG. I saw there is a default-affinity section in the k8s plugin config, but I am not able find any example of this and am not sure I can set the above settings there. Can you help me find where I can set this nodeAffinity setting?

i'm not very sure. @David Espejo (he/him) any idea if it's possible to set node affinities in the helm chart?

@Nandakumar Raghu what chart are you using? is it

flyte-binary

or

flyte-core

?

We are using

flyte-binary