https://flyte.org logo
#ask-the-community
Title
# ask-the-community
n

Nandakumar Raghu

11/30/2023, 3:27 PM
Hi All, We are currently using
default-node-selector:
in the k8s plugin config to select nodes on our MNG for flyte workers pods. This allows us to use either spot nodes of on-demand nodes on AWS. We want to set up nodeAffinity selectors with weights to try to use spot first and if not available, go to on-demand nodes. We are using flyte-binary chart and the
deployment
section in values.yaml has a
extraPodSpec
which I believe only applies to the flyte-binary pods. Are pod templates the only way to apply nodeAffinity selection rules to worker pods? Or is there any way to apply these via the values.yaml for flyte-binary?
s

Samhita Alla

12/01/2023, 9:13 AM
so is your ultimate goal to set up Flyte to run on spot instances? have you looked at the docs? https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/productionizing/spot_instances.html#using-spot-preemptible-instances does this help you in any way?
n

Nandakumar Raghu

12/06/2023, 11:00 AM
Ideally yes, we want the flyte worker pods to be launched on spot instances. We had this setup earlier, but we were not getting spot instances for the configuration we were requesting, so we switched to ondemand instances exclusively, however these are expensive, so we want to try to use node affinity selectors with weights instead of node selectors so k8s tries to get spot instances first, and only if not available provisions ondemand instances. I think we can achieve something by setting the nodeAffinity like this -
Copy code
nodeAffinity:
      preferredDuringSchedulingIgnoredDuringExecution:
      - weight: 100
        preference:
          matchExpressions:
          - key: node.myorg/lifecycle
            operator: In
            values:
            - flyte-spot
      - weight: 50
        preference:
          matchExpressions:
          - key: node.myorg/lifecycle
            operator: In
            values:
            - flyte
flyte-spot
is the spot MNG and
flyte
is the on demand MNG. I saw there is a default-affinity section in the k8s plugin config, but I am not able find any example of this and am not sure I can set the above settings there. Can you help me find where I can set this nodeAffinity setting?
s

Samhita Alla

12/06/2023, 1:14 PM
i'm not very sure. @David Espejo (he/him) any idea if it's possible to set node affinities in the helm chart?
d

David Espejo (he/him)

12/06/2023, 5:28 PM
@Nandakumar Raghu what chart are you using? is it
flyte-binary
or
flyte-core
?
I've been exploring and trying out options and so far, I think the best bet is to create a PodTemplate where you specify the
affinity
config and then set it as the default as indicated here. I haven't tried it my self yet but already tried the
default-affinity
section with no success
n

Nandakumar Raghu

12/07/2023, 9:22 AM
We are using
flyte-binary
4 Views