How do you configure node selectors with plugins such as pyspark and ray? For example add non-spot node selector to spark driver and spot node selector to spark executors or non-spot for ray head node and spot for worker nodes?

As far as I know this is handled (potentially differently) on the plugin level. As far as I know, the

spark

plugin as well as the

dask

plugin do have support for the

interruptible=True

flag in the

@task

decorator. When it’s set, only the workers/executors will be made interruptible. If I’m not mistaken, the

ray

plugin does not have this support yet (I had a brief look over the code and it doesn’t seem like it). As for other config values, this PR has a bit of context. cc @late-pencil-3873 on the behavior of

interruptible=True

for

spark

tasks

There is a fantastic new functionality called names pos templates landing this release which makes this very good. Cc @high-accountant-32689 do you have better links

@agreeable-kitchen-44189 Thanks! I'll test the pyspark interruptible thing next. It solves half of my use-case (I'd still like to use Ray on spot instances) and I can move forward now

@shy-room-84005, so in the next Flyte release (coming in next week) we're announcing the ability of setting pod templates alongside task definitions. We're still working on docs and examples. Also, keep in mind that unfortunately this functionality is not going to be available yet in the case of spark / ray / dask tasks. cc: @hallowed-mouse-14616 for confirmation. That said, adding support for interruptible to ray tasks seems like a bug/oversight and we should fix this. cc: @glamorous-carpet-83516, is that something that we could add to the ray tasks?

yes, we definitely can add it to ray task. I will work on it. However, if pod template can be used in ray task, you could use it first to workaround the issue.

@high-accountant-32689 correct. The

PodTemplate

work does not currently apply to the Ray, Spark, or Dask tasks. This is certainly something we should support.

It seems that only the labels and annotations appear on the spark driver & executor pods (for example I see a interruptible=true label and bar=test) but not node-selectors. I tried deploying the latest helm chart (v1.4.0-b0) to eks. Normal flyte workflows have their nodeselectors applied properly. Here is the relevant part of the helm chart I'm using:

k8s:
      plugins:
        k8s:
          default-labels:
            - bar: test
          default-annotations:
            - baz: non-overridden-value
          resource-tolerations:
            - <http://nvidia.com/gpu|nvidia.com/gpu>:
              - key: "<http://nvidia.com/gpu|nvidia.com/gpu>"
                operator: "Equal"
                value: "true"
                effect: "NoSchedule"
          default-node-selector:
            node-class: "worker-node"
            <http://aws.amazon.com/spot|aws.amazon.com/spot>: "false"
          interruptible-node-selector:
            node-class: "worker-node"
            <http://aws.amazon.com/spot|aws.amazon.com/spot>: "true"