How do you configure node selectors with plugins such as pyspark and ray? For example add non-spot node selector to spark driver and spot node selector to spark executors or non-spot for ray head node and spot for worker nodes?

As far as I know this is handled (potentially differently) on the plugin level. As far as I know, the

spark

plugin as well as the

dask

plugin do have support for the

interruptible=True

flag in the

@task

decorator. When it’s set, only the workers/executors will be made interruptible. If I’m not mistaken, the

ray

plugin does not have this support yet (I had a brief look over the code and it doesn’t seem like it). As for other config values, this PR has a bit of context. cc @Fabio Grätz on the behavior of

interruptible=True

for

spark

tasks

There is a fantastic new functionality called names pos templates landing this release which makes this very good. Cc @Eduardo Apolinario (eapolinario) do you have better links

@Bernhard Stadlbauer Thanks! I'll test the pyspark interruptible thing next. It solves half of my use-case (I'd still like to use Ray on spot instances) and I can move forward now

@Aleksander Lempinen, so in the next Flyte release (coming in next week) we're announcing the ability of setting pod templates alongside task definitions. We're still working on docs and examples. Also, keep in mind that unfortunately this functionality is not going to be available yet in the case of spark / ray / dask tasks. cc: @Dan Rammer (hamersaw) for confirmation. That said, adding support for interruptible to ray tasks seems like a bug/oversight and we should fix this. cc: @Kevin Su, is that something that we could add to the ray tasks?

yes, we definitely can add it to ray task. I will work on it. However, if pod template can be used in ray task, you could use it first to workaround the issue.

@Eduardo Apolinario (eapolinario) correct. The

PodTemplate

work does not currently apply to the Ray, Spark, or Dask tasks. This is certainly something we should support.

It seems that only the labels and annotations appear on the spark driver & executor pods (for example I see a interruptible=true label and bar=test) but not node-selectors. I tried deploying the latest helm chart (v1.4.0-b0) to eks. Normal flyte workflows have their nodeselectors applied properly. Here is the relevant part of the helm chart I'm using:

k8s:
      plugins:
        k8s:
          default-labels:
            - bar: test
          default-annotations:
            - baz: non-overridden-value
          resource-tolerations:
            - <http://nvidia.com/gpu|nvidia.com/gpu>:
              - key: "<http://nvidia.com/gpu|nvidia.com/gpu>"
                operator: "Equal"
                value: "true"
                effect: "NoSchedule"
          default-node-selector:
            node-class: "worker-node"
            <http://aws.amazon.com/spot|aws.amazon.com/spot>: "false"
          interruptible-node-selector:
            node-class: "worker-node"
            <http://aws.amazon.com/spot|aws.amazon.com/spot>: "true"