How do you configure node selectors with plugins s...
# ask-the-community
a
How do you configure node selectors with plugins such as pyspark and ray? For example add non-spot node selector to spark driver and spot node selector to spark executors or non-spot for ray head node and spot for worker nodes?
b
As far as I know this is handled (potentially differently) on the plugin level. As far as I know, the
spark
plugin as well as the
dask
plugin do have support for the
interruptible=True
flag in the
@task
decorator. When it’s set, only the workers/executors will be made interruptible. If I’m not mistaken, the
ray
plugin does not have this support yet (I had a brief look over the code and it doesn’t seem like it). As for other config values, this PR has a bit of context. cc @Fabio Grätz on the behavior of
interruptible=True
for
spark
tasks
k
There is a fantastic new functionality called names pos templates landing this release which makes this very good. Cc @Eduardo Apolinario (eapolinario) do you have better links
a
@Bernhard Stadlbauer Thanks! I'll test the pyspark interruptible thing next. It solves half of my use-case (I'd still like to use Ray on spot instances) and I can move forward now
@Ketan (kumare3) I assume you mean pod templates? I'd love to hear more!
e
@Aleksander Lempinen, so in the next Flyte release (coming in next week) we're announcing the ability of setting pod templates alongside task definitions. We're still working on docs and examples. Also, keep in mind that unfortunately this functionality is not going to be available yet in the case of spark / ray / dask tasks. cc: @Dan Rammer (hamersaw) for confirmation. That said, adding support for interruptible to ray tasks seems like a bug/oversight and we should fix this. cc: @Kevin Su, is that something that we could add to the ray tasks?
k
yes, we definitely can add it to ray task. I will work on it. However, if pod template can be used in ray task, you could use it first to workaround the issue.
d
@Eduardo Apolinario (eapolinario) correct. The
PodTemplate
work does not currently apply to the Ray, Spark, or Dask tasks. This is certainly something we should support.
a
It seems that only the labels and annotations appear on the spark driver & executor pods (for example I see a interruptible=true label and bar=test) but not node-selectors. I tried deploying the latest helm chart (v1.4.0-b0) to eks. Normal flyte workflows have their nodeselectors applied properly. Here is the relevant part of the helm chart I'm using:
Copy code
k8s:
      plugins:
        k8s:
          default-labels:
            - bar: test
          default-annotations:
            - baz: non-overridden-value
          resource-tolerations:
            - <http://nvidia.com/gpu|nvidia.com/gpu>:
              - key: "<http://nvidia.com/gpu|nvidia.com/gpu>"
                operator: "Equal"
                value: "true"
                effect: "NoSchedule"
          default-node-selector:
            node-class: "worker-node"
            <http://aws.amazon.com/spot|aws.amazon.com/spot>: "false"
          interruptible-node-selector:
            node-class: "worker-node"
            <http://aws.amazon.com/spot|aws.amazon.com/spot>: "true"