Aleksander Lempinen
02/24/2023, 12:47 PMBernhard Stadlbauer
02/24/2023, 1:07 PMspark
plugin as well as the dask
plugin do have support for the interruptible=True
flag in the @task
decorator. When it’s set, only the workers/executors will be made interruptible. If I’m not mistaken, the ray
plugin does not have this support yet (I had a brief look over the code and it doesn’t seem like it).
As for other config values, this PR has a bit of context.
cc @Fabio Grätz on the behavior of interruptible=True
for spark
tasksKetan (kumare3)
Aleksander Lempinen
02/24/2023, 6:10 PMEduardo Apolinario (eapolinario)
02/24/2023, 7:07 PMKevin Su
02/24/2023, 7:19 PMDan Rammer (hamersaw)
02/24/2023, 7:21 PMPodTemplate
work does not currently apply to the Ray, Spark, or Dask tasks. This is certainly something we should support.Aleksander Lempinen
02/27/2023, 3:50 AMk8s:
plugins:
k8s:
default-labels:
- bar: test
default-annotations:
- baz: non-overridden-value
resource-tolerations:
- <http://nvidia.com/gpu|nvidia.com/gpu>:
- key: "<http://nvidia.com/gpu|nvidia.com/gpu>"
operator: "Equal"
value: "true"
effect: "NoSchedule"
default-node-selector:
node-class: "worker-node"
<http://aws.amazon.com/spot|aws.amazon.com/spot>: "false"
interruptible-node-selector:
node-class: "worker-node"
<http://aws.amazon.com/spot|aws.amazon.com/spot>: "true"