I would also have a question with regards to spot instances. We have a subworkflow (workflow within a workflow) which starts tasks on spot nodes (

interruptible=True

). The task then fails with

Last Error: USER::Pod was terminated in response to imminent node shutdown.

The last log from the node where this task is run on is:

Deleting node <node-id> because it does not exist in the cloud provider

So we assume the node is reclaimed by GCP. Does anyone happen to know whether there might be a difference in how interruptible task failures are handled in subworkflows?

Wdym? Do you think retries were not done?

The workflow failed without retries, yes

Did you have atleast one retry on the task?

Oh thats good to know - we did not have that 👍

Cc @Dan Rammer (hamersaw) can you confirm you need 1 retry to set to activate interruptible retry

Yes, setting a task as

interruptible

just sets combinations of

NodeSelectors

,

Affinities

, and

Tolerations

depending on the configuration. It doesn't effect the number of retries available to a task.

Here’s the related docs PR