Looks like it was discussed here but there’s no conclusion

I’m going to try with

interruptible=False

to see if it alleviates the problem since it’s running on spot instances

Oh if it's using spot instances that explains a lot. the long running task is probably being evicted.

this is a flat in flytepropeller

The job is still getting evicted 😞

so basically, there is no way for flyte to tell the spot instances that it is unable to evict the pods. spot instances will give a warning (ie. mark the pod as deleted), and if the pod controller does not cleanup the resource within a predefined period then the spot instance will force delete the pod. this is what you're seeing, where flyte is looking for a pod that already started and can't find it. there are a few mechanisms within flyte that we use to combat this: (1) injecting finalizers. this is a way to tell k8s to not delete the resource until the finalizer is removed. however, in the case of spot instances these are ignored and the pod is deleted anyways. (2) system retries. the idea of using spot instances is typically to reduce the cost of compute. a simple mechanism to enable tasks here is to retry many times - which is configurable. so if you have system retries set to something like 50, flyte can retry 50 times. additionally, there is configuration for an 'interruptibleThreshold'. this means that when using interruptible you can say mark the first N retries as 'interruptible' but once it exceeds N retries the remainder are no longer marked as 'interruptible'. as you mentioned, this will likely not work either, as the spot instance is deleting the pods. (3) intra-task checkpointing. if the task is running some iterative operation the use of intra-task checkpointing allows the task to periodically save it's state. so even if it is interrupted, the next instance picks up at some mid-point rather than re-doing all of the work. afaik the only other fallback is to evaluate this workload on a non-spot instance.

happy to hop on a call and discuss options if you want!

intra task checkpointing may be a good approach for me instead of splitting the task into 2 separate tasks. I can try incorporating it into my code

and the orchestration and some data manipulation is what resides in Flyte

So I tried using the checkpoint and ran into this error

Expected exactly one checkpoint - found 0

What I did… Try reading checkpoint.. if empty trigger jenkins job and save the url of jenkins job (checkpoint.write()). Wait for job to finish

Question: Once a checkpoint is read, can the same data be “re-read”?

Also the intra-task checkpointing functionality can be a little unintuitive sometimes. Basically, each attempt of the task is provided the previous attempts checkpoint path and that is used. So the first attempt it should write the external job id to the checkpoint, and then in subsequent attempts you should read from the old, and write to the new immediately - this way it will be available in new attempts. This functionality is slated to be cleaned up in this issue . However, I will reiterate that this is pretty hacky, a plugin would be a better path.

and yes.. triggering jenkins is a web api call

I suppose I can look into writing the Jenkins plugin