Hello, Is Flyte capable of MANUALLY resuming the workflow from the failed task forward? By manually I mean either from the UI or CLI. I see a rerun button on the UI to the right of the tasks. Will clicking the rerun button resume the workflow or just rerun the failed task? What about manually marking a failed task as SUCCEEDED and resume the rest of the tasks forward?

Hi @Frank Shen Regarding resuming after failure, Intratask checkpoints could help you achieve that goal https://docs.flyte.org/projects/cookbook/en/latest/auto/core/control_flow/checkpoint.html#intratask-checkpoints As per manually changing the result of a Task, I'm not sure

Sorry @David Espejo (he/him), I forgot to add MANUALLY. I just amended my question. Please advise. Thanks!

Well, if you Relaunch the workflow it should run through the remaining of the DAG, starting by the failed Task

Hey @Frank Shen, so fixing the failed task and re-registering the workflow will currently require a completely new execution since the versions of everything change. This is an unfortunate byproduct of Flyte's very opinionated stance on execution reproducability, data lineages, etc. As @David Espejo (he/him) mentioned, you can use caching of the previous tasks as a workaround here, but obviously this means that any execution with the same

cache_version

and input values will use the same cache. The "rerun" button the task should only execute the individual task, and not resume the workflow. I should make sure you're aware of the "recover" functionality as well, although it may not directly apply here (ie. task updates). Recovery means that Flyte relaunches the workflow, but all of the nodes that were previously completed will be re-used. So if you have a workflow that had 3 tasks, where the first and second succeeded and the third failed due to pods being OOM killed or something. You could recover the workflow and Flyte would recover task executions 1 and 2 before reattempting the 3rd task execution. So this is basically what you want, but it will use the version of the task associated with the initial workflow rather than the updated one.

@Dan Rammer (hamersaw), “recover” functionality is exactly what I wanted. Could you tell / show me how to perform “recover”? I only see “relaunch” button on the workflow level.

Thanks a lot @David Espejo (he/him)!

Thanks to Dan for the great explanation! @Frank Shen please let us know anything else you may need

Ah... I also didn't realize this so far. So the recover behavior is what I expected to happen if I rerun a failed task, i.e. dependent task in the workflow will also run. This is a bit confusing in the UI... If we could also manually abort specific tasks and then recover the whole workflow, that would be exactly what my colleagues are asking for.

I believe they're the same.

When you look at a failed task, there is a button "rerun", and to it seemed the natural point to - well - rerun that task and if it succeeds recover the rest of the workflow. But this is not what this rerun button does... (i.e. launch further downstream tasks that are dependent on it). My colleagues also didn't intuitively understand what this rerun does. IMHO it would be much more clear if relaunch and rerun would also be renamed to clone, as this is much closer to what it actually does.