Hey, a question about the recover function in Flyte: I think at this given time it is not possible to recover a failed workflow with different task versions right? Looking at use cases where a workflow fails, the user registers a new version for the failed task and recovers the failed workflow with the latest tasks version. I think this can also be done via caching but is this maybe on some agenda?

given the static nature of how workflows are defined, i think this is a ways off, kinda goes against the reproducibility of flyte i feel.

Yeah makes complete sense to me - thank you!

@Yee - Jan brings up an interesting point in this. We've hit this too and I think this scenario is pretty common. Imagine we run our N step workflow. At step, say 5, it fails because the task has a problem. We need to fix our code so the task can complete but steps 1-4 are very long-running tasks and the outputs they created are large. What is the "most right" way to re-use those nicely preserved/immutable outputs from steps 1-4? So far, we've just re-ran the entire workflow or come up with not-so-great ways to re-use the outputs.

Could you two work on a ticket for this? Whatever form this takes, this will be a pretty core change to flyte. I can see some ways in which this can be accomplished, but they probably more fall in line with what terence was thinking… re-using prior outputs in a new workflow definition, rather than somehow augmenting the existing workflow or workflow execution. to do what you said today @Terence Kent you would’ve needed to have the foresight to make the outputs cached.