I think I found another bug: I tried to recover a workflow that was already itself a recovery. In that case the recovery did not work. IMHO that should work just like triggering the recovery on the original workflow. Or is it intended to work like it is?

@Felix Ruess tell me more. It should indeed work. I have tried recovering workflows and that has worked. It is possible that we may end up recomputing more, if the first recovered workflow was aborted earlier

The first recovered workflow failed with the same task as the original one, after another policy fix in our minio cluster, I recovered that one again and all tasks were re-run. I can share a screenshot, but not sure what you would see there, as I also then terminated that recover-of-the-recover, since I didn't want the tasks re-run...

All tasks were re run? That sounds wrong

yes

I've witnessed similar behavior where the recovery of a recovery restarts all tasks

@Maarten de Jong is this consistent

Yes

I see the bug - cc @Dan Rammer (hamersaw) This should have recovered state too https://github.com/flyteorg/flytepropeller/blob/752f55e9c7f7e357de707aa3eaae6e3af03b186f/pkg/controller/nodes/executor.go#L192

nice

@Felix Ruess can you file it or create a PR

sure, hope I understood correctly