https://flyte.org logo
#ask-the-community
Title
# ask-the-community
v

Vishal Dumont

09/15/2023, 3:55 PM
Hey Folks! I have a use-case where I'd like to backfill/rerun from middle of the workflow. A simple example:
Copy code
If I have one workflow(A) that has three sub workflows(X, Y, Z). 
- Y depends on X
- Z depends on Y

Essentially, X -> Y -> Z

Let's assume X, Y & Z were successful. However, subworkflow Y had generated bad data i.e. has bugs and needs to be re-run. 

My goal is to re-run Y and it's downstreams i.e. want Z to pick up the latest partitions of Y.
This would be similar to
Clear Downstreams
option in Airflow. Does something like this exist in Flyte? I looked at recover, but this only seems to be for Failed tasks.
cc: @Andrés Gómez Ferrer
k

Ketan (kumare3)

09/15/2023, 4:21 PM
Did you use caching? If not then simply use reference workflows to construct one on the fly and run
Open up a jupyter notebook use remote.fetch_launchplan and then simply compose and register from jupyter off you go
v

Vishal Dumont

09/15/2023, 5:15 PM
Right, so I can potentially invalidate the cache and run it, but it's still quite a bit of manual work if you have 200+ tasks in a workflow and you want to run it from a task inbetween. Ideally, I'd like to avoid using jupyter notebook and want to find an alternative in a production setting, something similar to how airflow offers it with the click of a button?
k

Ketan (kumare3)

09/16/2023, 1:51 AM
This is a feature request
I guess you could run one task and (single task) and then run the entire workflow
This would cause running subsequent tasks
Automatically
v

Vishal Dumont

09/18/2023, 1:03 PM
ahh got it, thank you for the support! 🙂
k

Ketan (kumare3)

09/18/2023, 1:31 PM
Can you please come back and tell if it works, also I will try to reproduce just to ensure it works (it should) One problem I see is in ReRun task, does not automatically copy the inputs to the launch form 😞 - please file an issue
v

Vishal Dumont

09/18/2023, 1:59 PM
I guess you could run one task and (single task) and then run the entire workflow
For the above, you're hoping running a single task invalidates the subsequent tasks as well, right? And when I re-run the workflow, it should run all the downstreams as well?
k

Ketan (kumare3)

09/18/2023, 2:05 PM
Yup