👋 I have a Flyte workflow I want to create that can be thought of as a data backfill job. The number of backfill jobs is variable and passed as a parameter. I want a flyte workflow that will take that input and create a chain of sequential tasks, where each job waits for the previous to finish. Can this be done, and, if so, how? I am familiar with using dynamic tasks for parallel jobs, but not sequential
f
freezing-airport-6809
05/22/2025, 5:32 PM
Infact there is a workflow that does this already
freezing-airport-6809
05/22/2025, 5:32 PM
Check pyflyte backfill command
freezing-airport-6809
05/22/2025, 5:32 PM
Or flyteremote.backfill
freezing-airport-6809
05/22/2025, 5:32 PM
You can look at the code
freezing-airport-6809
05/22/2025, 5:32 PM
It will do what you want I think almost exactly
o
orange-midnight-93312
05/22/2025, 5:33 PM
Thanks! Can you walk me through a way of doing it custom? I used the idea of a backfill as an abstraction, what I am actually doing is a little more complex than a standard backfill
orange-midnight-93312
05/22/2025, 5:33 PM
I am just looking for logic to include in a dynamic task to accomplish this