sparse-family-37927
08/18/2023, 5:25 PM@task
def example_task(num:int) -> None:
.....
@workflow
def test_workflow() -> None:
for i in range(5):
example_test(num=i)
It seems one way is that I can use dynamic workflow. However, in dynamic workflow, the tasks in the for loop will be running in parallel. I wanted the tasks can be running in sequence. So do you have a method to realize the for loop in sequence. Thanks!magnificent-teacher-86590
08/18/2023, 5:28 PMbroad-monitor-993
08/18/2023, 5:39 PMsparse-family-37927
08/18/2023, 5:52 PM@task
def example_task(num:int) -> None:
.....
@workflow
def test_workflow() -> None:
example_test_wf1 = example_test(num=0)
example_test_wf2 = example_test(num=1)
example_test_wf3 = example_test(num=2)
example_test_wf4 = example_test(num=3)
example_test_wf5 = example_test(num=4)
example_test_wf1 >> example_test_wf2
example_test_wf2 >> example_test_wf3
example_test_wf3 >> example_test_wf4
example_test_wf4 >> example_test_wf5
The code is kind of redundant and it will be more ugly if I have more tasks in the for loop. I am exploring if there is any more concise way.magnificent-teacher-86590
08/18/2023, 6:02 PMdep_out = 0
for i in range(5):
output, dep_out = example_test_wf(..., dep_out=dep_out)
and use dynamic task
i actually never tried this before so let me know if this works 😅sparse-family-37927
08/18/2023, 6:21 PMexample_test_wf
will be running in parallel task as far as I know (please let me know if there is some configuration that can make them running in sequence).magnificent-teacher-86590
08/18/2023, 6:22 PMsparse-family-37927
08/18/2023, 6:24 PMsparse-family-37927
08/18/2023, 7:13 PMworkflow
works well. It's kind of surprising as I thought for loop is not supported in workflow. It seems there are other people have the same impression as me (see the discussion here https://flyte-org.slack.com/archives/CP2HDHKE1/p1685726254115599?thread_ts=1685464169.404789&cid=CP2HDHKE1). I guess maybe Flyte official did some improvements there. Btw, I am using the latest Flytekit 1.9.0.
To test the for loop works in workflow, I tried the following, it worked and it was running in parallel, just like dynamic task.
@task
def example_task(num: int) -> None:
print(f"current num: {num}")
@workflow
def quick_test_workflow():
for i in range(6):
example_task(num=i)
magnificent-teacher-86590
08/18/2023, 8:01 PM