Hi Flyte community, question about the general des...
# ask-the-communityy
Yubo Wang
02/03/2023, 6:45 AMHi Flyte community, question about the general design of dynamic workflow. Why is dynamic workflow executed in a k8s pod instead of in the flyte propeller? Won’t this lead to potential performance issue due to the overhead of spinning up new k8s pods?
k
Ketan (kumare3)
02/03/2023, 6:46 AM
what do you mean?
The dynamic workflow is materialize at runtime, using “users” data.
Propeller cannot load users data and run “users code”
The materialization runs in a pod, but the generated workflow is executed by flytepropeller itself
Ketan (kumare3)
02/03/2023, 6:47 AM
this is required as user could load arbitrary amount of data, they could also load “dataframes” or other python objects that are not loadable in propeller
Ketan (kumare3)
02/03/2023, 6:48 AM
dynamic workflow materialization is extremely powerful. It is “pay as you go” model. you can only yeild the tasks that you want to evaluate lazily. The entire compute for the materialization can be a “spark job” a “ray job” or a simpley “python job”
Ketan (kumare3)
02/03/2023, 6:50 AM
also you do not have to use dynamic. if you workflow does not depend on data - just generate it using static materialization.
this can be done in 2 ways
Copy code
@workflow - DSLy
Yubo Wang
02/03/2023, 6:58 AMThanks @Ketan (kumare3) the imperative workflow is helpful to us. However, we want to provide user with: 1. a workflow they can make reference to 2. can determine the dag based on user’s input
Yubo Wang
02/03/2023, 7:03 AMlet’s discuss on this in the meeting
k
Ketan (kumare3)
02/03/2023, 7:21 AM
Sure
152 Views