Hi Flyte community, question about the general design of dynamic workflow. Why is dynamic workflow executed in a k8s pod instead of in the flyte propeller? Won’t this lead to potential performance issue due to the overhead of spinning up new k8s pods?
02/03/2023, 6:46 AM
what do you mean?
The dynamic workflow is materialize at runtime, using “users” data.
Propeller cannot load users data and run “users code”
The materialization runs in a pod, but the generated workflow is executed by flytepropeller itself
this is required as user could load arbitrary amount of data, they could also load “dataframes” or other python objects that are not loadable in propeller
dynamic workflow materialization is extremely powerful. It is “pay as you go” model. you can only yeild the tasks that you want to evaluate lazily. The entire compute for the materialization can be a “spark job” a “ray job” or a simpley “python job”
also you do not have to use dynamic. if you workflow does not depend on data - just generate it using static materialization.
this can be done in 2 ways