Hi Flyte community question about the general design of dyna

Question

Hi Flyte community question about the general design of dynamic workflow Why is dynamic workflow executed in a k8s pod instead of in the flyte propeller Won t this lead to potential performance issue due to the overhead of spinning up new k8s pods

Ketan (kumare3) · Answer

what do you mean The dynamic workflow is materialize at runtime using users data Propeller cannot load users data and run users code The materialization runs in a pod but the generated workflow is executed by flytepropeller itself

Ketan (kumare3) · Answer

this is required as user could load arbitrary amount of data they could also load dataframes or other python objects that are not loadable in propeller

Ketan (kumare3) · Answer

dynamic workflow materialization is extremely powerful It is pay as you go model you can only yeild the tasks that you want to evaluate lazily The entire compute for the materialization can be a spark job a ray job or a simpley python job

Ketan (kumare3) · Answer

also you do not have to use dynamic if you workflow does not depend on data just generate it using static materialization this can be done in 2 ways ``` workflow DSL``` OR I just wrote an entire backfill system using imperative workflows

Yubo Wang · Answer

Thanks < Ketan kumare3 > the imperative workflow is helpful to us However we want to provide user with 1 a workflow they can make reference to 2 can determine the dag based on user s input

Yubo Wang · Answer

let s discuss on this in the meeting

Ketan (kumare3) · Answer

Sure