Hi Flyte community, question about the general des...
# flyte-supportl
limited-raincoat-94253
02/03/2023, 6:45 AMHi Flyte community, question about the general design of dynamic workflow. Why is dynamic workflow executed in a k8s pod instead of in the flyte propeller? Won’t this lead to potential performance issue due to the overhead of spinning up new k8s pods?
f
freezing-airport-6809
02/03/2023, 6:46 AM
what do you mean?
The dynamic workflow is materialize at runtime, using “users” data.
Propeller cannot load users data and run “users code”
The materialization runs in a pod, but the generated workflow is executed by flytepropeller itself
freezing-airport-6809
02/03/2023, 6:47 AM
this is required as user could load arbitrary amount of data, they could also load “dataframes” or other python objects that are not loadable in propeller
👍 1
freezing-airport-6809
02/03/2023, 6:48 AM
dynamic workflow materialization is extremely powerful. It is “pay as you go” model. you can only yeild the tasks that you want to evaluate lazily. The entire compute for the materialization can be a “spark job” a “ray job” or a simpley “python job”
freezing-airport-6809
02/03/2023, 6:50 AM
also you do not have to use dynamic. if you workflow does not depend on data - just generate it using static materialization.
this can be done in 2 ways
Copy code
@workflow - DSLl
limited-raincoat-94253
02/03/2023, 6:58 AMThanks @freezing-airport-6809 the imperative workflow is helpful to us. However, we want to provide user with: 1. a workflow they can make reference to 2. can determine the dag based on user’s input
limited-raincoat-94253
02/03/2023, 7:03 AMlet’s discuss on this in the meeting
f
freezing-airport-6809
02/03/2023, 7:21 AM
Sure
154 Views