https://flyte.org logo
#ask-the-community
Title
# ask-the-community
y

Yubo Wang

02/03/2023, 6:45 AM
Hi Flyte community, question about the general design of dynamic workflow. Why is dynamic workflow executed in a k8s pod instead of in the flyte propeller? Won’t this lead to potential performance issue due to the overhead of spinning up new k8s pods?
k

Ketan (kumare3)

02/03/2023, 6:46 AM
what do you mean? The dynamic workflow is materialize at runtime, using “users” data. Propeller cannot load users data and run “users code” The materialization runs in a pod, but the generated workflow is executed by flytepropeller itself
this is required as user could load arbitrary amount of data, they could also load “dataframes” or other python objects that are not loadable in propeller
dynamic workflow materialization is extremely powerful. It is “pay as you go” model. you can only yeild the tasks that you want to evaluate lazily. The entire compute for the materialization can be a “spark job” a “ray job” or a simpley “python job”
also you do not have to use dynamic. if you workflow does not depend on data - just generate it using static materialization. this can be done in 2 ways
Copy code
@workflow - DSL
OR imperative workflows I just wrote an entire backfill system using imperative workflows. PR -
y

Yubo Wang

02/03/2023, 6:58 AM
Thanks @Ketan (kumare3) the imperative workflow is helpful to us. However, we want to provide user with: 1. a workflow they can make reference to 2. can determine the dag based on user’s input
let’s discuss on this in the meeting
k

Ketan (kumare3)

02/03/2023, 7:21 AM
Sure
75 Views