#246 [Feature] Move Propeller creation of CompiledWorkflowClosure to Admin Issue created by wild-endeavor Motivation: Why do you think this is important? Currently, as part of changes to be released in flyteorg/flytepropeller#103, we realized that Propeller doesn't always have all the information necessary to execute a dynamic task, which gets rendered internally as a subworkflow. To remedy this, in that PR, we have added code to retrieve the missing launch plan information. However, this call is expensive, and since it's done one at a time, can lead to degradation of propeller performance (imagine if a dynamic task yielded a lot of launch plans). There are two other cases of when propeller will need to contact Admin • After this is done, dynamic tasks that yield tasks that are fetched, will also not have the underlying task definitions and we will again need to retrieve them from Admin. • Dynamically created subworkflows will contain tasks (and other entities) that will need to be fetched from Admin (obviously, since they're not known at compile time by definition). Because there are so many vectors for this call, we should just move the entire compilation step of subworkflows over to Admin... Admin already does workflow compilation here. We can easily expose this as an endpoint. We can potentially also do a minor optimization and only call this endpoint from propeller if the additional information is necessary. Goal: What should the final outcome look like, ideally? We should add a new endpoint in Admin that will do this entire compilation and return the

CompiledWorkflowClosure

. Describe alternatives you've considered We considered two possibilities • Adding to the construct we have now, where we make one Admin call to fetch the definition of every entity and indeed this is what we're going to do in the short-term for launch plans, but longer term this was seen as an subpar solution. • Can ignore the interface check for launch plans and just run anyways. If the interface doesn't match at the end, then things will fail then. But this only works for launch plans (because for tasks and workflows, we literally need the definition). Flyte component ☐ Overall ☐ Flyte Setup and Installation scripts ☐ Flyte Documentation ☐ Flyte communication (slack/email etc) ☑︎ FlytePropeller ☑︎ FlyteIDL (Flyte specification language) ☐ Flytekit (Python SDK) ☑︎ FlyteAdmin (Control Plane service) ☐ FlytePlugins ☐ DataCatalog ☐ FlyteStdlib (common libraries) ☐ FlyteConsole (UI) ☐ Other flyteorg/flyte