Hello. I have a workflow that launches

core_launchplan

from a dynamic task. I want to configure it so that I can run it with only a subsection of

core_workflow

enabled, and I was hoping someone could give some suggestions how to do this. I've attached

example.py

that shows the structure. I tried a few different ideas to make this work but I haven't managed to get anything working: 1. Use conditions. It seems like having an

else

is required so to optionally turn on a set of tasks I think it would require an exponentially growing tree of conditions as the number of tasks increases. 2. Create the

core_launchplan

inside the dynamic task. This doesn't work because launchplans can't be launched fully dynamically. The launchplan must be registered before its possible to launch it from a dynamic task. 3. Generate all the tasks, workflows and launchplans from within a function. This doesn't work because tasks must be declared at module scope so flyte can find them to run them. 4. Generate just workflows from a function and pass the

core_lauchplan

through as an argument. I think this could work if I keep working at it but I'm currently having difficulties related to imports because flyte uses

pickle

to serialise

LaunchPlan

. I've attached

solution_idea.py

which demonstrates this idea. Hopefully that makes sense. If anyone has any ideas that would be much appreciated 🙂

@Thomas Newton you can use dybamic wirkflows nested in dynamic And finally as last resort use @eager (this is experimental and new) - but pure python workflows

You mean a single extra dynamic task within the `core_launchplan`'s workflow that creates another workflow containing the subset of tasks we want to enable?

Let me think - cc @Eduardo Apolinario (eapolinario) fyi

Thanks. I plan to try the eager mode idea, but I've been distracted with other stuff that last couple of days.

No worries Tom

I tried your eager mode suggestion. I did manage to get it working, but I had quite a few issues. Generally though, I think we would prefer to stick to static workflows if we can or dynamic if required. I've been playing around with this a bit and I think I found some stuff that helps but not a fully working solution. Discovery 1: We can use a conditional that returns a workflow in each of the conditional branches. This avoids needing to use a dynamic workflow when we have a relatively small number <10 modes that we want to run it in. Discovery 2: Both my discovery 1 and the dynamic workflow option require writing out another wrapper workflow and passing the long list of arguments through to each workflow variant. Using imperative workflows its possible to automatically generate this wrapper workflow and make the user experience a lot more streamlined. The problem I'm having is that I can't get conditional working with imperative workflows. With some minor changes to

flytekit

I was able to get if working but only on the remote not in local mode. I would be interested if you had any thoughts on this approach, or how we could make `conditional`s work on `ImperativeWorkflow`s?

ohh man, i did not

I'm not trying to use nested. Just normal conditional in imperative.

Cool will try today.

I just want to add a

conditional().if_(...)

to an imperative workflow

ohh that should be easy

With about 10 lines change to flytekit I had it working for remote execution but I couldn't get it working locally.

no worries, sorry i have been crazy busy lately as you can imagine haha

Not sure how helpful it is but this is what I ended up with https://github.com/Tom-Newton/flytekit/pull/4/files. This allows doing this:

def create_conditional(arg1, arg2):
    return conditional(...).if(...)...

imperative_workflow.add_entity(create_conditional, arg1=5, arg2=8)

This works when running on flyte. However when running locally it fails with

Hey @Thomas Newton where did you land with this? Could I be of assistance?

I actually have an idea I wanted to try, for solving my original problem of configurable core workflow launched by dynamic workflows. Previously I was a bit stuck because I wanted to use flyte conditionals from inside an imperative workflow. Regardless of how I solve my original problem supporting conditionals in imperative workflows would be a nice feature.

Sounds good, lmk if you want me to take a look at your revised approach

@Pryce / @Thomas Newton sorry for the delay, but Conditionals are not supported in Imperative. This is because conditionals create multiple nodes and today we do not have that in imperative. That being said, it can be added, i was trying, but got stuck on some interface decisions.

Thanks for looking into it. Could be a useful feature but probably it's quite niche. Hopefully my other idea will mitigate my need for it.

Hey @Thomas Newton, that sounds pretty cool! Would you mind sharing a gist or something with your code? I'd love to take a look at how you implemented this.

Here is an example of what I ended up doing.

Very cool! Thanks for sharing. We've also added experimental eager workflows recently, which may be another option if you run into any limitations with this solution: https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/eager_workflows.html