Hello. I have a workflow that launches `core_launc...
# ask-the-community
Hello. I have a workflow that launches
from a dynamic task. I want to configure it so that I can run it with only a subsection of
enabled, and I was hoping someone could give some suggestions how to do this. I've attached
that shows the structure. I tried a few different ideas to make this work but I haven't managed to get anything working: 1. Use conditions. It seems like having an
is required so to optionally turn on a set of tasks I think it would require an exponentially growing tree of conditions as the number of tasks increases. 2. Create the
inside the dynamic task. This doesn't work because launchplans can't be launched fully dynamically. The launchplan must be registered before its possible to launch it from a dynamic task. 3. Generate all the tasks, workflows and launchplans from within a function. This doesn't work because tasks must be declared at module scope so flyte can find them to run them. 4. Generate just workflows from a function and pass the
through as an argument. I think this could work if I keep working at it but I'm currently having difficulties related to imports because flyte uses
to serialise
. I've attached
which demonstrates this idea. Hopefully that makes sense. If anyone has any ideas that would be much appreciated 🙂
@Thomas Newton you can use dybamic wirkflows nested in dynamic And finally as last resort use @eager (this is experimental and new) - but pure python workflows
I can try and write an example with your idea
You mean a single extra dynamic task within the `core_launchplan`'s workflow that creates another workflow containing the subset of tasks we want to enable?
Thanks for the suggestion. This works but I'm wondering if there is a better option. It seems like the
task/workflow needs to be wrapped with another
. This is a little bit annoying because we need to pass though all the parameters. Also I was hoping to avoid another dynamic task because the workflow structure is known at submission time.
Let me think - cc @Eduardo Apolinario (eapolinario) fyi
Thanks. I plan to try the eager mode idea, but I've been distracted with other stuff that last couple of days.
No worries Tom
Same here we have been busy as well- sometimes messages get lost in seattle of messages
I tried your eager mode suggestion. I did manage to get it working, but I had quite a few issues. Generally though, I think we would prefer to stick to static workflows if we can or dynamic if required. I've been playing around with this a bit and I think I found some stuff that helps but not a fully working solution. Discovery 1: We can use a conditional that returns a workflow in each of the conditional branches. This avoids needing to use a dynamic workflow when we have a relatively small number <10 modes that we want to run it in. Discovery 2: Both my discovery 1 and the dynamic workflow option require writing out another wrapper workflow and passing the long list of arguments through to each workflow variant. Using imperative workflows its possible to automatically generate this wrapper workflow and make the user experience a lot more streamlined. The problem I'm having is that I can't get conditional working with imperative workflows. With some minor changes to
I was able to get if working but only on the remote not in local mode. I would be interested if you had any thoughts on this approach, or how we could make `conditional`s work on `ImperativeWorkflow`s?
I've just discovered this comment on github > Imperative example coming soon On a github issue about nested conditionals https://github.com/flyteorg/flyte/issues/1052#issuecomment-857165689. It sounds like what I want is possible. Did you end up making such an example? I can't find it.
ohh man, i did not
nested declarative works
nested imperative should work too -- are you using imperative?
I'm not trying to use nested. Just normal conditional in imperative.
Cool will try today.
@Thomas Newton do you want to create the conditional using an imperative structure or is it ok to use
and then just simply add it to an imperative workflow
I just want to add a
to an imperative workflow
ohh that should be easy
ok will work on that
it seems there is no direct way to add_node
With about 10 lines change to flytekit I had it working for remote execution but I couldn't get it working locally.
no worries, sorry i have been crazy busy lately as you can imagine haha
cc @Pryce can you help?>
Not sure how helpful it is but this is what I ended up with https://github.com/Tom-Newton/flytekit/pull/4/files. This allows doing this:
Copy code
def create_conditional(arg1, arg2):
    return conditional(...).if(...)...

imperative_workflow.add_entity(create_conditional, arg1=5, arg2=8)
This works when running on flyte. However when running locally it fails with
Hey @Thomas Newton where did you land with this? Could I be of assistance?
I actually have an idea I wanted to try, for solving my original problem of configurable core workflow launched by dynamic workflows. Previously I was a bit stuck because I wanted to use flyte conditionals from inside an imperative workflow. Regardless of how I solve my original problem supporting conditionals in imperative workflows would be a nice feature.
Sounds good, lmk if you want me to take a look at your revised approach
@Pryce / @Thomas Newton sorry for the delay, but Conditionals are not supported in Imperative. This is because conditionals create multiple nodes and today we do not have that in imperative. That being said, it can be added, i was trying, but got stuck on some interface decisions.
Thanks for looking into it. Could be a useful feature but probably it's quite niche. Hopefully my other idea will mitigate my need for it.
I ended up solving this without needing to use imperative or conditionals. I created a module scope dictionary of workflows for the different modes. This can easily be done using a function to create the different modes of the workflow. Then the dynamic task that launches them can take a mode argument which is a simple type and lookup the correct workflow in the dictionary. Probably I should have thought of this idea before, but its working now 🙂.
Hey @Thomas Newton, that sounds pretty cool! Would you mind sharing a gist or something with your code? I'd love to take a look at how you implemented this.
Here is an example of what I ended up doing.
Very cool! Thanks for sharing. We've also added experimental eager workflows recently, which may be another option if you run into any limitations with this solution: https://docs.flyte.org/projects/cookbook/en/latest/auto_examples/advanced_composition/eager_workflows.html