Thomas Newton
10/10/2023, 9:42 AMcore_launchplan
from a dynamic task. I want to configure it so that I can run it with only a subsection of core_workflow
enabled, and I was hoping someone could give some suggestions how to do this. I've attached example.py
that shows the structure. I tried a few different ideas to make this work but I haven't managed to get anything working:
1. Use conditions. It seems like having an else
is required so to optionally turn on a set of tasks I think it would require an exponentially growing tree of conditions as the number of tasks increases.
2. Create the core_launchplan
inside the dynamic task. This doesn't work because launchplans can't be launched fully dynamically. The launchplan must be registered before its possible to launch it from a dynamic task.
3. Generate all the tasks, workflows and launchplans from within a function. This doesn't work because tasks must be declared at module scope so flyte can find them to run them.
4. Generate just workflows from a function and pass the core_lauchplan
through as an argument. I think this could work if I keep working at it but I'm currently having difficulties related to imports because flyte uses pickle
to serialise LaunchPlan
. I've attached solution_idea.py
which demonstrates this idea.
Hopefully that makes sense. If anyone has any ideas that would be much appreciated 🙂Ketan (kumare3)
Thomas Newton
10/10/2023, 1:36 PM@dynamic
task/workflow needs to be wrapped with another @workflow
. This is a little bit annoying because we need to pass though all the parameters. Also I was hoping to avoid another dynamic task because the workflow structure is known at submission time.Ketan (kumare3)
Thomas Newton
10/12/2023, 2:14 PMKetan (kumare3)
Thomas Newton
10/20/2023, 5:07 PMflytekit
I was able to get if working but only on the remote not in local mode.
I would be interested if you had any thoughts on this approach, or how we could make `conditional`s work on `ImperativeWorkflow`s?Ketan (kumare3)
Thomas Newton
10/24/2023, 7:34 AMKetan (kumare3)
conditional().if_(...)
and then just simply add it to an imperative workflowThomas Newton
10/24/2023, 8:28 PMconditional().if_(...)
to an imperative workflowKetan (kumare3)
Thomas Newton
10/24/2023, 10:48 PMKetan (kumare3)
Thomas Newton
10/25/2023, 10:21 AMdef create_conditional(arg1, arg2):
return conditional(...).if(...)...
imperative_workflow.add_entity(create_conditional, arg1=5, arg2=8)
This works when running on flyte. However when running locally it fails withPryce
11/02/2023, 6:33 PMThomas Newton
11/02/2023, 6:36 PMPryce
11/02/2023, 6:44 PMKetan (kumare3)
Thomas Newton
11/06/2023, 2:23 PMPryce
11/20/2023, 5:21 PMThomas Newton
11/20/2023, 7:42 PMPryce
11/21/2023, 9:31 PM