calm-pilot-2010
10/10/2023, 9:42 AMcore_launchplan
from a dynamic task. I want to configure it so that I can run it with only a subsection of core_workflow
enabled, and I was hoping someone could give some suggestions how to do this. I've attached example.py
that shows the structure. I tried a few different ideas to make this work but I haven't managed to get anything working:
1. Use conditions. It seems like having an else
is required so to optionally turn on a set of tasks I think it would require an exponentially growing tree of conditions as the number of tasks increases.
2. Create the core_launchplan
inside the dynamic task. This doesn't work because launchplans can't be launched fully dynamically. The launchplan must be registered before its possible to launch it from a dynamic task.
3. Generate all the tasks, workflows and launchplans from within a function. This doesn't work because tasks must be declared at module scope so flyte can find them to run them.
4. Generate just workflows from a function and pass the core_lauchplan
through as an argument. I think this could work if I keep working at it but I'm currently having difficulties related to imports because flyte uses pickle
to serialise LaunchPlan
. I've attached solution_idea.py
which demonstrates this idea.
Hopefully that makes sense. If anyone has any ideas that would be much appreciated 🙂freezing-airport-6809
freezing-airport-6809
calm-pilot-2010
10/10/2023, 1:36 PMcalm-pilot-2010
10/11/2023, 9:53 PM@dynamic
task/workflow needs to be wrapped with another @workflow
. This is a little bit annoying because we need to pass though all the parameters. Also I was hoping to avoid another dynamic task because the workflow structure is known at submission time.freezing-airport-6809
calm-pilot-2010
10/12/2023, 2:14 PMfreezing-airport-6809
freezing-airport-6809
calm-pilot-2010
10/20/2023, 5:07 PMflytekit
I was able to get if working but only on the remote not in local mode.
I would be interested if you had any thoughts on this approach, or how we could make `conditional`s work on `ImperativeWorkflow`s?calm-pilot-2010
10/20/2023, 5:17 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
calm-pilot-2010
10/24/2023, 7:34 AMfreezing-airport-6809
freezing-airport-6809
conditional().if_(...)
and then just simply add it to an imperative workflowcalm-pilot-2010
10/24/2023, 8:28 PMconditional().if_(...)
to an imperative workflowfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
calm-pilot-2010
10/24/2023, 10:48 PMfreezing-airport-6809
freezing-airport-6809
calm-pilot-2010
10/25/2023, 10:21 AMdef create_conditional(arg1, arg2):
return conditional(...).if(...)...
imperative_workflow.add_entity(create_conditional, arg1=5, arg2=8)
This works when running on flyte. However when running locally it fails witheager-processor-63090
11/02/2023, 6:33 PMcalm-pilot-2010
11/02/2023, 6:36 PMeager-processor-63090
11/02/2023, 6:44 PMfreezing-airport-6809
calm-pilot-2010
11/06/2023, 2:23 PMcalm-pilot-2010
11/17/2023, 7:55 PMeager-processor-63090
11/20/2023, 5:21 PMcalm-pilot-2010
11/20/2023, 7:42 PMeager-processor-63090
11/21/2023, 9:31 PM