Channels
#ask-the-community
Title
# ask-the-community
a
Augie Palacios
02/27/2023, 9:18 PMSubworkflow question: Is it possible to use a workflow from another project in flyte as a subworkflow or do they both need to be in the same project? Another microservice I'm working with uses a workflow that I need and don't want to reproduce. Would I be able to use that as a subworkflow without having access to the microservice's underlying code?
j
Jay Ganbat
02/27/2023, 11:24 PMYeah we use reference launchplan in our pipelines
https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.reference_launch_plan.html#flytekit-reference-launch-plan
reference workflow
https://docs.flyte.org/projects/flytekit/en/latest/generated/flytekit.reference_workflow.html
so you can reference your workflow from another project within the same Flyte deployment and use it in another project
a
Augie Palacios
02/27/2023, 11:26 PMahh yes! thank you thats exactly what I was looking for. Did not know this launchplan type existed. thanks!
t
Taoufik Dachraoui
02/28/2023, 11:54 AMwhat is the difference between reference_launch_plan and reference_workflow?
d
Dan Rammer (hamersaw)
02/28/2023, 1:03 PM@Taoufik Dachraoui the difference is the same as using subworkflows vs launchplans. Basically, subworkflows work by inlining the subworkflow DAG into the top-level workflow whereas launchplans will executea new workflow and in the parent workflow track the status of that execution.
m
Melody Lui
02/28/2023, 10:11 PMhow do i import reference_launch_plan?
from flytekit import reference_launch_planj
Jay Ganbat
02/28/2023, 10:40 PM Copy code
from flytekit.core.launch_plan import LaunchPlan, reference_launch_plant
Taoufik Dachraoui
03/04/2023, 7:14 PMwhat is the status for flytekit-java? is it production ready?
a
Augie Palacios
03/06/2023, 7:23 PMhi @Jay Ganbat finally getting to implementing this and have a question on executing the launch plan if you dont mind...i've created the reference launch plan and passed my function into the wrapper function to create a
ReferenceLaunchPlanexecute()"Remote reference entities cannot be run locally. You must mock this out."j
Jay Ganbat
03/06/2023, 7:28 PMi think you need to submit a job to your remote cluster, i dont think reference LP work on locally. For local tests and runs i usually mock the wrapper.
a
Augie Palacios
03/06/2023, 7:32 PMRight, this was in my cluster actually. I had tried to register my new workflow that would execute the reference workflow then my new task at the end and got that message. For a quick overview this is what my code looks like when its oversimplified
Copy code
test_lp = reference_launch_plan(
project="test",
domain="development",
name="test.workflows.thing.my_workflow",
version="1"
)
thing = test_lp(
my_test_func(**kwargs)
)
test_data = thing.execute()j
Jay Ganbat
03/06/2023, 7:37 PMi think it needs to be a decorator, well that is what i have in our code base 😅
Copy code
@reference_launch_plan(
project=my_project, domain=my_domain, name=my_ref_wf_name, version=version
)
def my_wf(<necessary parameters of the ref LP>
) -> <return type>:
...a
Augie Palacios
03/06/2023, 7:39 PMahhh ok! I will try that out...was following launchplan docs to try and get a feel for it. that would make sense though, similar to the reference workflow
Cool, am able to execute the lp. Thanks for your help. Have you noticed that the version given isn't picked up but it inherits the version of the parent workflow instead? I passed in a string for the version in the reference launch plan but that version is not taking for whatever reason. I had to go and sync the version of my reference lp to match the version I gave my workflow. Also for vars that I try to pass into my launch plan they are being viewed as promises...does that happen in yalls setup?
j
Jay Ganbat
03/06/2023, 10:24 PMhmm that is odd, what do you mean it inherits the version from the parent. In our case it is getting the correct version specified in the decorator. oh that
version=versionhow are you calling your reference LP? is it from a dynamic task
a
Augie Palacios
03/06/2023, 10:31 PMSo I have my new workflow that executes the reference workflow and then adds an additional task on top of it. This new workflow is tagged as version
0.7.7@reference_launch_plan(project="test", domain="development", name="<http://test.workflows.thing.my|test.workflows.thing.my>_workflow", version="1")0.7.71I was just calling it like this:
Copy code
@workflow
def workflow(
city: str,
first_name: str,
last_name: str,
) -> str:
call_to_reference_decorator(
person_spec={
"first_name": f"{first_name}",
"last_name": f"{last_name}",
},
location={
"city": f"{city}",
"country": "USA",
},
)
return 'Done'first_namej
Jay Ganbat
03/06/2023, 10:34 PMthat looks like its not a reference LP but lives in your current project, so it should be looking at the overall tagged value right
shoudl this be like
@reference_launch_plan(project="some_other_test_project", domain="development", name="<http://test.workflows.thing.my|test.workflows.thing.my>_workflow", version="1")a
Augie Palacios
03/06/2023, 10:36 PMyea sorry, this is an example of what I have above the workflow code:
Copy code
@reference_launch_plan(project="test", domain="development", name="test.workflows.thing.my_workflow", version="1")
def call_to_reference_decorator(
first_name: str,
last_name: str,
city: str,
) -> str:
return strso have all that code in one file
j
Jay Ganbat
03/06/2023, 10:38 PMyour reference LP lives in a different project right?
a
Augie Palacios
03/06/2023, 10:39 PMYup! But don't I have to create a wrapper function for my launch plan so flyte knows which vars are to be expected?
or is the only line I actually need the decorator line for the lp?
j
Jay Ganbat
03/06/2023, 10:42 PMohh i think i might know why, your
call_to_reference_decoratorthat is how we are using it in our projects, flyte folks know a bit more about the internals 😅
a
Augie Palacios
03/06/2023, 10:44 PMwhen you say "matching signature" thats just the inputs and types along with the return type, right? thats what I thought I was doing but maybe I need to remove
return strj
Jay Ganbat
03/06/2023, 10:45 PMi think name might also need to match as well 🤔
well it might not need to, but yeah input and output types need to match
Ohhhh yeah it needs to have just 3 dots.
…return strjust found out its
ellipsisa
Augie Palacios
03/06/2023, 10:48 PMahhh ok. did not have that. also sorry realized my inputs did not match in that example bc I threw it together...they are matching in my code base but the reference lp has objects that do not exist in my repo so I replaced those types with
TypedDicttypingBut definitely did not have my decorator function matching that of the reference lp so maybe that would solve some issues 🤔 just seemed like when I tried to pass a workflow input into a TypedDict variable in the reference lp I get:
AssertionError: Failed to Bind variable dataset_spec for function bulk_inference.workflows.inference.run_on_dataset.so that error comes about from
lp_func(dataset_spec={"dataset_id": dataset_id})j
Jay Ganbat
03/06/2023, 10:57 PMinteresting, yeah probably something to do with type conversion, i just always try to keep typing as primitive as possible haha
a
Augie Palacios
03/06/2023, 10:59 PMhaha yes agreed. I'm stealing a workflow that one of our data scientists has created so adhere to his definitions which have a few nested configs in there. May have to try and get creative to patch this up for now
4 Views