https://flyte.org
Join Slack
Subworkflow question: Is it possible to use a workflow from another project in flyte as a subworkflo...
# flyte-support
s
Subworkflow question: Is it possible to use a workflow from another project in flyte as a subworkflow or do they both need to be in the same project? Another microservice I'm working with uses a workflow that I need and don't want to reproduce. Would I be able to use that as a subworkflow without having access to the microservice's underlying code?
m
so you can reference your workflow from another project within the same Flyte deployment and use it in another project
s
ahh yes! thank you thats exactly what I was looking for. Did not know this launchplan type existed. thanks!
👍 1
b
what is the difference between reference_launch_plan and reference_workflow?
h
@busy-agent-82804 the difference is the same as using subworkflows vs launchplans. Basically, subworkflows work by inlining the subworkflow DAG into the top-level workflow whereas launchplans will executea new workflow and in the parent workflow track the status of that execution.
👍 1
c
how do i import reference_launch_plan? 
from flytekit import reference_launch_plan
?
m
Copy code
from flytekit.core.launch_plan import LaunchPlan, reference_launch_plan
im doing it like above
🙏 1
b
what is the status for flytekit-java? is it production ready?
s
hi @magnificent-teacher-86590 finally getting to implementing this and have a question on executing the launch plan if you dont mind...i've created the reference launch plan and passed my function into the wrapper function to create a 
ReferenceLaunchPlan
object. If I call the 
execute()
method on that object I get an Exception 
"Remote reference entities cannot be run locally. You must mock this out."
. What is the correct way to finally execute the lp?
m
i think you need to submit a job to your remote cluster, i dont think reference LP work on locally. For local tests and runs i usually mock the wrapper.
s
Right, this was in my cluster actually. I had tried to register my new workflow that would execute the reference workflow then my new task at the end and got that message. For a quick overview this is what my code looks like when its oversimplified
Copy code
test_lp = reference_launch_plan(
    project="test", 
    domain="development", 
    name="test.workflows.thing.my_workflow", 
    version="1"
)
thing = test_lp(
    my_test_func(**kwargs)
)
test_data = thing.execute()
with the execute function causing the error
m
i think it needs to be a decorator, well that is what i have in our code base 😅
Copy code
@reference_launch_plan(
    project=my_project, domain=my_domain, name=my_ref_wf_name, version=version
)
def my_wf(<necessary parameters of the ref LP>
) -> <return type>:
    ...
👀 1
s
ahhh ok! I will try that out...was following launchplan docs to try and get a feel for it. that would make sense though, similar to the reference workflow
👍 1
Cool, am able to execute the lp. Thanks for your help. Have you noticed that the version given isn't picked up but it inherits the version of the parent workflow instead? I passed in a string for the version in the reference launch plan but that version is not taking for whatever reason. I had to go and sync the version of my reference lp to match the version I gave my workflow. Also for vars that I try to pass into my launch plan they are being viewed as promises...does that happen in yalls setup?
m
hmm that is odd, what do you mean it inherits the version from the parent. In our case it is getting the correct version specified in the decorator. oh that 
version=version
? i shoudlve said `version=<your ref LP version as string>
how are you calling your reference LP? is it from a dynamic task
s
So I have my new workflow that executes the reference workflow and then adds an additional task on top of it. This new workflow is tagged as version 
0.7.7
for example and then my reference launch plan is: 
@reference_launch_plan(project="test", domain="development", name="<http://test.workflows.thing.my|test.workflows.thing.my>_workflow", version="1")
. When I tried to serialize and register the workflow it was looking for ``test.workflows.thing.my_workflow`` version 
0.7.7
instead of version 
1
that I defined in the decorator
I was just calling it like this:
Copy code
@workflow
def workflow(
    city: str,
    first_name: str,
    last_name: str,
) -> str:
    call_to_reference_decorator(
        person_spec={
            "first_name": f"{first_name}",
            "last_name": f"{last_name}",
        },
        location={
            "city": f"{city}",
            "country": "USA",
        },
    )
    return 'Done'
for whatever reason if I just passed in 
first_name
or other vars as the raw variables and not string formatted I got a Binding issue
m
that looks like its not a reference LP but lives in your current project, so it should be looking at the overall tagged value right shoudl this be like 
@reference_launch_plan(project="some_other_test_project", domain="development", name="<http://test.workflows.thing.my|test.workflows.thing.my>_workflow", version="1")
s
yea sorry, this is an example of what I have above the workflow code:
Copy code
@reference_launch_plan(project="test", domain="development", name="test.workflows.thing.my_workflow", version="1")
def call_to_reference_decorator(
    first_name: str,
    last_name: str,
    city: str,
) -> str:
    return str
doing my best to simplify my current function since it takes many vars and objects as inputs...hopefully its still making sense
so have all that code in one file
m
your reference LP lives in a different project right?
s
Yup! But don't I have to create a wrapper function for my launch plan so flyte knows which vars are to be expected?
or is the only line I actually need the decorator line for the lp?
m
ohh i think i might know why, your 
call_to_reference_decorator
need to have a matching signature as your reference workflow or launchplan, in names and parameter and output tyoes
that is how we are using it in our projects, flyte folks know a bit more about the internals 😅
s
when you say "matching signature" thats just the inputs and types along with the return type, right? thats what I thought I was doing but maybe I need to remove 
return str
?
m
i think name might also need to match as well 🤔
well it might not need to, but yeah input and output types need to match Ohhhh yeah it needs to have just 3 dots. 
…
so remote 
return str
👍 1
just found out its 
ellipsis
haha not 3 dots 🤣
😂 1
s
ahhh ok. did not have that. also sorry realized my inputs did not match in that example bc I threw it together...they are matching in my code base but the reference lp has objects that do not exist in my repo so I replaced those types with 
TypedDict
from the 
typing
library
👍 1
But definitely did not have my decorator function matching that of the reference lp so maybe that would solve some issues 🤔 just seemed like when I tried to pass a workflow input into a TypedDict variable in the reference lp I get: 
AssertionError: Failed to Bind variable dataset_spec for function bulk_inference.workflows.inference.run_on_dataset.
so that error comes about from 
lp_func(dataset_spec={"dataset_id": dataset_id})
for whatever reason, but works if I string format the dataset_id 🤷
m
interesting, yeah probably something to do with type conversion, i just always try to keep typing as primitive as possible haha
s
haha yes agreed. I'm stealing a workflow that one of our data scientists has created so adhere to his definitions which have a few nested configs in there. May have to try and get creative to patch this up for now
👍 1
160 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by