https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Augie Palacios

02/27/2023, 9:18 PM
Subworkflow question: Is it possible to use a workflow from another project in flyte as a subworkflow or do they both need to be in the same project? Another microservice I'm working with uses a workflow that I need and don't want to reproduce. Would I be able to use that as a subworkflow without having access to the microservice's underlying code?
so you can reference your workflow from another project within the same Flyte deployment and use it in another project
a

Augie Palacios

02/27/2023, 11:26 PM
ahh yes! thank you thats exactly what I was looking for. Did not know this launchplan type existed. thanks!
t

Taoufik Dachraoui

02/28/2023, 11:54 AM
what is the difference between reference_launch_plan and reference_workflow?
d

Dan Rammer (hamersaw)

02/28/2023, 1:03 PM
@Taoufik Dachraoui the difference is the same as using subworkflows vs launchplans. Basically, subworkflows work by inlining the subworkflow DAG into the top-level workflow whereas launchplans will executea new workflow and in the parent workflow track the status of that execution.
m

Melody Lui

02/28/2023, 10:11 PM
how do i import reference_launch_plan?
from flytekit import reference_launch_plan
?
j

Jay Ganbat

02/28/2023, 10:40 PM
Copy code
from flytekit.core.launch_plan import LaunchPlan, reference_launch_plan
im doing it like above
t

Taoufik Dachraoui

03/04/2023, 7:14 PM
what is the status for flytekit-java? is it production ready?
a

Augie Palacios

03/06/2023, 7:23 PM
hi @Jay Ganbat finally getting to implementing this and have a question on executing the launch plan if you dont mind...i've created the reference launch plan and passed my function into the wrapper function to create a
ReferenceLaunchPlan
object. If I call the
execute()
method on that object I get an Exception
"Remote reference entities cannot be run locally. You must mock this out."
. What is the correct way to finally execute the lp?
j

Jay Ganbat

03/06/2023, 7:28 PM
i think you need to submit a job to your remote cluster, i dont think reference LP work on locally. For local tests and runs i usually mock the wrapper.
a

Augie Palacios

03/06/2023, 7:32 PM
Right, this was in my cluster actually. I had tried to register my new workflow that would execute the reference workflow then my new task at the end and got that message. For a quick overview this is what my code looks like when its oversimplified
Copy code
test_lp = reference_launch_plan(
    project="test", 
    domain="development", 
    name="test.workflows.thing.my_workflow", 
    version="1"
)
thing = test_lp(
    my_test_func(**kwargs)
)
test_data = thing.execute()
with the execute function causing the error
j

Jay Ganbat

03/06/2023, 7:37 PM
i think it needs to be a decorator, well that is what i have in our code base 😅
Copy code
@reference_launch_plan(
    project=my_project, domain=my_domain, name=my_ref_wf_name, version=version
)
def my_wf(<necessary parameters of the ref LP>
) -> <return type>:
    ...
a

Augie Palacios

03/06/2023, 7:39 PM
ahhh ok! I will try that out...was following launchplan docs to try and get a feel for it. that would make sense though, similar to the reference workflow
Cool, am able to execute the lp. Thanks for your help. Have you noticed that the version given isn't picked up but it inherits the version of the parent workflow instead? I passed in a string for the version in the reference launch plan but that version is not taking for whatever reason. I had to go and sync the version of my reference lp to match the version I gave my workflow. Also for vars that I try to pass into my launch plan they are being viewed as promises...does that happen in yalls setup?
j

Jay Ganbat

03/06/2023, 10:24 PM
hmm that is odd, what do you mean it inherits the version from the parent. In our case it is getting the correct version specified in the decorator. oh that
version=version
? i shoudlve said `version=<your ref LP version as string>
how are you calling your reference LP? is it from a dynamic task
a

Augie Palacios

03/06/2023, 10:31 PM
So I have my new workflow that executes the reference workflow and then adds an additional task on top of it. This new workflow is tagged as version
0.7.7
for example and then my reference launch plan is:
@reference_launch_plan(project="test", domain="development", name="<http://test.workflows.thing.my|test.workflows.thing.my>_workflow", version="1")
. When I tried to serialize and register the workflow it was looking for ``test.workflows.thing.my_workflow`` version
0.7.7
instead of version
1
that I defined in the decorator
I was just calling it like this:
Copy code
@workflow
def workflow(
    city: str,
    first_name: str,
    last_name: str,
) -> str:
    call_to_reference_decorator(
        person_spec={
            "first_name": f"{first_name}",
            "last_name": f"{last_name}",
        },
        location={
            "city": f"{city}",
            "country": "USA",
        },
    )
    return 'Done'
for whatever reason if I just passed in
first_name
or other vars as the raw variables and not string formatted I got a Binding issue
j

Jay Ganbat

03/06/2023, 10:34 PM
that looks like its not a reference LP but lives in your current project, so it should be looking at the overall tagged value right shoudl this be like
@reference_launch_plan(project="some_other_test_project", domain="development", name="<http://test.workflows.thing.my|test.workflows.thing.my>_workflow", version="1")
a

Augie Palacios

03/06/2023, 10:36 PM
yea sorry, this is an example of what I have above the workflow code:
Copy code
@reference_launch_plan(project="test", domain="development", name="test.workflows.thing.my_workflow", version="1")
def call_to_reference_decorator(
    first_name: str,
    last_name: str,
    city: str,
) -> str:
    return str
doing my best to simplify my current function since it takes many vars and objects as inputs...hopefully its still making sense
so have all that code in one file
j

Jay Ganbat

03/06/2023, 10:38 PM
your reference LP lives in a different project right?
a

Augie Palacios

03/06/2023, 10:39 PM
Yup! But don't I have to create a wrapper function for my launch plan so flyte knows which vars are to be expected?
or is the only line I actually need the decorator line for the lp?
j

Jay Ganbat

03/06/2023, 10:42 PM
ohh i think i might know why, your
call_to_reference_decorator
need to have a matching signature as your reference workflow or launchplan, in names and parameter and output tyoes
that is how we are using it in our projects, flyte folks know a bit more about the internals 😅
a

Augie Palacios

03/06/2023, 10:44 PM
when you say "matching signature" thats just the inputs and types along with the return type, right? thats what I thought I was doing but maybe I need to remove
return str
?
j

Jay Ganbat

03/06/2023, 10:45 PM
i think name might also need to match as well 🤔
well it might not need to, but yeah input and output types need to match Ohhhh yeah it needs to have just 3 dots.
so remote
return str
just found out its
ellipsis
haha not 3 dots 🤣
a

Augie Palacios

03/06/2023, 10:48 PM
ahhh ok. did not have that. also sorry realized my inputs did not match in that example bc I threw it together...they are matching in my code base but the reference lp has objects that do not exist in my repo so I replaced those types with
TypedDict
from the
typing
library
But definitely did not have my decorator function matching that of the reference lp so maybe that would solve some issues 🤔 just seemed like when I tried to pass a workflow input into a TypedDict variable in the reference lp I get:
AssertionError: Failed to Bind variable dataset_spec for function bulk_inference.workflows.inference.run_on_dataset.
so that error comes about from
lp_func(dataset_spec={"dataset_id": dataset_id})
for whatever reason, but works if I string format the dataset_id 🤷
j

Jay Ganbat

03/06/2023, 10:57 PM
interesting, yeah probably something to do with type conversion, i just always try to keep typing as primitive as possible haha
a

Augie Palacios

03/06/2023, 10:59 PM
haha yes agreed. I'm stealing a workflow that one of our data scientists has created so adhere to his definitions which have a few nested configs in there. May have to try and get creative to patch this up for now
4 Views