https://flyte.org logo
Join Slack
Powered by
Hey Flyte Team! I’m trying to use LaunchPlans/Refe...
# flytekit
g
Hey Flyte Team! I’m trying to use LaunchPlans/ReferenceLaunchPlans together and came across an edge case that I was hoping to get some feedback on. I have 2 projects (Project A and Project B). In Project A I have a workflow that uses a LaunchPlan, and executes fine if being called from Project A. However, if I try to call this workflow via a ReferenceLaunchPlan in Project B, Flyte attempts to find the LaunchPlan in Project B instead of Project A. This is unexpected, and I’m wondering if anyone has ideas on how to get around this. The following error occurs at runtime:
Copy code
[User] incorrectly specified launchplan projectb:development:projecta.workflows.test_workflow.wrapper_workflow:1, caused by: [NotFound] No launch plan retrieved from Admin, caused by: rpc error: code = NotFound desc = missing entity of type LAUNCH_PLAN with identifier project:"projectb" domain:"development" name:"projecta.workflows.test_workflow.wrapper_workflow" version:"1"
Project A:
Copy code
└── projecta
    └── workflows
        └── test_workflow.py (shown below)
Copy code
from flytekit import LaunchPlan, dynamic, task, workflow


@task
def dummy_task(a: int) -> int:
    return a


@workflow
def wrapper_workflow(a: int) -> int:
    return dummy_task(a=a)


@dynamic
def dynamic_lp(a: int) -> int:
    # In our real-world example, we are using LaunchPlan here rather than subworkflow to handle large fan out
    lp = LaunchPlan.get_or_create(wrapper_workflow)
    return lp(a=a)


@workflow
def workflow_test(a: int) -> int:
    return dynamic_lp(a=a)
Project B:
Copy code
from flytekit import workflow
from flytekit.core.launch_plan import reference_launch_plan


def interface(a: int) -> int:
    ...


rlp = reference_launch_plan(
    project="projecta",
    domain="development",
    name="projecta.workflows.test_workflow.workflow_test",
    version="1",
)(interface)


@workflow
def workflow_calling_reference(a: int) -> int:
    return rlp(a=a)
This provides an example, where Project A’s workflow is registered, and subsequently called in Project B via a reference launch plan
y
can you click on the json tab for the dynamic_lp task and see if you see an environment variable 
_F_SS_C
defined on it please?
g
yes, it’s set to a long hash
y
got it
g
looks like project/domain/version are null after deserializing 
_F_SS_C
y
sorry let me get back to this in a bit… will need to do some digging.
i can see what’s happening, just not sure what the fix is.
g
thanks for looking into it @Yee! Let me know if I can help in any way!
y
you’re saying you did
Copy code
SerializationSettings.from_transport(...)
this is on the dynamic task 
dynamic_lp
it shouldn’t be none… it should be the project/domain that you initially registered the projecta file to
g
I haven’t done anything with this:
Copy code
SerializationSettings.from_transport(...)
What should I be doing with it?
y
oh you said
looks like project/domain/version are null after deserializing 
_F_SS_C
g
sorry, I took the string and base64 decode/decompressed with gzip manually
y
basically the same thing.
but it does handle some json stuff
is it still none if you use from_transport?
g
same thing, None for project and domain
This is when checking the 
_F_SS_C
on the 
dynamic_lp
task’s config
y
and how was that registered?
like what was the command?
g
Copy code
pyflyte --config ${FLYTE_INTERNAL_CONFIGURATION_PATH} serialize --in-container-config-path flyte.cfg workflows -f ${TMP_DIR}/
flyte-cli register-files \
    --domain ${FLYTE_INTERNAL_DOMAIN} \
    --project ${FLYTE_INTERNAL_PROJECT} \
    --version ${FLYTE_INTERNAL_VERSION} \
    --kubernetes-service-account ${FLYTE_AUTH_KUBERNETES_SERVICE_ACCOUNT} \
    $TMP_DIR/*
y
@Ketan (kumare3) can you remind me how this pattern works? I don’t see code in flytectl that overrides the 
_F_SS_C
env var
285 Views
Open in Slack
PreviousNext
Powered by