hi there hoping someone can help me out with launching a wor Flyte #flytekit

hi there! hoping someone can help me out with laun...

Sharon Gong

04/14/2022, 7:19 PM

hi there! hoping someone can help me out with launching a workflow. I'm able to launch a workflow in flytectl that I'm unable to using flytekit's

FlyteRemote

. for some context, this workflow was created by copying over the

PythonFunctionWorkflow

and it redefines the workflow's module and name. details in 🧵

Sharon Gong

04/14/2022, 7:21 PM

the error I get from

FlyteRemote

is this

Copy code

KeyError(resource_type: LAUNCH_PLAN
project: "sp-one-model"
domain: "staging"
name: "onemodel.models.subscribers.onemodel"
version: "2eb9f303-a29e-42e7-909f-21ea0916e56c"
)

where the stacktrace is

Copy code

File "/Users/sharong/repo/jasper-api/backend/jasper_api/routes.py", line 235, in execute
    lp = remote.fetch_launch_plan(project, domain, workflow, version)
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/remote.py", line 443, in fetch_launch_plan
    workflow = self.fetch_workflow(wf_id.project, wf_id.domain, wf_id.name, wf_id.version)
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/remote.py", line 414, in fetch_workflow
    return FlyteWorkflow.promote_from_closure(compiled_wf, node_launch_plans)
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/workflow.py", line 159, in promote_from_closure
    tasks=tasks,
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/workflow.py", line 106, in promote_from_model
    for node in base_model_non_system_nodes
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/workflow.py", line 106, in <dictcomp>
    for node in base_model_non_system_nodes
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/nodes.py", line 89, in promote_from_model
    tasks,
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/component_nodes.py", line 119, in promote_from_model
    tasks=tasks,
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/workflow.py", line 106, in promote_from_model
    for node in base_model_non_system_nodes
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/workflow.py", line 106, in <dictcomp>
    for node in base_model_non_system_nodes
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/nodes.py", line 89, in promote_from_model
    tasks,
  File "/Users/sharong/.pyenv/versions/3.7.4/envs/jasper-api/lib/python3.7/site-packages/flytekit/remote/component_nodes.py", line 108, in promote_from_model
    base_model.launchplan_ref, node_launch_plans[base_model.launchplan_ref]

Sharon Gong

04/14/2022, 7:23 PM

I compared the workflow against the original workflow and I noticed the attributes

lhs

and

location

were not defined. it seems like

lhs

needs to be defined - can you elaborate on what the attribute is for and how it's defined?

Sharon Gong

04/14/2022, 7:23 PM

cc @Dylan Wilder

Yee

04/14/2022, 7:55 PM

hey can you share the code please?

Yee

04/14/2022, 7:55 PM

feel free to redact as much as you need… just want to see the structure of it

Yee

04/14/2022, 7:56 PM

also which flytekit version are you on?

Sharon Gong

04/14/2022, 7:57 PM

of how we're using flytekit? it's just this and it breaks during the fetch. we're using

flytekit==0.23.0

and i've also tried

flytekit==0.25.0

Copy code

remote = FlyteRemote.from_config(config_file_path='flyte.config')
        lp = remote.fetch_launch_plan(project, domain, workflow, version)

Dylan Wilder

04/14/2022, 9:30 PM

hey i think we've tracked this down @Yee to an issue unrelated to our code. imaging this DAG: wf_a -> wf_b -> lp_c where

->

translates to invokes as in wf_b is a subworkflow of a.

Dylan Wilder

04/14/2022, 9:31 PM

from flytectl we can fetch/launch lp_c, wf_b, and wf_a all just fine

Dylan Wilder

04/14/2022, 9:31 PM

from flyteremote we can fetch lp_c and wf_b but not wf_a which fails with a key error on lp_c

Dylan Wilder

04/14/2022, 9:32 PM

if however, we use wf_c instead of lp_c there are no issues

Dylan Wilder

04/14/2022, 9:34 PM

last observation, the chain wf_a -> lp_b -> lp_c works fine 😄

Dylan Wilder

04/14/2022, 9:36 PM

in plain english: workflows can invoke a launch plan fine, but if there is a third level of recursion (workflow invokes workflow invokes launch plan) we run into issues

Yee

04/14/2022, 9:36 PM

sorry was afk…

Dylan Wilder

04/14/2022, 9:38 PM

no worries i just blasted you with data

Yee

04/14/2022, 9:38 PM

just kinda confused.

Dylan Wilder

04/14/2022, 9:39 PM

yea i tried my best over slack, can jump on a quick call maybe

Yee

04/14/2022, 9:39 PM

when you say wf_a, wf_b and lp_c… are these locally defined python objects? or fetched entities? or does it not matter

Yee

04/14/2022, 9:39 PM

yeah i’m down

Yee

04/14/2022, 9:39 PM

let me hop into a room real quick

Yee

04/15/2022, 9:44 PM

i’m gonna add some more tests

Yee

04/15/2022, 9:44 PM

https://github.com/flyteorg/flytekit/pull/957

Yee

04/15/2022, 9:44 PM

but that is the root of the issue

Yee

04/15/2022, 9:45 PM

and from the todo clearly branch nodes don’t quite work yet either, but will submit that as a separate PR.

Dylan Wilder

04/19/2022, 2:58 PM

what version will this be released under?

Yee

04/19/2022, 3:57 PM

not sure yet. probably 0.32.6

165 Views

Open in Slack

Previous Next