Hey all, I'm creating a workflow using `flytekit.a...
# ask-the-communityj
joe
05/10/2023, 12:28 AMHey all, I'm creating a workflow using
flytekit.approve Copy code
raise _InactiveRpcError(state) # pytype: disable=not-instantiable
grpc._channel._InactiveRpcError: <_InactiveRpcError of RPC that terminated with:
status = StatusCode.INTERNAL
details = "failed to compile workflow for [resource_type:WORKFLOW project:"myproject" domain:"workflows" name:"myproject.workflows.pytorch_training.model_training_and_approval_workflow" version:"DwDJxBGH4EDglOXa-ZZwOA==" ] with err failed to compile workflow with err Collected Errors: 1
Error 0: Code: VariableNameNotFound, Node Id: n1, Description: Variable [o0] not found on node [n1].
"
debug_error_string = "UNKNOWN:Error received from peer {created_time:"2023-05-09T17:24:07.713904-07:00", grpc_status:13, grpc_message:"failed to compile workflow for [resource_type:WORKFLOW project:\"myproject\" domain:\"workflows\" name:\"myproject.workflows.pytorch_training.model_training_and_approval_workflow\" version:\"DwDJxBGH4EDglOXa-ZZwOA==\" ] with err failed to compile workflow with err Collected Errors: 1\n\tError 0: Code: VariableNameNotFound, Node Id: n1, Description: Variable [o0] not found on node [n1].\n"}"
>k
Kevin Su
05/10/2023, 12:29 AMseems like n1 task doesn’t have output. could you share code snippet?
j
joe
05/10/2023, 12:30 AMyeah, the workflow looks roughly like:
Copy code
@workflow
def model_training_and_approval_workflow(
my_args: int
):
model_scores, mlflow_run = model_training_workflow(my_args = my_args)
flytekit.approve(upstream_item=model_scores,name="model_score_review",timeout=timedelta(seconds=300),)
model_staging(mlflow_run=mlflow_run)k
Kevin Su
05/10/2023, 12:31 AMcould you share the @task code also
j
joe
05/10/2023, 12:44 AMI was able to reproduce it with a paired down workflow:
Copy code
@task
def mytask() -> int:
return 1
@workflow
def simple_workflow(
):
model_scores = mytask()
flytekit.approve(
upstream_item=model_scores,
name="model_score_review",
timeout=timedelta(seconds=300),
)y
Yee
05/10/2023, 1:00 AM
this works for me
Copy code
$ cat core/approve_sample.py
from datetime import timedelta
from flytekit import task, workflow, approve
@task
def mytask() -> int:
return 1
@workflow
def simple_workflow(
):
model_scores = mytask()
approve(
upstream_item=model_scores,
name="model_score_review",
timeout=timedelta(seconds=300),
)Yee
05/10/2023, 1:01 AM
what version is your backend on?
j
joe
05/10/2023, 1:48 AMSee image below from the flyte console and we're using version
1.3.0joe
05/10/2023, 1:54 AMI bumped up the helm release to 1.5.0 and getting the same error
j
justin hallquist
05/10/2023, 9:33 PM@Yee for clarity, it worked against a remote cluster when you ran it?
i'm seeing the same issue running the above simple example against my eks cluster deployment
justin hallquist
05/10/2023, 9:33 PMworks locally fine tho
y
Yee
05/10/2023, 9:39 PM
yeah no i was testing on our reference implementation
Yee
05/10/2023, 9:39 PM
it’s on EKS
Yee
05/10/2023, 9:39 PM
can you look at logs on the backed maybe?
Yee
05/10/2023, 9:39 PM
admin
j
justin hallquist
05/10/2023, 9:43 PMya one sec
justin hallquist
05/10/2023, 9:45 PMseems to be the similiar output and the rest of the logs look standard:
Copy code
{
"json": {
"src": "workflow_manager.go:107",
"wf": "taunty.myworkflow"
},
"level": "debug",
"msg": "Failed to compile workflow with id [resource_type:WORKFLOW project:\"taunty\" domain:\"workflows\" name:\"taunty.myworkflow\" version:\"dnoVpjSXY87EJpY-fIcyzg==\" ] with err failed to compile workflow with err Collected Errors: 1\n\tError 0: Code: VariableNameNotFound, Node Id: n1, Description: Variable [o0] not found on node [n1].\n",
"ts": "2023-05-10T21:29:28Z"
}y
Yee
05/10/2023, 9:47 PM
can you kubectl get the flyteadmin or flyte-binary pod -o yaml | grep image
j
justin hallquist
05/10/2023, 9:48 PM Copy code
<http://cr.flyte.org/flyteorg/flyteadmin:v1.1.72|cr.flyte.org/flyteorg/flyteadmin:v1.1.72>j
joe
05/13/2023, 1:48 AMbumping the images to 1.6.0 fixed the issue. Thanks all!
151 Views