Hey all I m creating a workflow using `flytekit approve` The

Question

Hey all I m creating a workflow using `flytekit approve` The workflow runs fine on my native local machine but when I try to register the workflow I get the following error ``` raise InactiveRpcError state pytype disable=not instantiable grpc channel InactiveRpcError lt InactiveRpcError of RPC that terminated with status = StatusCode INTERNAL details = failed to compile workflow for resource type WORKFLOW project myproject domain workflows name myproject workflows pytorch training model training and approval workflow version DwDJxBGH4EDglOXa ZZwOA== with err failed to compile workflow with err Collected Errors 1 Error 0 Code VariableNameNotFound Node Id n1 Description Variable o0 not found on node n1 debug error string = UNKNOWN Error received from peer created time 2023 05 09T17 24 07 713904 07 00 grpc status 13 grpc message failed to compile workflow for resource type WORKFLOW project myproject domain workflows name myproject workflows pytorch training model training and approval workflow version DwDJxBGH4EDglOXa ZZwOA== with err failed to compile workflow with err Collected Errors 1 n tError 0 Code VariableNameNotFound Node Id n1 Description Variable o0 not found on node n1 n gt ``` Any idea what s causing this

Kevin Su · Answer

seems like n1 task doesn t have output could you share code snippet

joe · Answer

yeah the workflow looks roughly like ``` workflow def model training and approval workflow my args int model scores mlflow run = model training workflow my args = my args flytekit approve upstream item=model scores name= model score review timeout=timedelta seconds=300 model staging mlflow run=mlflow run ```

Kevin Su · Answer

could you share the task code also

joe · Answer

I was able to reproduce it with a paired down workflow ``` task def mytask gt int return 1 workflow def simple workflow model scores = mytask flytekit approve upstream item=model scores name= model score review timeout=timedelta seconds=300 ```

Yee · Answer

this works for me ```$ cat core approve sample py from datetime import timedelta from flytekit import task workflow approve task def mytask gt int return 1 workflow def simple workflow model scores = mytask approve upstream item=model scores name= model score review timeout=timedelta seconds=300 ```

Yee · Answer

what version is your backend on

joe · Answer

See image below from the flyte console and we re using version `1 3 0` for the flyte helm release

joe · Answer

I bumped up the helm release to 1 5 0 and getting the same error

justin hallquist · Answer

< Yee> for clarity it worked against a remote cluster when you ran it i m seeing the same issue running the above simple example against my eks cluster deployment

justin hallquist · Answer

works locally fine tho

Yee · Answer

yeah no i was testing on our reference implementation

Yee · Answer

it s on EKS

Yee · Answer

can you look at logs on the backed maybe

Yee · Answer

admin

justin hallquist · Answer

ya one sec

justin hallquist · Answer

seems to be the similiar output and the rest of the logs look standard ``` json src workflow manager go 107 wf taunty myworkflow level debug msg Failed to compile workflow with id resource type WORKFLOW project taunty domain workflows name taunty myworkflow version dnoVpjSXY87EJpY fIcyzg== with err failed to compile workflow with err Collected Errors 1 n tError 0 Code VariableNameNotFound Node Id n1 Description Variable o0 not found on node n1 n ts 2023 05 10T21 29 28Z ```

Yee · Answer

can you kubectl get the flyteadmin or flyte binary pod o yaml | grep image

justin hallquist · Answer

```<http cr flyte org flyteorg flyteadmin v1 1 72|cr flyte org flyteorg flyteadmin v1 1 72>```

joe · Answer

bumping the images to 1 6 0 fixed the issue Thanks all