Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-on-gcp
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
integrations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
show-and-tell
sig-large-models
torch-elastic
workflow-building-ui-proj
writing-w-sfloris
Title
v
Victor Gustavo da Silva Oliveira
04/26/2023, 6:52 PMHello all
Can someone help me with a dynamic workflow situation? In local tests it runs OK, but when I upload it to Flyte in my cluster, it queue the tasks but never run them...
The picture shows what's happening.
j
Jay Ganbat
04/26/2023, 7:08 PMwhats your flytepropeller log say
v
Victor Gustavo da Silva Oliveira
04/26/2023, 7:14 PM1: 0: 0: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
1: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
2: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
3: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
4: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
5: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
6: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
7: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
8: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
9: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
10: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
11: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
12: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
13: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
14: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
15: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]
16: InvalidArgument: Invalid fields for event message, caused by [rpc error: code = InvalidArgument desc = missing project]j
Jay Ganbat
04/26/2023, 7:14 PMmissing project ?
maybe registration problem 🤔
how are you calling your workflow in the dynamic task
v
Victor Gustavo da Silva Oliveira
04/26/2023, 7:16 PM@workflow
def main_workflow()-> pd.DataFrame:
...
return dynamic workflowIn local it works just fine
j
Jay Ganbat
04/26/2023, 7:27 PMi think local you dont need that project parameter, maybe its an issue on the back end, you can see your queued workflow registered in the console right
v
Victor Gustavo da Silva Oliveira
04/26/2023, 7:42 PMYup, but it never starts
Other workflows (without dynamic) works just right
j
Jay Ganbat
04/26/2023, 7:58 PMwe saw a similar issue when updating flyte backend
i think we saw it when updating to flytekit 1.3 could be something similar
k
Kevin Su
04/26/2023, 10:07 PMwhat version of propeller and flytekit you are using? mind sharing the code snippet
v
Victor Gustavo da Silva Oliveira
04/27/2023, 11:37 AMWe've recently upgraded my flyte environment to 1.3... Now we are using everything in v1.3.0.
I've shared the code in this thread, earlier
If you need more info, I'm here 🙂
If you notice in image, the type is showing as
unknown{6 items
config:{}0 items
container:{7 items
args:[18 items
0:"pyflyte-execute"
1:"--inputs"
2:"{{.input}}"
3:"--output-prefix"
4:"{{.outputPrefix}}"
5:"--raw-output-data-prefix"
6:"{{.rawOutputDataPrefix}}"
7:"--checkpoint-path"
8:"{{.checkpointOutputPrefix}}"
9:"--prev-checkpoint"
10:"{{.prevCheckpointPrefix}}"
11:"--resolver"
12:"flytekit.core.python_auto_container.default_task_resolver"
13:"--"
14:"task-module"
15:"src.features.workflow.workflow"
16:"task-name"
17:"features_workflow"
]
command:[]0 items
config:[]0 items
env:[1 item
0:{2 items
key:"_F_SS_C"
value:""
}
]
image:""
ports:[]0 items
resources:{2 items
limits:[2 items
0:{2 items
name:1
value:"1"
}
1:{2 items
name:3
value:"2Gi"
}
]
requests:[]0 items
}
}
id:{5 items
domain:"development"
name:"src.features.workflow.workflow.features_workflow"
project:"flyte-tasks"
resourceType:1
version:"a1df825"
}
interface:{2 items
inputs:{1 item
variables:{2 items
activities_dataset:{2 items
description:"activities_dataset"
type:{1 item
structuredDatasetType:{1 item
columns:[]0 items
}
}
}
dataset:{2 items
description:"dataset"
type:{1 item
structuredDatasetType:{1 item
columns:[]0 items
}
}
}
}
}
outputs:{1 item
variables:{1 item
o0:{2 items
description:"o0"
type:{1 item
structuredDatasetType:{1 item
columns:[]0 items
}
}
}
}
}
}
metadata:{2 items
retries:{}0 items
runtime:{3 items
flavor:"python"
type:1
version:"1.5.0"
}
}
type:"python-task"
}e
Eduardo Apolinario (eapolinario)
05/08/2023, 6:15 PM@Victor Gustavo da Silva Oliveira, flytekit 1.3.0 (and 1.3.1) had an issue with dynamic workflows. This was fixed in 1.3.2 and beyond. More details about the issue can be found here.
v
Victor Gustavo da Silva Oliveira
05/10/2023, 1:30 PMThose are great news! Thanks @Eduardo Apolinario (eapolinario)
@Eduardo Apolinario (eapolinario) I've updated to Flyte 1.8.0 using flyte-core for kubernetes, and the problem persists... Is there something that I need to do to make it work?
I'm using flytekit 1.6.1 to register tasks
e
Eduardo Apolinario (eapolinario)
05/18/2023, 3:34 PM@Victor Gustavo da Silva Oliveira, I'm confused, can you clarify what you mean by Flyte 1.8.0? I can hop on a call to debug this
v
Victor Gustavo da Silva Oliveira
05/18/2023, 4:13 PM@Eduardo Apolinario (eapolinario) I'm using the version available on GitHub (Flyteconsole 1.8.0, and the other components in the versions in
values.yaml