New RunLLM response generated >>> Question Hi We have deploy Flyte #runllm-flyte-trial

New RunLLM response generated: >>>*# Question* Hi...

ancient-wolf-19325

08/28/2024, 2:47 PM

New RunLLM response generated: >>>*# Question* Hi ! We have deployed Flyte in our GCP project and we are using Flyte version 1.13.0 at backend/server. Meanwhile we were using

flytekit==1.11.0

flyteidl==1.11.0

and

flytekitplugins-pod==1.11.0

packages on local machine to submit the job remotely. Everythig was working fine without any issues. Recently when I update the above mentioned python packages from

1.11.0

1.13.0

then I am getting the following error when I try to submit the job:

Copy code

FAILED ci/e2e-tests/hello_world/test_hello_world.py::test_workflow_hello_wf - flytekit.exceptions.user.FlyteInvalidInputException: USER:BadInputToAPI: error=None, cause=<_InactiveRpcError of RPC that terminated with:
	status = StatusCode.INVALID_ARGUMENT
	details = "failed to compile workflow for [resource_type:WORKFLOW project:"playground" domain:"dev" name:"hello.hello_wf" version:"c2d8jj"] with err failed to compile workflow with err Collected Errors: 1
	Error 0: Code: TaskReferenceNotFound, Node Id: start-node, Description: Referenced Task [resource_type:TASK name:"hello.hello_world" version:"c2d8jj"] not found.
"
	debug_error_string = "UNKNOWN:Error received from peer  {grpc_message:"failed to compile workflow for [resource_type:WORKFLOW project:\"playground\" domain:\"dev\" name:\"hello.hello_wf\" version:\"c2d8jj\"] with err failed to compile workflow with err Collected Errors: 1\n\tError 0: Code: TaskReferenceNotFound, Node Id: start-node, Description: Referenced Task [resource_type:TASK name:\"hello.hello_world\" version:\"c2d8jj\"] not found.\n", grpc_status:3, created_time:"2024-08-28T14:11:52.293396625+00:00"}"

ancient-wolf-19325

08/28/2024, 2:47 PM

>>

So, everything is working fine if I am using the python packages of version

1.11.0

and if I upgrade to

1.13.0 or 1.13.4

then it didn't work. Are there any compatibility issues with other packages that I should keep in mind to upgrade the flytekit and flyteidl to higher version ? What could be cause of this error ? # Answer The error you're encountering,

TaskReferenceNotFound

, indicates that the task

hello.hello_world

with version

c2d8jj

is not found in the Flyte backend. This issue can arise due to several reasons, especially when upgrading Flytekit and FlyteIDL versions. Potential Causes and Solutions: 1. Task Registration: - Ensure that the tasks are registered correctly with the new version. When upgrading Flytekit and FlyteIDL, the task definitions might change, requiring re-registration. - Use

flytectl

to register the tasks and workflows again after upgrading the packages. 2. Version Mismatch: - Ensure that the versions of Flytekit, FlyteIDL, and Flytekit plugins are compatible with the Flyte backend version. According to Flyte documentation, version mismatches can lead to such issues. 3. Dependency Conflicts: - Check for any dependency conflicts that might have arisen due to the upgrade. Sometimes, upgrading one package might require upgrading other related packages to maintain compatibility. 4. Configuration Changes: - Verify if there are any configuration changes required for the new versions. Sometimes, new versions introduce changes that need to be reflected in the configuration files. Steps to Resolve: 1. Re-register Tasks and Workflows:

Copy code

flytectl register files --project playground --domain dev --archive flyte-workflow-package.tgz --config /root/flyte_config.yaml --version c2d8jj

2. Check Compatibility: - Ensure that the Flyte backend version (1.13.0) is compatible with Flytekit and FlyteIDL versions (1.13.0). Refer to the Flyte documentation for compatibility details. 3. Verify Dependencies: - Use

pip list

to check the installed versions of Flytekit, FlyteIDL, and related plugins. Ensure they are all on version 1.13.0. Would you like more detailed steps or further assistance? Sources: - Flyte GitHub Issue - Flyte documentation Slack Link https://flyte-org.slack.com/archives/CP2HDHKE1/p1724856411.760609 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

Open in Slack

Previous Next