wave Hi Flyte community I am seeing a weird behavior with p Flyte #flyte-support

:wave: Hi Flyte community!! I am seeing a weird be...

worried-iron-1001

11/05/2024, 10:19 PM

👋 Hi Flyte community!! I am seeing a weird behavior with pyflyte register with --non-fast. It says

Successfully registered 5 entities

but actually skips to register Tasks. On workflow execution it uses old task code. Is this a known issue? Do we know the fix?

Copy code

Successfully serialized 5 flyte objects
[DEBUG][selector_events.py:54] Using selector: EpollSelector
[x] Registration <task_name> type TASK skipped!
[✔] Registration <wf_name> type WORKFLOW successful with version 18c812f241-dirty
[✔] Registration <wf_name> type LAUNCH_PLAN successful with version 18c812f241-dirty
[✔] Registration <lp_name> type LAUNCH_PLAN successful with version 18c812f241-dirty
[✔] Registration <lp_name> type LAUNCH_PLAN successful with version 18c812f241-dirty
Successfully registered 5 entities

Thanks in advance!!

worried-iron-1001

11/06/2024, 12:49 AM

Copy code

Request rejected by the API, due to Invalid input.
RPC Failed, with Status: StatusCode.INVALID_ARGUMENT
	Details: <task_name> task with different structure already exists:

After removing

--skip-errors

, above it the actual error. I am not sure why I am seeing this. This is just registration of task with different docker image.

freezing-airport-6809

11/06/2024, 1:14 AM

really odd

worried-iron-1001

11/06/2024, 1:27 AM

I did some more debugging. If the version string is same, then Task registration is failing after second run. But, then why only Task fails? Seems like a bug in pyflyte.

Copy code

pyflyte register --non-fast --image "${FLYTE_WORKFLOW_IMAGE}" --version "${COMMIT_HASH}" "${FLYTE_BINARY_DIR}"

worried-iron-1001

11/06/2024, 1:30 AM

I think we need to pass unique uuid as version everytime. Not sure if this is expected behavior.

freezing-airport-6809

11/06/2024, 4:21 AM

Can you send the code snippet

freezing-airport-6809

11/06/2024, 4:21 AM

It should has the code to generate a new version

worried-iron-1001

11/12/2024, 4:18 AM

COMMIT_HASH

is the issues. It was static string, hence seeing the error. I append current timestamp which fixed the issue. But

Successfully registered 5 entities

was misleading. If it registers 4, then message should explicitly say

Successfully registered 4 entities

. Also, if registering will same version is not allowed, then registration should also fail for Launchplans, & workflows and not just tasks.

29 Views

Open in Slack

Previous Next