New RunLLM response generated:
>>>*# Question*
Hi everyone!
I have a workflow that has subworkflows - these workflows have maptasks. in some cases: some arraynodes get stuck on "running" status preventing next-in-queue tasks/arraynodes to start despite that the logs show that the process finished.
both the master workflow and subworkflows have FAIL_AFTER_EXECUTABLE_NODES_COMPLETE failure policy
maptasks are set with concurrency=2, min_success_ratio=0.1
Image flytekit version: 1.12.0
cluster flyte version: 1.13.1
any idea on why that happens ?
# Answer
This issue might be related to a known problem where tasks or array nodes appear stuck in a "running" state even though they have completed. This can occur due to eventing issues where the status is not correctly updated in the Flyte console, as noted in
Flyte Issue #5335. Additionally, there have been reports of similar behavior when using map tasks with specific configurations, such as
min_success_ratio
, which can lead to unexpected task states (
Flyte Issue #5201).
Would you like more detailed information on potential solutions or workarounds for this issue?
Sources:
-
Flyte Issue #5335
-
Flyte Issue #5201
Slack Link
https://.slack.com/archives/CP2HDHKE1/p1729588929.510089 👍 0 button 👎 0 button 💡
Hint: Mention @runllm in the thread for followups.