New RunLLM response generated:
>>>*# Question*
Hi. I have some Flyte workflow executions that use mapped tasks. Sometimes these executions finish and show ‘SUCCESS’, but at the same time when you look at detailed task status, some task instances still show as ‘Running’. Looking at K8 pod status, all pods are complete. One particular nature of these workflows is that, max task parallelism is smaller than the number of task instances.
# Answer
This issue where Flyte workflow executions show 'SUCCESS' but some task instances still display as 'Running' while all Kubernetes pods are complete can occur due to a delay in updating the node status in the Flyte UI. This could be related to FlytePropeller not processing the nodes promptly or a bottleneck in FlytePropeller's processing.
According to
Flyte discussion, this can happen when there is a delay in updating the node status in the UI. Additionally, if max parallelism is smaller than the number of task instances, it can cause tasks to be queued, leading to inconsistencies in status reporting.
Would you like more detailed steps on how to investigate this issue further or potential configurations to mitigate it?
Sources:
-
Flyte discussion on task status delay
-
Flyte discussion on task status delay
Slack Link
https://flyte-org.slack.com/archives/CP2HDHKE1/p1720057051.041349 👍 0 button 👎 0 button 💡
Hint: Mention @runllm in the thread for followups.