Hey! We have this workflow that even though a task...
# flyte-supportc
colossal-solstice-11091
03/14/2023, 5:26 PMHey! We have this workflow that even though a task failed, a subworkflow node looks like it is still running in flyteconsole 🧵 Anyone seen this before?
colossal-solstice-11091
03/14/2023, 5:26 PMMain execution:
colossal-solstice-11091
03/14/2023, 5:27 PM.
colossal-solstice-11091
03/14/2023, 5:27 PMWhen I click into the
generate-hades-uri-V0m
magnificent-teacher-86590
03/14/2023, 7:00 PMwhat is the failure policy of your workflow
c
colossal-solstice-11091
03/14/2023, 8:23 PMis there an easy endpoint/command to tell? I couldn’t see anything on
Copy code
flytectl get workflow EpisodeJobWorkflow -p scum -d production --version c5632bb2-bdfe-40dd-a3ae-d8b632bfb849 -o yaml Copy code
<https://flyte.spotify.net/api/v1/workflows/scum/production/EpisodeJobWorkflow/c5632bb2-bdfe-40dd-a3ae-d8b632bfb849>colossal-solstice-11091
03/14/2023, 8:24 PM Copy code
metadata:
retries:
retries: 5
runtime:
flavor: java
type: FLYTE_SDK
version: 0.0.1
type: java-task Copy code
"metadata": {},m
magnificent-teacher-86590
03/14/2023, 8:39 PMhmm is it still stuck at running?
magnificent-teacher-86590
03/14/2023, 8:40 PMi think best place to find the issue is look in the
flytepropellerflyteadminc
colossal-solstice-11091
03/14/2023, 8:43 PMyes 😕
colossal-solstice-11091
03/14/2023, 8:46 PM Copy code
Failed to abort node [wpybz-n4]. Error: [SystemError] system error, caused by: rpc error: code = PermissionDenied desc = Cannot abort an already terminate workflow executionh
hallowed-mouse-14616
03/14/2023, 8:48 PM@colossal-solstice-11091 do you have access to the FlyteWorkflow CR? So eventing to admin is best-effort in that if there is a failure the UI may not reflect the actual status of execution during aborts.
hallowed-mouse-14616
03/14/2023, 8:48 PMThe above message suggests that the workflow has already been succesfully aborted and nothing should be currently running. Just that the UI was not updated to correctly reflect this.
c
colossal-solstice-11091
03/14/2023, 9:01 PMyes I have access
Copy code
kubectl get fly -n scum-productioncolossal-solstice-11091
03/14/2023, 9:01 PMok, so its expected the UI may be incorrect?
h
hallowed-mouse-14616
03/14/2023, 9:03 PMi believe the CR name should be the execution ID of the workflow. propeller does clean these up after a configurable amount of time, and all of the Pods, etc that are started as part of the workflow evaluation are linked to the CR using k8s ownership. So if the CR has been cleaned up, then we can be sure that nothing related to that workflow is running.
👍 1
hallowed-mouse-14616
03/14/2023, 9:04 PMdo you know what version of flyteadmin / propeller you're running? i've tried to repro this a few different ways now, and i'm unable to find a bug in the UI marking the subworkflow correctly as "aborted".
c
colossal-solstice-11091
03/14/2023, 9:08 PMmakes sense it stopped showing those failed to abort message in the logs 10 hours ago
colossal-solstice-11091
03/14/2023, 9:08 PMwe use this version https://github.com/flyteorg/flyte/releases/tag/v1.3.0
colossal-solstice-11091
03/14/2023, 9:09 PM<http://github.com/flyteorg/flytepropeller|github.com/flyteorg/flytepropeller> v1.1.62FROM <http://ghcr.io/flyteorg/flyteadmin:v1.1.70|ghcr.io/flyteorg/flyteadmin:v1.1.70><http://ghcr.io/flyteorg/flyteconsole:v1.4.1|ghcr.io/flyteorg/flyteconsole:v1.4.1>153 Views