Hey! We have this workflow that even though a task...
# ask-the-communitys
Sonja Ericsson
03/14/2023, 5:26 PMHey! We have this workflow that even though a task failed, a subworkflow node looks like it is still running in flyteconsole 🧵 Anyone seen this before?
Sonja Ericsson
03/14/2023, 5:26 PMMain execution:
Sonja Ericsson
03/14/2023, 5:27 PM.
Sonja Ericsson
03/14/2023, 5:27 PMWhen I click into the
generate-hades-uri-V0j
Jay Ganbat
03/14/2023, 7:00 PMwhat is the failure policy of your workflow
s
Sonja Ericsson
03/14/2023, 8:23 PMis there an easy endpoint/command to tell? I couldn’t see anything on
Copy code
flytectl get workflow EpisodeJobWorkflow -p scum -d production --version c5632bb2-bdfe-40dd-a3ae-d8b632bfb849 -o yaml Copy code
<https://flyte.spotify.net/api/v1/workflows/scum/production/EpisodeJobWorkflow/c5632bb2-bdfe-40dd-a3ae-d8b632bfb849>Sonja Ericsson
03/14/2023, 8:24 PM Copy code
metadata:
retries:
retries: 5
runtime:
flavor: java
type: FLYTE_SDK
version: 0.0.1
type: java-task Copy code
"metadata": {},j
Jay Ganbat
03/14/2023, 8:39 PMhmm is it still stuck at running?
Jay Ganbat
03/14/2023, 8:40 PMi think best place to find the issue is look in the
flytepropellerflyteadmins
Sonja Ericsson
03/14/2023, 8:43 PMyes 😕
Sonja Ericsson
03/14/2023, 8:46 PM Copy code
Failed to abort node [wpybz-n4]. Error: [SystemError] system error, caused by: rpc error: code = PermissionDenied desc = Cannot abort an already terminate workflow executiond
Dan Rammer (hamersaw)
03/14/2023, 8:48 PM@Sonja Ericsson do you have access to the FlyteWorkflow CR? So eventing to admin is best-effort in that if there is a failure the UI may not reflect the actual status of execution during aborts.
Dan Rammer (hamersaw)
03/14/2023, 8:48 PMThe above message suggests that the workflow has already been succesfully aborted and nothing should be currently running. Just that the UI was not updated to correctly reflect this.
s
Sonja Ericsson
03/14/2023, 9:01 PMyes I have access
Copy code
kubectl get fly -n scum-productionSonja Ericsson
03/14/2023, 9:01 PMok, so its expected the UI may be incorrect?
d
Dan Rammer (hamersaw)
03/14/2023, 9:03 PMi believe the CR name should be the execution ID of the workflow. propeller does clean these up after a configurable amount of time, and all of the Pods, etc that are started as part of the workflow evaluation are linked to the CR using k8s ownership. So if the CR has been cleaned up, then we can be sure that nothing related to that workflow is running.
Dan Rammer (hamersaw)
03/14/2023, 9:04 PMdo you know what version of flyteadmin / propeller you're running? i've tried to repro this a few different ways now, and i'm unable to find a bug in the UI marking the subworkflow correctly as "aborted".
s
Sonja Ericsson
03/14/2023, 9:08 PMmakes sense it stopped showing those failed to abort message in the logs 10 hours ago
Sonja Ericsson
03/14/2023, 9:08 PMwe use this version https://github.com/flyteorg/flyte/releases/tag/v1.3.0
Sonja Ericsson
03/14/2023, 9:09 PM<http://github.com/flyteorg/flytepropeller|github.com/flyteorg/flytepropeller> v1.1.62FROM <http://ghcr.io/flyteorg/flyteadmin:v1.1.70|ghcr.io/flyteorg/flyteadmin:v1.1.70><http://ghcr.io/flyteorg/flyteconsole:v1.4.1|ghcr.io/flyteorg/flyteconsole:v1.4.1>153 Views