Hey We have this workflow that even though a task failed a s

Question

Hey We have this workflow that even though a task failed a subworkflow node looks like it is still running in flyteconsole thread Anyone seen this before

Sonja Ericsson · Answer

Main execution

Sonja Ericsson · Answer

When I click into the `generate hades uri V0` subworkflow nothing is running

Jay Ganbat · Answer

what is the failure policy of your workflow

Sonja Ericsson · Answer

is there an easy endpoint command to tell I couldn t see anything on ```flytectl get workflow EpisodeJobWorkflow p scum d production version c5632bb2 bdfe 40dd a3ae d8b632bfb849 o yaml``` or ```<https flyte spotify net api v1 workflows scum production EpisodeJobWorkflow c5632bb2 bdfe 40dd a3ae d8b632bfb849>``` metadata is empty

Sonja Ericsson · Answer

```metadata retries retries 5 runtime flavor java type FLYTE SDK version 0 0 1 type java task``` ``` metadata ```

Jay Ganbat · Answer

hmm is it still stuck at running

Jay Ganbat · Answer

i think best place to find the issue is look in the `flytepropeller` log and `flyteadmin` log

Sonja Ericsson · Answer

yes confused

Sonja Ericsson · Answer

```Failed to abort node wpybz n4 Error SystemError system error caused by rpc error code = PermissionDenied desc = Cannot abort an already terminate workflow execution```

Dan Rammer (hamersaw) · Answer

< Sonja Ericsson> do you have access to the FlyteWorkflow CR So eventing to admin is best effort in that if there is a failure the UI may not reflect the actual status of execution during aborts

Dan Rammer (hamersaw) · Answer

The above message suggests that the workflow has already been succesfully aborted and nothing should be currently running Just that the UI was not updated to correctly reflect this

Sonja Ericsson · Answer

yes I have access ```kubectl get fly n scum production``` doesn t give me anything

Sonja Ericsson · Answer

ok so its expected the UI may be incorrect

Dan Rammer (hamersaw) · Answer

i believe the CR name should be the execution ID of the workflow propeller does clean these up after a configurable amount of time and all of the Pods etc that are started as part of the workflow evaluation are linked to the CR using k8s ownership So if the CR has been cleaned up then we can be sure that nothing related to that workflow is running

Dan Rammer (hamersaw) · Answer

do you know what version of flyteadmin propeller you re running i ve tried to repro this a few different ways now and i m unable to find a bug in the UI marking the subworkflow correctly as aborted

Sonja Ericsson · Answer

makes sense it stopped showing those failed to abort message in the logs 10 hours ago

Sonja Ericsson · Answer

we use this version <https github com flyteorg flyte releases tag v1 3 0>

Sonja Ericsson · Answer

` v1 1 62` `FROM ` ``