https://flyte.org logo
#ask-the-community
Title
# ask-the-community
m

MUhwjR9VF3K2QJhL2VvNGoFY

08/12/2023, 2:34 AM
Is there some some way to kill a workflow in “Aborting” state? Flyte propeller has been looping over it for multiple days. I’ve tried deleting the CRD directly but that seems to hang.
k

Ketan (kumare3)

08/12/2023, 3:40 AM
You have to remove the finalizer- but before you do that, can you share with us either some interesting logs or if any problem that you see
They should only happen if flyteadmin has deleted the workflow, or for some reason the status propagation failed
m

MUhwjR9VF3K2QJhL2VvNGoFY

08/12/2023, 3:45 AM
workflow was aborted from the console, but flytepropeller is still dealing with it
k

Ketan (kumare3)

08/12/2023, 3:46 AM
Surr but what’s the error message
m

MUhwjR9VF3K2QJhL2VvNGoFY

08/12/2023, 3:51 AM
This one is popular:
{"json":{"exec_id":"execution","node":"n0/dn6/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn5","ns":"namespace","res_ver":"39998306","routine":"worker-0","wf":"flyte_workflow"},"level":"info","msg":"Finalizing dynamic workflow RetryAttempt [0]","ts":"2023-08-12T03:48:54Z"}
And then lots of variations of this:
Request failed due to [rpc error: code = AlreadyExists desc = have already recorded task execution phase ABORTED
I would guess status propagation is still ongoing
k

Ketan (kumare3)

08/12/2023, 3:55 AM
What version are you using
This looks like a fixed bug
m

MUhwjR9VF3K2QJhL2VvNGoFY

08/12/2023, 3:55 AM
Right now 1.9.0 but this has been going on since Wednesday
k

Ketan (kumare3)

08/12/2023, 3:56 AM
Hmm cc @Dan Rammer (hamersaw) this should be ignored in events
Seems like replay of abort is missing
You can remove the finalizer for now
m

MUhwjR9VF3K2QJhL2VvNGoFY

08/12/2023, 3:59 AM
Cool that fixed it, thanks
I could grab/sanitize more logs if you think that would be helpful but they look pretty repetitive
k

Ketan (kumare3)

08/12/2023, 4:20 AM
Ya I think I see the issue, but will have to check the code if this is really possible
16 Views