Channels
#ask-the-community
Title
# ask-the-community
m
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 2:34 AMIs there some some way to kill a workflow in “Aborting” state? Flyte propeller has been looping over it for multiple days. I’ve tried deleting the CRD directly but that seems to hang.
k
Ketan (kumare3)
08/12/2023, 3:40 AM
You have to remove the finalizer- but before you do that, can you share with us either some interesting logs or if any problem that you see
They should only happen if flyteadmin has deleted the workflow, or for some reason the status propagation failed
m
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:45 AMworkflow was aborted from the console, but flytepropeller is still dealing with it
k
Ketan (kumare3)
08/12/2023, 3:46 AM
Surr but what’s the error message
m
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:51 AMThis one is popular:
{"json":{"exec_id":"execution","node":"n0/dn6/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn5","ns":"namespace","res_ver":"39998306","routine":"worker-0","wf":"flyte_workflow"},"level":"info","msg":"Finalizing dynamic workflow RetryAttempt [0]","ts":"2023-08-12T03:48:54Z"}And then lots of variations of this:
Request failed due to [rpc error: code = AlreadyExists desc = have already recorded task execution phase ABORTEDI would guess status propagation is still ongoing
k
Ketan (kumare3)
08/12/2023, 3:55 AM
What version are you using
This looks like a fixed bug
m
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:55 AMRight now 1.9.0 but this has been going on since Wednesday
k
Ketan (kumare3)
08/12/2023, 3:56 AM
Hmm cc @Dan Rammer (hamersaw) this should be ignored in events
Seems like replay of abort is missing
You can remove the finalizer for now
m
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:59 AMCool that fixed it, thanks
I could grab/sanitize more logs if you think that would be helpful but they look pretty repetitive
k
Ketan (kumare3)
08/12/2023, 4:20 AM
Ya I think I see the issue, but will have to check the code if this is really possible
5 Views