Is there some some way to kill a workflow in “Abor...
# ask-the-communitym
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 2:34 AMIs there some some way to kill a workflow in “Aborting” state? Flyte propeller has been looping over it for multiple days. I’ve tried deleting the CRD directly but that seems to hang.
k
Ketan (kumare3)
08/12/2023, 3:40 AM
You have to remove the finalizer- but before you do that, can you share with us either some interesting logs or if any problem that you see
Ketan (kumare3)
08/12/2023, 3:40 AM
They should only happen if flyteadmin has deleted the workflow, or for some reason the status propagation failed
m
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:45 AMworkflow was aborted from the console, but flytepropeller is still dealing with it
k
Ketan (kumare3)
08/12/2023, 3:46 AM
Surr but what’s the error message
m
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:51 AMThis one is popular:
{"json":{"exec_id":"execution","node":"n0/dn6/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn5","ns":"namespace","res_ver":"39998306","routine":"worker-0","wf":"flyte_workflow"},"level":"info","msg":"Finalizing dynamic workflow RetryAttempt [0]","ts":"2023-08-12T03:48:54Z"}MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:53 AMAnd then lots of variations of this:
Request failed due to [rpc error: code = AlreadyExists desc = have already recorded task execution phase ABORTEDMUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:54 AMI would guess status propagation is still ongoing
k
Ketan (kumare3)
08/12/2023, 3:55 AM
What version are you using
Ketan (kumare3)
08/12/2023, 3:55 AM
This looks like a fixed bug
m
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:55 AMRight now 1.9.0 but this has been going on since Wednesday
k
Ketan (kumare3)
08/12/2023, 3:56 AM
Hmm cc @Dan Rammer (hamersaw) this should be ignored in events
Ketan (kumare3)
08/12/2023, 3:56 AM
Seems like replay of abort is missing
Ketan (kumare3)
08/12/2023, 3:57 AM
You can remove the finalizer for now
m
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 3:59 AMCool that fixed it, thanks
MUhwjR9VF3K2QJhL2VvNGoFY
08/12/2023, 4:01 AMI could grab/sanitize more logs if you think that would be helpful but they look pretty repetitive
k
Ketan (kumare3)
08/12/2023, 4:20 AM
Ya I think I see the issue, but will have to check the code if this is really possible
24 Views