Is there some some way to kill a workflow in “Abor...
# flyte-supportm
melodic-potato-82122
08/12/2023, 2:34 AMIs there some some way to kill a workflow in “Aborting” state? Flyte propeller has been looping over it for multiple days. I’ve tried deleting the CRD directly but that seems to hang.
f
freezing-airport-6809
08/12/2023, 3:40 AM
You have to remove the finalizer- but before you do that, can you share with us either some interesting logs or if any problem that you see
freezing-airport-6809
08/12/2023, 3:40 AM
They should only happen if flyteadmin has deleted the workflow, or for some reason the status propagation failed
m
melodic-potato-82122
08/12/2023, 3:45 AMworkflow was aborted from the console, but flytepropeller is still dealing with it
f
freezing-airport-6809
08/12/2023, 3:46 AM
Surr but what’s the error message
m
melodic-potato-82122
08/12/2023, 3:51 AMThis one is popular:
{"json":{"exec_id":"execution","node":"n0/dn6/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn5","ns":"namespace","res_ver":"39998306","routine":"worker-0","wf":"flyte_workflow"},"level":"info","msg":"Finalizing dynamic workflow RetryAttempt [0]","ts":"2023-08-12T03:48:54Z"}melodic-potato-82122
08/12/2023, 3:53 AMAnd then lots of variations of this:
Request failed due to [rpc error: code = AlreadyExists desc = have already recorded task execution phase ABORTEDmelodic-potato-82122
08/12/2023, 3:54 AMI would guess status propagation is still ongoing
f
freezing-airport-6809
08/12/2023, 3:55 AM
What version are you using
freezing-airport-6809
08/12/2023, 3:55 AM
This looks like a fixed bug
m
melodic-potato-82122
08/12/2023, 3:55 AMRight now 1.9.0 but this has been going on since Wednesday
f
freezing-airport-6809
08/12/2023, 3:56 AM
Hmm cc @hallowed-mouse-14616 this should be ignored in events
freezing-airport-6809
08/12/2023, 3:56 AM
Seems like replay of abort is missing
freezing-airport-6809
08/12/2023, 3:57 AM
You can remove the finalizer for now
m
melodic-potato-82122
08/12/2023, 3:59 AMCool that fixed it, thanks
melodic-potato-82122
08/12/2023, 4:01 AMI could grab/sanitize more logs if you think that would be helpful but they look pretty repetitive
f
freezing-airport-6809
08/12/2023, 4:20 AM
Ya I think I see the issue, but will have to check the code if this is really possible
66 Views