https://flyte.org
Join Slack
Is there some some way to kill a workflow in “Aborting” state? Flyte propeller has been looping over...
# flyte-support
m
Is there some some way to kill a workflow in “Aborting” state? Flyte propeller has been looping over it for multiple days. I’ve tried deleting the CRD directly but that seems to hang.
f
You have to remove the finalizer- but before you do that, can you share with us either some interesting logs or if any problem that you see
They should only happen if flyteadmin has deleted the workflow, or for some reason the status propagation failed
m
workflow was aborted from the console, but flytepropeller is still dealing with it
f
Surr but what’s the error message
m
This one is popular: 
{"json":{"exec_id":"execution","node":"n0/dn6/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn9/dn0/dn5","ns":"namespace","res_ver":"39998306","routine":"worker-0","wf":"flyte_workflow"},"level":"info","msg":"Finalizing dynamic workflow RetryAttempt [0]","ts":"2023-08-12T03:48:54Z"}
And then lots of variations of this: 
Request failed due to [rpc error: code = AlreadyExists desc = have already recorded task execution phase ABORTED
I would guess status propagation is still ongoing
f
What version are you using
This looks like a fixed bug
m
Right now 1.9.0 but this has been going on since Wednesday
f
Hmm cc @hallowed-mouse-14616 this should be ignored in events
Seems like replay of abort is missing
You can remove the finalizer for now
m
Cool that fixed it, thanks
I could grab/sanitize more logs if you think that would be helpful but they look pretty repetitive
f
Ya I think I see the issue, but will have to check the code if this is really possible
86 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by