https://flyte.org logo
#flytekit
Title
# flytekit
e

Edgar Trujillo

06/30/2022, 1:04 AM
Hi, is it possible to manually hard delete executions through
flytectl
? I have 3 workflow executions that have been stuck in
ABORTING
for the past ~2 hours.
k

Ketan (kumare3)

06/30/2022, 1:45 AM
That is odd, they should never be stuck
Would you be open to helping us debug
e

Edgar Trujillo

06/30/2022, 2:57 AM
Sure, I also believe this was my own doing by running
flytectl delete execution
on a
FAILED
workflow execution...
flytectl delete execution --help
led me to believe calling
delete
on it would be safe.
k

Ketan (kumare3)

06/30/2022, 3:22 AM
Calling delete should be safe
Aah now I see
This is indeed a big
Bug
Cc @katrina I think delete / abort should not abort a failed workflow
@Edgar Trujillo can you please file a bug
And sadly for now the only solution you have is to update the table Phase manually
e

Edgar Trujillo

06/30/2022, 4:06 AM
will do... any docs you can link for the manual update?
k

Ketan (kumare3)

06/30/2022, 4:19 AM
sadly no, we will tell you the name of the table and probably provide a query
k

katrina

07/01/2022, 4:25 PM
yeah that's a great point, we shouldn't allow you to abort an already terminated workflow. adding that check on our end is simple. right now we just assume that an already terminated workflow means the CRD has already been deleted in k8s but this assumption doesn't hold up with finalizers
e

Edgar Trujillo

07/05/2022, 4:55 PM
Any update on this? the 3 wf executions are still stuck on
aborting
and unable to schedule new wf since they are stuck in
waiting
and move to
failed
due to
timeout in node
k

katrina

07/05/2022, 5:01 PM
even if they're misreporting
aborting
the executions should indeed be terminated. this is just a reporting issue (unfortunately)
the timeout looks like a separate issue, are there any logs that point to why that's occuring?
e

Edgar Trujillo

07/05/2022, 5:57 PM
You're right, the timeout issue was due to a node requesting more than the default resource quotas. 🤦‍♂️
k

katrina

07/05/2022, 6:06 PM
ah cool, glad you figured it out! also filed https://github.com/flyteorg/flyte/issues/2665 to keep track of the reporting issue, thanks for bringing this up!
e

Edgar Trujillo

07/05/2022, 6:08 PM
Thx for the help! Feel free to close the issue i created to prevent duplicates: https://github.com/flyteorg/flyte/issues/2653
k

katrina

07/05/2022, 6:16 PM
oh I did not see this, awesome
27 Views