Hi, is it possible to manually hard delete executions through

flytectl

? I have 3 workflow executions that have been stuck in

ABORTING

for the past ~2 hours.

That is odd, they should never be stuck

Sure, I also believe this was my own doing by running

flytectl delete execution

on a

FAILED

workflow execution...

flytectl delete execution --help

led me to believe calling

delete

on it would be safe.

Calling delete should be safe

will do... any docs you can link for the manual update?

sadly no, we will tell you the name of the table and probably provide a query

yeah that's a great point, we shouldn't allow you to abort an already terminated workflow. adding that check on our end is simple. right now we just assume that an already terminated workflow means the CRD has already been deleted in k8s but this assumption doesn't hold up with finalizers

Any update on this? the 3 wf executions are still stuck on

aborting

and unable to schedule new wf since they are stuck in

waiting

and move to

failed

due to

timeout in node

even if they're misreporting

aborting

the executions should indeed be terminated. this is just a reporting issue (unfortunately)

You're right, the timeout issue was due to a node requesting more than the default resource quotas. 🤦‍♂️

ah cool, glad you figured it out! also filed https://github.com/flyteorg/flyte/issues/2665 to keep track of the reporting issue, thanks for bringing this up!

Thx for the help! Feel free to close the issue i created to prevent duplicates: https://github.com/flyteorg/flyte/issues/2653

oh I did not see this, awesome