#3832 [BUG] Pending kubernetes task pods of cancelled executions are not removed on workflow cancellation Issue created by victorchurikovcommonground Describe the bug I have a workflow that runs multiple @dynamics that call other @tasks, on cancellation of the workflow either from UI or from flytekit FlyteRemote.terminate(execution, reason), the pods that were pending due to being unschedulable remain after the workflow was cancelled. Pending pods remaining after cancellation is a problem because a kubernetes autoscaler (GKE's in this case) still periodically makes attempts to satisfy their scheduling requirements, and if autoscaling conditions are satisfied later, the pod may even trigger a scale up of additional nodes even when it is not desirable because the workflow is cancelled. Expected behavior Cancelling a workflow should clean up leftover pending pods, perhaps by deleting them Additional context to reproduce 1. Create a workflow with tasks that make unsatisfiable scheduling requests (such as high amounts of cpu/mem that is not available) 1.1 If not reproducible, have it call tasks from a @dynamic 2. Execute the workflow and wait for task pods to become Pending 3. Cancel the workflow either from UI or from FlyteRemote.terminate() 4. Observe that the pending pods are not removed and kubernetes still periodically attempts to satisfy their requests (can be seen with kubectl describe pod or kubectl events) Screenshots In this example some pods can be seen Pending 4 hours after their workflow has been cancelled: Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte