Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Flyte

Hi on my GCP Flyte deployment Flyte deletes dynamic tasks from k8s as soon as they complete whether they pass or fail. Regular python tasks are deleted after the GC Interval which is 30h on our cluster.
This makes it very hard to debug dynamic tasks if something goes wrong because the pod is no longer in k8s.
Is this something that I can stop Flyte from doing via a config option?

dyn_cleanup_py.py

On my cluster the following script spawns 10 `tsk` tasks as part of a dynamic workflow and all 10 are deleted from k8s as soon as they complete

Hi <@U026D9YKXTL>, sorry for the late response. When you refer to regular python tasks do you mean executing a single task? or executing a a non-dynamic workflow containing only python tasks?

Hi <@U029U35LRDJ> no worries. I’m referring to the latter case. Running a @workflow that contains python tasks. Those are not cleaned up immediately (which is my expectation)

We’ve stopped using `map_tasks`, but I noticed the same behavior with `map_tasks` as the `@dynamic` tasks as well. If that helps troubleshooting.

I can also send my helm values file to show you our setup

values-dev-gke.yaml

<@U026D9YKXTL> thanks for you patience on this. It looks like there is a `delete-resource-on-finalize` that is set on the plugins configmap for flytepropeller. It controls, as you may guess, whether or not k8s resources are deleted when flyte attempts to finalize the task. In your above configuration this may look like:

```configmap:
  k8s:
    plugins:
      k8s:
        default-env-vars:
          - FLYTE_STATSD_HOST: "flytestatsd.datadog.svc.cluster.local"
          - FLYTE_STATSD_DISABLE_TAGS: "True"
        delete-resource-on-finalize: false
        create-container-error-grace-period: 8m0s```
In my testing I was able to not delete dynamic task pods on completion / failure. Let me know if this works for you and / or if you run into any more issues.

I should note that this will not work in map tasks as they do not flow through the propeller plugin manager. We are actively working on a number of map task improvements like individual subtask retries, better flyteconsole support (viewing individual subtask phases / retries), improved parallelism, etc that should be available in the next release.

Hi <@U029U35LRDJ> thanks for getting back to me. I will try running with `delete-resource-on-finalize: false` and hopefully that will resolve the issue