Hi on my GCP Flyte deployment Flyte deletes dynamic tasks from k8s as soon as they complete whether they pass or fail. Regular python tasks are deleted after the GC Interval which is 30h on our cluster. This makes it very hard to debug dynamic tasks if something goes wrong because the pod is no longer in k8s. Is this something that I can stop Flyte from doing via a config option?

@Dan Rammer (hamersaw) any ideas?

Hi @Nicholas LoFaso, sorry for the late response. When you refer to regular python tasks do you mean executing a single task? or executing a a non-dynamic workflow containing only python tasks?

Hi @Dan Rammer (hamersaw) no worries. I’m referring to the latter case. Running a @workflow that contains python tasks. Those are not cleaned up immediately (which is my expectation)

@Nicholas LoFaso thanks for you patience on this. It looks like there is a

delete-resource-on-finalize

that is set on the plugins configmap for flytepropeller. It controls, as you may guess, whether or not k8s resources are deleted when flyte attempts to finalize the task. In your above configuration this may look like:

configmap:
  k8s:
    plugins:
      k8s:
        default-env-vars:
          - FLYTE_STATSD_HOST: "flytestatsd.datadog.svc.cluster.local"
          - FLYTE_STATSD_DISABLE_TAGS: "True"
        delete-resource-on-finalize: false
        create-container-error-grace-period: 8m0s

In my testing I was able to not delete dynamic task pods on completion / failure. Let me know if this works for you and / or if you run into any more issues.

Hi @Dan Rammer (hamersaw) thanks for getting back to me. I will try running with

delete-resource-on-finalize: false

and hopefully that will resolve the issue