https://flyte.org logo
#announcements
Title
# announcements
n

Nicholas LoFaso

01/31/2022, 6:00 PM
Hi on my GCP Flyte deployment Flyte deletes dynamic tasks from k8s as soon as they complete whether they pass or fail. Regular python tasks are deleted after the GC Interval which is 30h on our cluster. This makes it very hard to debug dynamic tasks if something goes wrong because the pod is no longer in k8s. Is this something that I can stop Flyte from doing via a config option?
On my cluster the following script spawns 10
tsk
tasks as part of a dynamic workflow and all 10 are deleted from k8s as soon as they complete
y

Yee

01/31/2022, 7:58 PM
@Dan Rammer (hamersaw) any ideas?
👀 1
d

Dan Rammer (hamersaw)

02/01/2022, 1:21 PM
Hi @Nicholas LoFaso, sorry for the late response. When you refer to regular python tasks do you mean executing a single task? or executing a a non-dynamic workflow containing only python tasks?
n

Nicholas LoFaso

02/01/2022, 1:23 PM
Hi @Dan Rammer (hamersaw) no worries. I’m referring to the latter case. Running a @workflow that contains python tasks. Those are not cleaned up immediately (which is my expectation)
We’ve stopped using
map_tasks
, but I noticed the same behavior with
map_tasks
as the
@dynamic
tasks as well. If that helps troubleshooting.
I can also send my helm values file to show you our setup
d

Dan Rammer (hamersaw)

02/03/2022, 3:59 PM
@Nicholas LoFaso thanks for you patience on this. It looks like there is a
delete-resource-on-finalize
that is set on the plugins configmap for flytepropeller. It controls, as you may guess, whether or not k8s resources are deleted when flyte attempts to finalize the task. In your above configuration this may look like:
Copy code
configmap:
  k8s:
    plugins:
      k8s:
        default-env-vars:
          - FLYTE_STATSD_HOST: "flytestatsd.datadog.svc.cluster.local"
          - FLYTE_STATSD_DISABLE_TAGS: "True"
        delete-resource-on-finalize: false
        create-container-error-grace-period: 8m0s
In my testing I was able to not delete dynamic task pods on completion / failure. Let me know if this works for you and / or if you run into any more issues.
I should note that this will not work in map tasks as they do not flow through the propeller plugin manager. We are actively working on a number of map task improvements like individual subtask retries, better flyteconsole support (viewing individual subtask phases / retries), improved parallelism, etc that should be available in the next release.
n

Nicholas LoFaso

02/04/2022, 2:43 PM
Hi @Dan Rammer (hamersaw) thanks for getting back to me. I will try running with
delete-resource-on-finalize: false
and hopefully that will resolve the issue
🙏 1
31 Views