elegant-toddler-67101
07/19/2023, 2:19 PMkind-kite-58745
07/19/2023, 5:10 PMgc-interval
which is 30 minutes by default. It attempts to periodically delete FlyteWorkflow CRDs from the cluster, but this alone does not clean up pods
There’s also the delete-resource-on-finalize
option, which cleans up resources related to the FlyteWorkflow when it is terminated based on the FlyteWorkflow’s finalizer. The docs say:
Instructs the system to delete the resource upon successful execution of a k8s pod rather than have the k8s garbage collector clean it up. This ensures that no resources are kept around (potentially consuming cluster resources). This, however, will cause k8s log links to expire as soon as the resource is finalized.
If this option frees cluster resources and causes logs to disappear, then it seems to be deleting pods and their containers, which is what we want to achieve.
What’s not clear to me from this description is if it cleans up a pod after the pod completes, or after the workflow completes. The name of “delete-resource-on-finalize” seems to suggest it happens when the finalizer is activated on the FlyteWorkflow, but the description says it happens “upon successful execution of a k8s pod”. Either of those should be fine, let’s try configuring these and see if it works.
You can configure these in the helm values under configmap.schedulerConfig.scheduler
, like this:
configmap:
schedulerConfig:
gc-interval: 10m #can leave blank for 30m
delete-resource-on-finalize: true
I’ll have a chance to test this tomorrow so I’ll let you know if it actually works, meanwhile you can try it yourselfaverage-finland-92144
07/19/2023, 6:40 PM