cc @cool-lifeguard-49380 / @broad-monitor-993

This is due to earlier stable version not supporting sidecars

Btw, I think this might be because they changed defaults

Let me double check that

For PyTorch jobs, it seems like it was always None

@cool-lifeguard-49380 I don't see the clean pod policy being set by the flyte plugin, when using Elastic. And in the latest rc version of the training operator, it defaults to None.

I am looking at the latest stable release here: https://github.com/kubeflow/training-operator/blob/v1.7-branch/pkg/apis/kubeflow.org/v1/pytorch_defaults.go The clean pod policy is set to None there as well.

I installed kubeflow training operator

1.8.0-rc0

and ran this workflow:

from flytekit import task, workflow
from flytekitplugins.kfpytorch import Elastic


@task(
    task_config=Elastic(nnodes=2)
)
def test_cleanup():
    raise Exception("Non recoverable")


@workflow
def wf():
    test_cleanup()

The pod entry point catches the error so the pods are

Completed

, not failed:

NAME                 READY  STATUS   RESTARTS  AGE
f3b52c20cc11c4c6496b-n0-0-worker-0  0/1   Completed  0     46s
f3b52c20cc11c4c6496b-n0-0-worker-1  0/1   Completed  0     46s

The completed pods are sticking around for a bit until e.g. the respective node gets deleted but they are not consuming any resources. I would say this is the expected behaviour. If you want to clean up completed pods I’d recommend configuring flytepropeller to delete the respective k8s resource when an attempt completes:

configmap:
 k8s:
  plugins:
   k8s:
    delete-resource-on-finalize: true

With this, the pytorchjob is immediately deleted which causes the training operator to delete the pods as well. --- I’m now wondering whether my example above is too simple to cause the behaviour you describe. Could you please try what happens for you? If you have a minimal example that I could try, I’m happy to do that.

You could tag me as reviewer 🙏