Hi all, I'm using Flyte in GCP with GCSFuse CSI driver. The driver mounts a sidecar container, which waits for all other containers to terminate before terminating itself. The problem is that Flyte Copilot also does the same - waits for all other containers to terminate, which causes a deadlock between the two where they wait on each other to terminate. I had a workaround for that where I explicitly sent a SIGTERM to the GCSFuse sidecar, which terminated it. (For some reason it worked only 90% of the time) But after upgrading my GKE cluster to 1.29, the GCSFuse sidecar is injected as an init-container with

restartPolicy: Always

, which means my workaround doesn't work anymore. Does anyone know how can I fix it easily? Our pipelines are now stuck because of this, as it is impossible to downgrade the GKE cluster Thanks!

Do you need to use copilot?

yes, we use raw containers, flytekit isn't good enough

Can you explain more please

We use many different docker images that we had before moving to Flyte. I had to change them to work with Flyte, but basically our Flyte workflow is built mostly of

ContainerTask

Ohh so flytekit is not the problem raw containers tasks help in migration

You're saying it's not related to copilot? Right now I tried creating my own copilot image, specifically ignoring the GCSFuse sidecar, which seems to work and everything terminates (although now I get an error regarding the container task outputs, but might be unrelated)

This is done by the engine

I did end up fixing it by making a new copilot image ignoring the GCSFuse sidecar in its watcher although of course this isn't maintainable and I'd rather have a fix coming from you guys