Hello, recently our flyte cluster (

v1.12.0

) has been throwing some intermittent error while starting task as our workload increase where they are many pod starting at the same time. This happen once every 1-2 days, which isn't that bad but still breaks our workflow requiring manual intervention on alerts. Retries doesn't seem to work, because the task are not even started.

Grace period [3m0s] exceeded|containers with unready status: [f6140adc65c3a4d47000-n1-0]|failed to reserve container name "f6140adc65c3a4d47000-n1-0...": name "f6140adc65c3a4d47000-n1-0..." is reserved for "189f5dbee3fbc4a26ff7a619fcea34ad..."

Grace period [0s] exceeded|containers with unready status: [f9fd2612618924083000-n2-0]|failed to sync secret cache: timed out waiting for the condition

So far, we have increased the grace period for

config.K8sPluginConfig

create-container-config-error-grace-period: 0s
create-container-error-grace-period: 3m0s

Beside upgrade the cluster resource for the load, are there other config that we can tweak to improve this or to have retries kicking in? Thanks!

Cc @white-painting-22485 fyi (on gcp) @broad-train-34581 this is on gke right? The k8s cluster is having some problem, starting pods. @freezing-boots-56761 do you remember this

Yes gke

Cc @high-park-82026 also

yes i remember this. this is the relevant PR: https://github.com/flyteorg/flyte/commit/355d383d27dab5b8c11067cae9848d172ffad12f we had it set to 10m (bursting to ~10k pods, ~1.5k nodes). almost 3 years ago but IIRC, basically the issue is that the creation of the container by the runtime takes too long. flyte used to see this error and either marked the task as failed or recreated the pod, adding further to the load on the cluster/nodes. just waiting a bit to let it auto-resolve worked best for us. i’m a bit surprised this error STILL exists though lol

lol gke, but I think image streaming may solve it

this is after the image is fetched and the container is started iirc. the name conflicts happen because the context deadline is reached while waiting for the container to come up and the runtime recreates the container, but the previous one is still coming up and didn’t actually fail 😬

yes after setting to 10mins for

create-container-error-grace-period: 3m0s

, we almost never seen this the error

failed to reserve container name

anymore for 3 days so far. today there was just one

failed to sync secret cache

and I ended up increasing

create-container-config-error-grace-period: 0s

Is this g visor error

just checking here to see if i missed anything or if this is known

We have run at 10 higher scale on AWS and never seen it or even on Azure

will do, thanks folks!

@broad-train-34581 please keep us in the loop

Yeah I've never seen this too and we run some heavy loads.

we didn’t have gvisor on, just the custom gcp container runtime running on the COS images for us.

@broad-train-34581 shameless plug, but in union we have a new system that can reuse containers, so dramatically speeds up workflows to run in milliseconds and does not create so many pods 😊 . We would love to work with you