Hey all. I'm back with a quick question. I have a situation where a workflow fans out to bunch of nodes, 100+, at some point we're hitting the image pull qps limit and since it takes too long for the pod to start, flyte marks the node as failed and nukes the workflow. So my question is, is there a config I can set to have flyte wait longer for the pod to spin up before giving up?

cc @hallowed-mouse-14616

@wooden-airline-96737 do you know what the specific error is?

ImagePullBackoff

? or something like

task active timeout [%s] expired

?

Sure, it's an imagepullbackoff error when pull QPS exceeded. What I see in Flyte is:

[1/1] currentAttempt done. Last Error: USER::[1/1] currentAttempt done. Last Error: USER::containers with unready status: [primary]|Back-off pulling image ...

At this point this node has failed and the whole worfklow failed. But the actual pod will eventually start and do whatever it was supposed to do.

Yeah, so we're fixing the Pod deletion on

ImagePullBackoff

error in this PR. Basically, if Flyte fails a task because of

ImagePullBackoff

it will then delete the Pod. This was particularly troublesome when the

ImagePullBackoff

was because the image did not exist - then the Pod would never start (and therefore never complete) and just stick around taking up resources.

Interesting, thanks for the link, will follow. If I do retry won't I be basically spawning multiple pods for the same task?

Currently yes, when the linked PR is merged, the first attempt (that failed) will be aborted (so Pod deleted) and then a new Pod will be created for the 2nd attempt. I think the proposal for a grace period configuration is a cleaner solution, but given the lower priority it would likely need to come as a community contribution.

Excellent, thanks!

I don't believe increasing that configuration will currently do anything to fix this issue. I brought it up because we could add another configuration value for something like "image-pull-backoff-grace-period" that is applied similarly. If this is something you would like to see (or better yet, contribute!) feel free to create a github issue and we can track it.

Aha, ok, makes sense.

We think on our end the issue was related to kubelet

registryPullQPS

and

registryBurst

being set too low.