I'm not sure if my question belongs here or in #CREL4QVAQ ... I'm seeing map-tasks commonly fail with the error:

[85]: code:"ContainersNotReady|ImagePullBackOff" message:"containers with unready status: [f0ff4b2cb0d6d44d1907-n1-0-dn0-0-dn10-0-85]|Back-off pulling image \"(sanitized).<http://dkr.ecr.us-east-1.amazonaws.com/flyte-plaster:23.5.19\%22%22|dkr.ecr.us-east-1.amazonaws.com/flyte-plaster:23.5.19\"">

This is part of a workflow that employs a mappable task for one section. The workflow will complete successfully maybe half of the time. The other half, this error occurs specifically for the map task. I can see from logs that some elements of the mapped task are executing successfully, and in this case it is number 85 (out of 100) that failed. These are all executing on the same node (this is by design for mapped tasks, yes?), and they all use the same image, which is clearly already on the node since other elements from the mapped task "list" are executing. Edit: The above was incorrect. I misread/misunderstood the "node" aspect: a map task runs within a single "workflow node", not a single computer-node (e.g. ec2-instance). In fact, the 100 elements of the map task are being executed across 5 different machines. Still, the puzzle remains (see the reply) - the task that supposedly failed due to "ContainersNotRead|ImagePullBackoff" was in fact running when the job failed. So how could the container not be ready, or the image need pulling? Thoughts?

When you register the tasks, which image tag are you using? If it is not exaclty equals to an image in ECR, you recieve this error

Hi @dry-ability-69144 - we use a single image for all tasks, so when a new image is built, a new registration pass is done for all tasks/workflows. This workflow described above works about half the time, so I don't think it is a mismatch in image tag spec, otherwise it would never work.

We're seeing this exact behavior as well with a single custom image. We've seen it with dynamic tasks what spin up a bunch of child tasks as well as in map task. Did you happen to identify any root causes here?

We have not, I'm hoping this will catch the attention of some Union engineers. 🙂. I also continue to see this -- just as you report - in map tasks, and also sometimes in @dynamic workflows that execute a number (e.g. 15-20) of "child" tasks.

Is this for map tasks only?

No, as indicated above, we both see it for non-map tasks as well.

We have not really seen this, but let me cc @hallowed-mouse-14616 on this, he is actually completely reworking maptasks in the back - its a massive refactor and makes map tasks extremely powerful - goal 1: Make it the same (just less buggy) goal 2: Make it more powerful - extend the paradigm to many other task types etc

@microscopic-furniture-57275 can you provide a little more information? Specifically: (1) which logs are you referring to that indicate the task is executing? k8s? flyte? and do you have the messages? (2) do you have the pod yaml dump from k8s? this would help immensely. We handle transitioning k8s pod state to a flyte phase in this function. Basically, there is a mapping of values. What I'm wondering is if there is something like an

ImagePullBackoff

(as indicated by your message), then Flyte marks the task as a retryable failure, then between that time and when Flyte actually cleans up the failed task it starts. The 20 second runtime seems suspect, but it could be the case. If this is the case we could explore adding a grace period configuration option similar to the CreateContainerError. The idea is that Flyte wouldn't immediately fail, rather it would ensure that N seconds have passed since beginning the task.

The image pullback off could be because of throttling - @hallowed-mouse-14616 - great point. I think you might be right

@hallowed-mouse-14616 thanks very much for your thoughtful reply. I'm on a deadline for some unrelated work, so may not be able to provide this information for a few days -- but we need to solve this and I will absolutely get back to you with more logs and investigation results. Your theory sounds plausible and a grace period sounds like a good solution in this case. Also note that @crooked-artist-67935 reported seeing the same things as I have, so perhaps Sam can as well provide more log/repro info. I'll need to try to repro to get the log information you want -- I went back to the execution in the Flyte Console that I ref'd in the OP, but whereas previously the maptask could be unfolded, and I could see execution/log info on each element of the mapped task, it now only shows a summary of the tasks - I'm not sure if this is some UI change or this is behavior after a task is much older and the pods no longer exist. Or that I'm misremembering. 🙂. More later.

@microscopic-furniture-57275 do you know what version of flytepropeller you're running. There was an issue, that I fixed, where Flyte didn't correctly clean up

ImagePullBackoff

resources. So the task would fail, but Flyte would leave the

Pod

until completion. I'm wondering if this is the issue here.

@hallowed-mouse-14616 we are using the flyte binary backend, which to my understanding has the propeller component integrated? The flyte-binary version as of those logs above was 1.5.0, which we've since bumped to 1.6.2 a few weeks ago. I haven't made tests since the last upgrade.

which to my understanding has the propeller component integrated

correct. the single binary has all of the components bundled.

which we've since bumped to 1.6.2

so it is likely be that this fix was not in the previous deployment.

ok, let me do some testing with our current versions and create a simple repro case if I see the problem still...

We are certainly going to add a grace period for the

ImagePullBackoff

failure as well (issue pending). Because right now, the first time Flyte sees the

ImagePullBackoff

it will fail the task, deleting the Pod, and then if retries are configured launch a new Pod. It probably makes more sense to wait a configurable amount of time on

ImagePullBackoff

before declaring a task a failure.

Hey @hallowed-mouse-14616, I did a test with flytekit 1.7 and I still see the failing map-tasks issue that actually end up running successfully. See attached image indicating two failed map tasks (out of 100) and see logging from one on these that Flyte reports as failed (though it actually completed successfully). It seems Flyte is seeing the initial ImagePull error and marking it as failed, but a retry causes the image to get fetched and the task runs to completion. But my workflow aborted as soon as Flyte decided one of them failed. 😞 I'm not sure what the right answer here is: this could be fixed either by Flyte waiting a bit before calling it failed, or (from what I read) by us bumping up our

registryPullQPS

and/or

registryBurst

config params for

kubelet

to fix the initial "Pull QPS exceeded" which seems to be the root cause.

Did you just upgrade flytekit? The mentioned fix will require flytepropeller to be updated.

We are running v1.6.2 of the flyte single binary.

re ^ I'll get a PR submitted by tomorrow. will update this thread accordingly.

Great, thanks for the update!

just created an issue - https://github.com/flyteorg/flyte/issues/3843

Thanks @hallowed-mouse-14616! I'll look for this to show up in a subsequent release. Presumably this will be a release of flyte-propeller, in our case bundled in the single flyte binary, since I'd assume all logic related to marking pods as failed exists in that backend code.

Exactly, so this was merged in propeller in the v1.1.105 release which should be included in the next flyte single binary release.

Sorry all, I was out last week. It seems like there is a solution proposed above, which is great. On our end we tuned some k8s params to decrease the amount of node consolidation that was happening and increased our EC2 pull limits which seems to have alleviated the issue for now.

@hallowed-mouse-14616 Above you mention that your grace period for image-pull timeouts will be included in the next flyte single binary release. I've looked around and can't seem to find a release history (or version numbers I understand) for the single-binary. I found the page that documents how to install it via a Helm chart, and the README.md in the helm charts folder refers to various version numbers, including

Version 0.1.10

,

AppVersion 1.16.0

and

flyteagent.deployment.image.tag

version of

1.62b1

, among others. I'm not finding any clear-cut "release/version history" for the flyte-binary as I might expect. Can you shed any light on this? I'm trying to understand how to look out for what you refer to above, the "next flyte single binary release". Thanks!