Hi all I've managed to get a local demo cluster working on my old laptop with workflows running etc, but I've followed the exact same steps on my new one and I'm getting the following error whenever I run a workflow. Note, I've tried the cookbook hello_world example and the greeting workflow, but both give the same error. I think I'm doing something stupid, but I just can't figure it out.

containers with unready status: [f0de017c01c7140d8a4c-n0-0]|Back-off pulling image "<http://cr.flyte.org/flyteorg/flytekit:py3.9-1.5.0|cr.flyte.org/flyteorg/flytekit:py3.9-1.5.0>"

Any ideas?

Interesting, can you try docker pull cr.flyte.org/flyteorg/flytekit:py3.9-1.5.0

Works for me.

@brief-fish-13206 seems like a proxy or something else

So docker pull works fine, but in the container logs I see this

2023-04-14 14:27:17 E0414 13:27:17.174653      73 kuberuntime_image.go:51] "Failed to pull image" err="rpc error: code = Unknown desc = failed to pull and unpack image \"<http://ghcr.io/flyteorg/flytekit:py3.9-latest|ghcr.io/flyteorg/flytekit:py3.9-latest>\": failed to resolve reference \"<http://ghcr.io/flyteorg/flytekit:py3.9-latest|ghcr.io/flyteorg/flytekit:py3.9-latest>\": failed to do request: Head \"<https://ghcr.io/v2/flyteorg/flytekit/manifests/py3.9-latest>\": x509: certificate signed by unknown authority" image="<http://ghcr.io/flyteorg/flytekit:py3.9-latest|ghcr.io/flyteorg/flytekit:py3.9-latest>"

i cant repro in a fresh sandbox. @brief-fish-13206: this probably sounds dumb, but can you restart your docker daemon and fire up a new sandbox? i definitely had weird issues in older docker desktop for mac versions, but havent in awhile.

Restarting didn't work, but kubectl did. I'm going to raise a ticket with Docker. Just so I can give them all the details. Is the sandbox puling the image a Docker-in-Docker, or is it pulling from within Kubernetes?

it’s running k3s on containerd within the sandbox container.

Cc @mammoth-librarian-92821 looks like a similar problem

Thanks, I've raised a ticket with Docker, let's see what they say

it’s not a private repo though.

Docker gave up on helping me. I'm going to raise a bug with K3s. Can you guys help me with this question please (in the context of the sandbox)? Cluster Configuration: <!-- Provide some basic information on the cluster configuration. For example, "3 servers, 2 agents". -->

@brief-fish-13206 i’d love to spend some time digging into this with you. Will DM.

@brief-fish-13206 this is what I was saying like a proxy / firewall on your machine

proposal here: https://github.com/flyteorg/flyte/pull/3609

Yeah you were right. I wonder if it’s down to the Mac chip docker version. Everything was the same on my old Mac (intel chip) and it worked fine.

cool. if its not merged by then, I'll trigger a separate build that you can test with.

Yep, that's working! It did take a long time (3 mins to do the greeting wf) though. Don't know if it's related to the change or not

Ok good to know it’s working. The image is running a nightly version of Flyte.

Thanks for your help again 🙂

It might be because of an initial image pull. Is it faster if you run it again?

Yep, just took 14secs instead of 3 mins

Ok. Was likely just pulling down the image.

Soon it will be lower after the next release