Hey all I m having a little trouble running a workflow in th

Question

Hey all I m having a little trouble running a workflow in the Flyte sandbox on my local machine in particular the workflow that I m attempting to run is failing to pull the image that I ve built within the sandbox Here you can see the containers that I have running on my host ```$ docker ps gt gt gt CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES dbf8f5dcb150 <http cr flyte org flyteorg flyte sandbox dind bfa1dd4e6057b6fc16272579d61df7b1832b96a7|cr flyte org flyteorg flyte sandbox dind bfa1dd4e6057b6fc16272579d61df7b1832b96a7> tini flyte entrypoi About an hour ago Up About an hour 0 0 0 0 30081 30082 gt 30081 30082 tcp 0 0 0 0 30084 gt 30084 tcp 2375 2376 tcp 0 0 0 0 30086 30088 gt 30086 30088 tcp flyte sandbox``` From which we can then find the images that exist inside the `dbf8f5dcb150` container ```$ docker exec it dbf8f5dcb150 docker image ls gt gt gt REPOSITORY TAG IMAGE ID CREATED SIZE papermill exploration latest 3c40c6deb126 23 minutes ago 948MB ``` I can see my project in there under the tag `papermill exploration latest` I then serialize and submit my workflow spec as follows ```pyflyte pkgs workflows package f image papermill exploration latest flytectl register files project flytesnacks domain development archive flyte package tgz version v2``` All of which works ```$ flytectl get workflows project flytesnacks domain development gt gt gt | VERSION | NAME | CREATED AT | | v2 | workflows workflow nb to python wf | 2022 12 12T12 41 53 987960Z | | v1 | workflows workflow nb to python wf | 2022 12 12T12 33 08 295661Z | 2 rows``` I then attempt to invoke the workflow but the resulting pod cannot pull the image ```$ flytectl get execution project flytesnacks domain development azlfqvzfsbz4lr8pbmlt gt gt gt | NAME | LAUNCH PLAN NAME | TYPE | PHASE | SCHEDULED TIME | STARTED | ELAPSED TIME | ABORT DATA TRUNC | ERROR DATA TRUNC | | azlfqvzfsbz4lr8pbmlt | workflows workflow nb to python wf | LAUNCH PLAN | FAILED | | 2022 12 12T13 07 23 548693519Z | 23 161600293s | | 1 1 currentAttempt done Last Error USER containers | | | | | | | | | | with unready status azlfqvzfsbz4lr8pbmlt n | 1 rows $ docker exec it dbf8f5dcb150 kubectl n flytesnacks development describe pod azlfqvzfsbz4lr8pbmlt n0 0 gt gt gt Events Type Reason Age From Message Normal Scheduled 27m default scheduler Successfully assigned flytesnacks development azlfqvzfsbz4lr8pbmlt n0 0 to dbf8f5dcb150 Normal Pulling 25m x4 over 27m kubelet Pulling image papermill exploration latest Warning Failed 25m x4 over 27m kubelet Failed to pull image papermill exploration latest rpc error code = Unknown desc = Error response from daemon pull access denied for papermill exploration repository does not exist or may require docker login denied requested access to the resource is denied Warning Failed 25m x4 over 27m kubelet Error ErrImagePull Warning Failed 25m x6 over 27m kubelet Error ImagePullBackOff Normal BackOff 2m22s x106 over 27m kubelet Back off pulling image papermill exploration latest ``` Have I missed something here Are the pods not authenticated against the docker repo Or am I not specifying my images correctly

Samhita Alla · Answer

Have you built the docker image using `flytectl sandbox exec docker build tag papermill exploration latest ` command

Tom Stokes · Answer

Thanks for the response Yeah I did that s why you can see the `papermill exploration latest` image inside the container running k3s I think that I ve found the issue StackOverflow message suggests that when using the ` docker` flag with `k3s` the host docker is used rather than `containerd` you can see this argument in the In that instance the image will be found if the `imagePullPolicy` is set to `IfNotPresent` However the default pods in the sandbox are run using the `Always` policy I ve fixed this by configuring a default pod template as per the applying that and then restarting the propellor component which then uses the correct policy and fixes the problem ```$ docker exec i $K3S CONTAINER ID kubectl n flyte n flytesnacks development get pod ap4xl5hwmwmgnkwm4spz n0 0 o yaml | yq spec containers 0 imagePullPolicy gt gt gt IfNotPresent``` where the template looks as follows ```apiVersion v1 kind PodTemplate metadata name default pod template template spec containers name default image overwrite me imagePullPolicy IfNotPresent```

Tom Stokes · Answer

Whilst I m happy to have fixed it thanks for the great documentation it seems strange to me that this doesn t work out of the box Can I raise the issue somewhere to see if this can be fixed Happy to contribute a PR as well if that s an option

Samhita Alla · Answer

Of course Please feel free to file an issue This used to work but not sure what changed < Eduardo Apolinario eapolinario > can Tom create a PR to fix this issue

Yee · Answer

teammate and i were looking through the code when he rememberd we don t set it this is controlled by k8s and the default k8s behavior is that if your image tag is ` latest ` then it will set the pull policy to Always

Yee · Answer

if you just tag it with anything else it should work without the pod template

Tom Stokes · Answer

Ah good catch Will give this a go instead Thanks both

Ingo Kemmerzell · Answer

We have a similar problem I haven t built an image we just want to try out the Getting started example I guess the problem is caused by our proxy Is there an option to pass the HTTP PROXY NO PROXY env variables ```Events Type Reason Age From Message Normal Scheduled 4m40s default scheduler Successfully assigned flytesnacks development fca72778d3e6f4c5c8ab n0 0 to bc1ca7e70b3c Normal Pulling 3m21s x4 over 4m40s kubelet Pulling image Warning Failed 3m21s x4 over 4m40s kubelet Failed to pull image rpc error code = Unknown desc = failed to pull and unpack image failed to resolve reference failed to do request Head dial tcp lookup on 53 no such host Warning Failed 3m21s x4 over 4m40s kubelet Error ErrImagePull Warning Failed 2m56s x6 over 4m39s kubelet Error ImagePullBackOff Normal BackOff 2m43s x7 over 4m39s kubelet Back off pulling image ```

Samhita Alla · Answer

When you are spinning up a demo cluster you can send env variables `flytectl demo start env HTTP PROXY= ` Let me know if this works

Ingo Kemmerzell · Answer

Thank you Samhita Unfortunately it doesn t solve my problem The env variables are passed to the docker container `docker exec flyte sandbox env` But now I m getting a different error message which shows the certificate of is not accepted ```Events Type Reason Age From Message Normal Scheduled 18m default scheduler Successfully assigned flytesnacks development axpsj4sv5xsj6z2krc57 n0 0 to d3482e31a88c Normal Pulling 16m x4 over 18m kubelet Pulling image Warning Failed 16m x4 over 18m kubelet Failed to pull image rpc error code = Unknown desc = failed to pull and unpack image failed to resolve reference failed to do request Head x509 certificate signed by unknown authority Warning Failed 16m x4 over 18m kubelet Error ErrImagePull Warning Failed 16m x6 over 17m kubelet Error ImagePullBackOff Normal BackOff 2m59s x65 over 17m kubelet Back off pulling image ```

Samhita Alla · Answer

Have you tried any of the solutions posted in <https stackoverflow com questions 50768317 docker pull certificate signed by unknown authority> thread

Ingo Kemmerzell · Answer

The container doesn t include docker or podman I m using crictl instead I don t have problems to pull images from but throws the shown error message On the host system I can use podman or crictl to pull images from Most of the utilities in the SO article are not available in the container

Samhita Alla · Answer

< Yee> < Eduardo Apolinario eapolinario > any idea how to resolve this issue

Eduardo Apolinario (eapolinario) · Answer

< Ingo Kemmerzell> just to confirm how are you starting the sandbox Can you share the exact command you used Setting env vars via ` env` should be enough for k3s really containerd to pick them up

Ingo Kemmerzell · Answer

The env options were successfully passed to k3s which caused a different error message x509 unknown authority instead of dial tcp lookup on 53 no such host I could pull images with crictl in the container from but received the x509 error message when trying to connect to In the meantime I ve submitted an internal ticket to our proxy team to check the proxy and IPS settings I guess they have changed something because now everything seems to work fine I can submit the workflow examples without getting error messages < Samhita Alla> < Eduardo Apolinario eapolinario > thank you very much for your support much appreciated