https://flyte.org logo
j

James Evers

09/21/2022, 4:54 PM
hi everyone, i'm having some trouble pulling images on a sandbox deployment. everything works fine with the base image (the one used when i started the deployment with
flytectl sandbox start --source .
), but when i try to build another image in the sandbox (
flytectl sandbox exec -- <docker build statement
), the pod fails to pull the image even though i can see that its been successfully build via
flytectl sandbox exec -- docker image ls
. anyone have any idea where i might be going wrong?
y

Yee

09/21/2022, 5:51 PM
I feel like i’ve seen this recently - can you get the -o yaml dump for the pod
j

James Evers

09/21/2022, 5:51 PM
yep, just a sec
y

Yee

09/21/2022, 5:52 PM
and grep -i imagepullpolicy
that might not matter, just curious
j

James Evers

09/21/2022, 5:57 PM
Copy code
imagePullPolicy: Always
y

Yee

09/21/2022, 5:58 PM
can you run
flytectl sandbox exec -- docker pull <image>
i’m hoping that fails.
j

James Evers

09/21/2022, 5:58 PM
ah it does fail!
Copy code
Error response from daemon: repository <http://cr.flyte.org/flyteorg/tensorflow-kit|cr.flyte.org/flyteorg/tensorflow-kit> not found: name unknown: repository name not known to registry
y

Yee

09/21/2022, 5:59 PM
that’s the image you built?
j

James Evers

09/21/2022, 5:59 PM
yep, i also tried building it without the
<http://cr.flyte.org/flyteorg|cr.flyte.org/flyteorg>
prefix
y

Yee

09/21/2022, 6:00 PM
oh yeah you should probably just build locally without any registry
behind the scenes it assumed dockerhub, but yeah irrelevant
um
let me do some more digging around here.
j

James Evers

09/21/2022, 6:01 PM
just as a sanity check i'll try building it with the tag
tensorflow-kit:latest
actually, one thing that is popping up now that might be helpful is this error when i try to register the workflow:
executing:
Copy code
pyflyte register --project flytesnacks --domain development --version v1.4 --image tensorflow-kit:latest src
throws:
Copy code
AssertionError: Image Config with name tensorflow not found in the configuration
y

Yee

09/21/2022, 6:29 PM
did you start this with flytectl sandbox? or flytectl demo
j

James Evers

09/21/2022, 6:29 PM
flytectl sandbox start --source .
y

Yee

09/21/2022, 6:30 PM
oh it’s in the first message sorry
j

James Evers

09/21/2022, 6:30 PM
np!
i think i may not fully understand the relationship between some of the flyteadmin commands and the sandbox
y

Yee

09/21/2022, 6:31 PM
mm
j

James Evers

09/21/2022, 6:31 PM
in this case, i have two images: 1. a base/default image 2. an image with some tensorflow stuff in it
y

Yee

09/21/2022, 6:31 PM
can you name your image something other than latest?
the version
:tst123
j

James Evers

09/21/2022, 6:32 PM
yep sure
ah interesting, let me try
while im waiting for the image to build -- how does
pyflyte register
use the image passed to that command?
for instance, if i have a deployment running on a default/base
image a
, but i want to register a workflow that will run on some specialized
image b
, i should register the workflow with
--image image-b
, right?
y

Yee

09/21/2022, 6:36 PM
currently it’s “fast register” only. so that means the image is set in the task’s image field. separately, your code is zipped up locally and uploaded to admin (which writes it to minio). at run time, your code is downloaded from minio unzipped and then task execution runs
j

James Evers

09/21/2022, 6:37 PM
oooo
y

Yee

09/21/2022, 6:38 PM
here
download this guy: https://blog.flyte.org/flyte-cheat-sheet the pdf
top right corner
multi-image workflows… that is how you can construct workflows with tasks that use two different images
though that doesn’t tell you the registration command
j

James Evers

09/21/2022, 6:39 PM
yeah i had been looking at that one
another thing im a little confused about here
y

Yee

09/21/2022, 6:40 PM
the registration command is
Copy code
--image mydefaultimage:cafe123 --image other=nondefault:beef
j

James Evers

09/21/2022, 6:40 PM
ahhhhhh
okay that makes sense
so if my
flyte.config
file has
Copy code
[images]
tensorflow = tensorflow-kit:v1
my register command should be:
Copy code
pyflyte register --image default:v1 tensorflow=tensorflow-kit:tst123
y

Yee

09/21/2022, 6:42 PM
delete flyte.config
well you can delete it at least
if you don’t delete it then you need to specify it on the command line.
if you don’t specify it on the command line then
pyflyte --config xyz …
j

James Evers

09/21/2022, 6:44 PM
okay i just got the workflow to register with your suggestion
thanks for the help!!
2 Views