Hi, unionml community! I'm trying to run the MNIST: Digits Classification Tutorial in my local Flyte cluster, but I'm running into issues. Specifically, I hope to train, run batch inference, and serve the model via FastAPI within my local Flyte Sandbox. I followed these steps: 1. I set up a local sandbox environment by running

flytectl demo start

, 2. I ensured my

~/.flyte/config.yaml

is pointed at my local environment. 3. I started a unionml project by running

unionml init mnist

and navigating into the directory. 4. I ran

unionml deploy app:model

to deploy a Model object named "model" in the app.py file within the current working directory. 5. I ran

unionml train app:model

to train the model. When I deploy the model, I see these logs:

[unionml] deploying app:model
2022-09-04 19:09:54,211 unionml INFO Building docker container in flyte demo cluster.
2022-09-04 19:09:54,862 unionml INFO unable to prepare context: unable to evaluate symlinks in Dockerfile path: lstat /root/Dockerfile: no such file or directory
2022-09-04 19:09:54,895 unionml INFO Deploying workflow digits_classifier.train
2022-09-04 19:09:55,230 unionml INFO Deploying workflow digits_classifier.predict
2022-09-04 19:09:55,294 unionml INFO Deploying workflow digits_classifier.predict_from_features

When I train the model, I see that a workflow is triggered in the sandbox Flyte Console, but the workflow fails on the

dataset_task

node. The logs for that task are:

[1/1] currentAttempt done. Last Error: USER::containers with unready status: [ffa8038a9524b4920a61-n0-0]|Back-off pulling image "unionml:digits-classifier-a31c8bb4d7f235f31b25a3449e3dcc0c1f82e9be"

It looks like I'm getting an error with finding pulling an image, but I'm not sure what I'm doing wrong 😅. Could you advise on how I can get this working?

So the image is built locally but not available to the demo kubernetes

yeah, you can build the image inside the container

flytectl demo start --source . (in the mnist directory)
flytectl demo exec -- docker build -t mnist:v1 .

@glamorous-carpet-83516, I don’t think the

docker build …

command is required cause it’s automatically handled by

model.remote

.

Question: is there some way I can shorten iteration cycles when writing the code? The flow I was using to make changes was 1. Modify the app.py module. 2. Commit changes to git. 3.

unionml deploy app:model

Step three takes 5-10 minutes, even for very small code changes. Is there some way I can shorten this to <1 minute? I would assume that Docker would cache pre-built and unchanged layers, but it looks like that's not happening.

Yes this can use fast registration- like pyflyte run

hi @astonishing-ocean-79318, yes so if you follow the Deploying to Flyte guide, you’ll see instructions on how to build the docker image from within the local flyte cluster

Cool. I'll take a look and reply if I have other questions!

also, re: fast registration, I made an issue to track this: https://github.com/unionai-oss/unionml/issues/163

demo

this is weird, that rebuilding the container took this long

I'm happy to provide any diagnostic logs if it's helpful.

Ohh ya

I'm free tomorrow after 3 PM ET

Sure - also we are working on the fast register for unionml

You're so welcome! If you want, I can send a Zoom, just DM me your email

Sent