Hello! I have a question regarding multi-image workflows. We have task1 which is simply a tensorflow job and task2 is a spark job. When we run

pyflyte package

it fails because it does not have all the dependencies --> flytekitplugins-spark + tensorflow. I thought by specifying

container_image

differently for those tasks and importing those modules within the task context was magically solving this problem but not? Do we need to install all python source code dependencies when we run

pyflyte package

command?

yes, you need all dependencies locally to compile the workflow. otherwise, python will failed to import module

This looks quite promising but when will it build the image? does it support using a private docker image registry and private pip index? is it possible to specify package versions?

• before registration. when you run

pyflyte run

, it will build the image first if image not found. • it uses docker under the hood, so I think it should support private pip as well. I haven’t try it, but let me know if you run into any issue. • yes, you can specify package version. like`pandas==1,4.0`

Alright, thank you!

yes, pyflyte register use fast register by default

I’ve been testing imageSpec but when I run the pyflyte register command, it builds the image many times. I have set an image_spec to multiple tasks and think it’s building the image for each task. I see this log multiple times, with the same image name and version, am I doing something wrong?

Image flytekit:_CB4mqOFLHJIgtJSHe9hyQ.. not found. Building...
Run command: envd build --path /var/folders/l5/b5c4dzc12n3cd4pbyrhkwxd40000gp/T/flyte-3wc4a3dt/sandbox/local_flytekit/833f8fa328235df82b6a50a4958bb4fd

which version of flytekit you are using

1.6.0b4

and

tensorflow_image_spec=ImageSpec(
    base_image="tensorflow/tensorflow:2.12.0-gpu",
    python_version="3.9"
)