Team! Good morning. I have a doubt about packaging...
# ask-the-community
f
Team! Good morning. I have a doubt about packaging/registering Flyte projects. Usually, I would build a docker image containing my code, my system dependencies and my python dependencies. But for what I’m seeing, when registering things, workflows get packaged and kind of mounted into the container. Is this correct?
So, for what I’m seeing, it would be better to have an image only for system dependencies and libraries, and without my code inside it
k
@Franco Bocci this is good for fast iteration. When you are developing. But in production having a fully built image will be faster
Also look into imagespec, It will do this underneath
f
Is it different for production? I’m a bit lost because I see 3 commands involved: •
pyflyte register
pyflyte package
flytectl register
And in all scenarios the code seems to be separate from the “container image”
(I’m probably getting something wrong though)
s
you might want to use registration instead of fast-registration if you want the code to be available in the image. you can use
pyflyte register --non-fast
. if you're using
ImageSpec
, you can copy the code into the image by specifying
source_root
f
So, there are two ways for this to work? One with the code as part of my image, and once where the image only contains dependencies and the code is handled separately?
Let’s assume I don’t want to use
ImageSpec
, and I want to build my own images for my workflows. There are two scenarios: • While developing • When trying to go to production I’ll start with the development scenario, and I’ll have my repository with my code defined in something like (this is 100% an example):
Copy code
|Dockerfile
|pyproject.toml
|poetry.lock
|src/
|- workflows/
|- internal_library/
I would set up somehow a workflow to build and push my image to a docker registry. This image would contain my dependencies and my code (everything in
src/
, both Flyte related and non Flyte related). According to this , the next step would be running
pyflyte register workflows --image ...
with the image I previously built. But my image had my code already. So when I execute these newly registered workflow, how do I know whether I’m using this or the previous workflows?
s
So, there are two ways for this to work?
no, you'll need to use registration if you want flyte to use the code available in the image.
pyflyte register
uses fast-registration by default. you could also use
pyflyte package
and
flytectl register
commands.
So when I execute these newly registered workflow, how do I know whether I’m using this or the previous workflows?
you'll need to rebuild the image every time you modify your code. for that reason, fast registration is preferred over registration while developing workflows.
f
If my built image has my code, and I
fast-register
, what happens? Because I’d be registering the local version of the workflows, on top of an image which would contain a previous version of those workflows
s
that shouldn't be a problem.
flytekit should still use your latest workflow code.
f
So this is the source of my question. If Flyte knows how to pick up my latest workflow code, why would I pack it in my image in the first place?
s
when deploying to production, it's useful to have code in the image to ensure reproducibility via code versioning.