https://flyte.org
Join Slack
Team! Good morning. I have a doubt about packaging/registering Flyte projects. Usually, I would buil...
# flyte-support
t
Team! Good morning. I have a doubt about packaging/registering Flyte projects. Usually, I would build a docker image containing my code, my system dependencies and my python dependencies. But for what I’m seeing, when registering things, workflows get packaged and kind of mounted into the container. Is this correct?
So, for what I’m seeing, it would be better to have an image only for system dependencies and libraries, and without my code inside it
f
@thankful-tailor-28399 this is good for fast iteration. When you are developing. But in production having a fully built image will be faster
Also look into imagespec, It will do this underneath
t
Is it different for production? I’m a bit lost because I see 3 commands involved: • 
pyflyte register
• 
pyflyte package
• 
flytectl register
And in all scenarios the code seems to be separate from the “container image”
(I’m probably getting something wrong though)
t
you might want to use registration instead of fast-registration if you want the code to be available in the image. you can use 
pyflyte register --non-fast
. if you're using 
ImageSpec
, you can copy the code into the image by specifying 
source_root
t
So, there are two ways for this to work? One with the code as part of my image, and once where the image only contains dependencies and the code is handled separately?
Let’s assume I don’t want to use 
ImageSpec
, and I want to build my own images for my workflows. There are two scenarios: • While developing • When trying to go to production I’ll start with the development scenario, and I’ll have my repository with my code defined in something like (this is 100% an example):
Copy code
|Dockerfile
|pyproject.toml
|poetry.lock
|src/
|- workflows/
|- internal_library/
I would set up somehow a workflow to build and push my image to a docker registry. This image would contain my dependencies and my code (everything in 
src/
, both Flyte related and non Flyte related). According to this , the next step would be running 
pyflyte register workflows --image ...
with the image I previously built. But my image had my code already. So when I execute these newly registered workflow, how do I know whether I’m using this or the previous workflows?
t
So, there are two ways for this to work?
no, you'll need to use registration if you want flyte to use the code available in the image. 
pyflyte register
uses fast-registration by default. you could also use 
pyflyte package
and 
flytectl register
commands.
So when I execute these newly registered workflow, how do I know whether I’m using this or the previous workflows?
you'll need to rebuild the image every time you modify your code. for that reason, fast registration is preferred over registration while developing workflows.
t
If my built image has my code, and I 
fast-register
, what happens? Because I’d be registering the local version of the workflows, on top of an image which would contain a previous version of those workflows
t
that shouldn't be a problem.
flytekit should still use your latest workflow code.
t
So this is the source of my question. If Flyte knows how to pick up my latest workflow code, why would I pack it in my image in the first place?
t
when deploying to production, it's useful to have code in the image to ensure reproducibility via code versioning.
45 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by