https://flyte.org logo
#ask-the-community
Title
# ask-the-community
a

Alex Papanicolaou

02/23/2023, 5:33 PM
Hey all, I have a quick question about fast registration. The question is, if my project is formed as a Python package and installed in the docker image, will fast registration load the code into the image and “override” the installed package? I would have thought “Yes” but it hasn’t been working that way.
For more clarify, here is my project structure with the workflows packaged up.
Copy code
├── Dockerfile
├── poetry.lock
├── pyproject.toml
└── src
    └── pipelines
        ├── __init__.py
        ├── example_wf.py
        └── another_example_wf.py
And here is the
Dockerfile
that builds the image with package installed.
Copy code
####################################################
# Wheel stage: build the wheel
####################################################
FROM python:$PYTHON_VERSION-slim-buster AS wheel

ARG WORKING_DIR="/root"

ENV PATH="/root/.local/bin:${PATH}"

RUN pip install pipx && \
    pipx install poetry==$POETRY_VERSION && \
    pipx ensurepath

ENV VENV /opt/venv
# Virtual environment
RUN python -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"

WORKDIR $WORKING_DIR

# Copy the actual code
COPY . .

RUN poetry export -f requirements.txt -o requirements.txt --without-hashes && \
    poetry build --format=wheel -vvv && \
    mkdir -p /export && \
    mv dist/*.whl requirements.txt /export

####################################################
# Final stage: an image with the package installed
####################################################
FROM python-base AS final

COPY --from=wheel /export/requirements.txt .
RUN pip install -r requirements.txt

COPY --from=wheel /export/*.whl .
RUN pip install $(echo *.whl) --constraint requirements.txt


# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG TAG
ENV FLYTE_INTERNAL_IMAGE $TAG
y

Yee

02/23/2023, 9:49 PM
hey
so that’s not quite how it works no…
this notion of fast register came about because people change their code a lot more frequently than they change their dependencies.
so to save people from having to build images constantly, we introduced fast register
but it’s currently pretty simple. it just zips up the directory from root, uploads it to s3. and at run time first downloads and unzips the code
so if your code is packaged in a whl, i.e. as a dependency, no unf it won’t get picked up
does that make sense?
this is one area i want to do more work in actually. currently it’s just implemented as just that, there’s no interface around it. would love to explore a bit of making an interface around this, making it a bit pluggable
a

Alex Papanicolaou

02/24/2023, 3:55 AM
Hey @Yee, thanks for the response. So I’m pretty sure I had gathered all that before. But my interpretation of hot loading at task run was that it would download the code into the container, unzip it into the
destionationDirectory
, and then put
destionationDirectory
at the head of
PYTHONPATH
so the import would supersede the module on the image on execution of the task code.
y

Yee

02/24/2023, 4:01 AM
It doesn’t do that second bit. But the hot loading yeah
You can have pythonpath just set in the docker image.
Is that not what you’re seeing? Maybe I need to take a closer look at the docker file again.
Can you download the tar and take a look at it? It should be in json… if you look at the JSON tab, far right, in the UI
a

Alex Papanicolaou

02/24/2023, 4:47 AM
You can have pythonpath just set in the docker image.
Good point. if the dir is empty in the base image unless there’s a hot load, then I should get the behavior I want.
I had read through the source code for pyflyte and thought i saw the pythonpath getting updated. but my tests and your experience says no.
Just had to dump this into that final image in the dockerfile:
Copy code
# Prepending the fast-registration destination directory so that if code is hot loaded
# for fast registration, the hot loaded code will supersede the installed module
ENV PYTHONPATH="/root:$PYTHONPATH"