Hello I ve been trying to avoid using `pyflyte register` and Flyte #ask-the-community

Hello, I've been trying to avoid using `pyflyte re...

Nuno Martins

08/03/2023, 8:36 AM

Hello, I've been trying to avoid using

pyflyte register

, and instead using

pyflyte package

with

flytectl register

in order to have more control over what I package. Currently my directory structure is the following:

Copy code

root
  src
    workflows.py
    flyte_tasks.py

Inside workflows.py, i do

from src.flyte_tasks import task1

To register everything, I'm running:

Copy code

pyflyte -k src package -f -p <PROJECT_NAME> -d <DEV> --image mycustomimage:test
flytectl register files -p <PROJECT_NAME> -d <DEV> --archive --version <VERSION> --destinationDirectory /root flyte-package.tgz

Inside

mycustomimage

, I have set PYTHONPATH=/root The problem is that, once the workflow is registered and I run it in the Flyte UI, I always get the following error:

Copy code

[1/1] currentAttempt done. Last Error: USER::j.load_task(loader_args=resolver_args)  │
│                                                                              │
│ /opt/venv/lib/python3.9/site-packages/flytekit/core/utils.py:295 in wrapper  │
│                                                                              │
│ ❱ 295 │   │   │   │   return func(*args, **kwargs)                           │
│                                                                              │
│ /opt/venv/lib/python3.9/site-packages/flytekit/core/python_auto_container.py │
│ :235 in load_task                                                            │
│                                                                              │
│ ❱ 235 │   │   task_module = importlib.import_module(name=task_module)  # typ │
│                                                                              │
│ /usr/local/lib/python3.9/importlib/__init__.py:127 in import_module          │
│                                                                              │
│ ❱ 127 │   return _bootstrap._gcd_import(name[level:], package, level)        │
│ in _gcd_import:1030                                                          │
│ in _find_and_load:1007                                                       │
│ in _find_and_load_unlocked:972                                               │
│ in _call_with_frames_removed:228                                             │
│ in _gcd_import:1030                                                          │
│ in _find_and_load:1007                                                       │
│ in _find_and_load_unlocked:984                                               │
╰──────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'src'

Am I doing something wrong?

Jan Fiedler

08/03/2023, 9:15 AM

Can you share your full Dockerfile?

Nuno Martins

08/03/2023, 9:23 AM

Here it is

Copy code

FROM python:3.9-slim-buster

ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag

# Venv variables
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
ENV PATH="${VENV}/bin:$PATH"

# Path to the Flyte Client Secret file
ENV FLYTE_CLIENT_SECRET_FILE=/flyte-configs/flyte-secret

# Update apt packages
RUN apt-get update -y && \
    apt-get upgrade -y && \
    apt-get install -y curl git gcc
RUN rm -rf /var/lib/apt/lists/*

# Install pip dependencies
COPY requirements.txt .
RUN python -m pip install --upgrade pip
RUN pip install -r requirements.txt --no-cache-dir

# Install flytectl
RUN curl -sL <https://ctl.flyte.org/install> | bash

# Copy Flyte configs
RUN mkdir -p /flyte-configs
COPY config/flyte-dev-config.yaml /flyte-configs/flyte-dev-config.yaml

CMD ["bash"]
ENTRYPOINT [ "" ]

Doubt the problem is here, as when using pyflyte register , I manage to run all workflows.

Jan Fiedler

08/03/2023, 9:26 AM

I think you are missing a

COPY . /root

at some point. Your workflow and src code need to be in

/root

inside the container

Nuno Martins

08/03/2023, 9:27 AM

Are you sure? Don't these get unpacked in the --destinationDirectory I pass in flytectl register? I don't want to rebuild the docker image everytime something in the code changes

Jan Fiedler

08/03/2023, 9:31 AM

I think yes! I think it only works like this if you have the workflow aswell as all tasks in 1 python file. But i also could be wrong about it

Nuno Martins

08/03/2023, 9:32 AM

But how come then I can do pyflyte register and run the workflows? Only problem I have is with pyflyte package + flytectl register

Jan Fiedler

08/03/2023, 9:39 AM

This is indeed suspicious, how does your full pyflyte register command looks like?

Nuno Martins

08/03/2023, 9:49 AM

pyflyte register -p <PROJECT> -d <DOMAIN> --image myimage:tag src

Victor Churikov

08/03/2023, 1:23 PM

I don’t use pyflyte/flytectl for registration, but I achieve the same using

FlyteRemote

class in Python using its

register_script

and _`source_path` +

copy_all

args. The source_path_ gets copied inside the container and I don’t have to rebuild my image every time. Works with multiple files, it copies the folder recursively so you don’t have to put everything in one file

Samhita Alla

08/04/2023, 5:22 AM

Are you fast-registering your workflows, @Nuno Martins?

Nuno Martins

08/04/2023, 11:15 AM

Figured it out meanwhile. Yes it was the fast registering flag missing, and also needed

-d /root

in pyflyte package. Thanks for the help.

Jan Fiedler

08/04/2023, 1:46 PM

Nice! Thanks for the update

15 Views

Open in Slack

Previous Next