quiet-evening-69124
08/03/2023, 8:36 AMpyflyte register
, and instead using pyflyte package
with flytectl register
in order to have more control over what I package. Currently my directory structure is the following:
root
src
workflows.py
flyte_tasks.py
Inside workflows.py, i do from src.flyte_tasks import task1
To register everything, I'm running:
pyflyte -k src package -f -p <PROJECT_NAME> -d <DEV> --image mycustomimage:test
flytectl register files -p <PROJECT_NAME> -d <DEV> --archive --version <VERSION> --destinationDirectory /root flyte-package.tgz
Inside mycustomimage
, I have set PYTHONPATH=/root
The problem is that, once the workflow is registered and I run it in the Flyte UI, I always get the following error:
[1/1] currentAttempt done. Last Error: USER::j.load_task(loader_args=resolver_args) │
│ │
│ /opt/venv/lib/python3.9/site-packages/flytekit/core/utils.py:295 in wrapper │
│ │
│ ❱ 295 │ │ │ │ return func(*args, **kwargs) │
│ │
│ /opt/venv/lib/python3.9/site-packages/flytekit/core/python_auto_container.py │
│ :235 in load_task │
│ │
│ ❱ 235 │ │ task_module = importlib.import_module(name=task_module) # typ │
│ │
│ /usr/local/lib/python3.9/importlib/__init__.py:127 in import_module │
│ │
│ ❱ 127 │ return _bootstrap._gcd_import(name[level:], package, level) │
│ in _gcd_import:1030 │
│ in _find_and_load:1007 │
│ in _find_and_load_unlocked:972 │
│ in _call_with_frames_removed:228 │
│ in _gcd_import:1030 │
│ in _find_and_load:1007 │
│ in _find_and_load_unlocked:984 │
╰──────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'src'
Am I doing something wrong?faint-activity-87590
08/03/2023, 9:15 AMquiet-evening-69124
08/03/2023, 9:23 AMFROM python:3.9-slim-buster
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
# Venv variables
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root
ENV PATH="${VENV}/bin:$PATH"
# Path to the Flyte Client Secret file
ENV FLYTE_CLIENT_SECRET_FILE=/flyte-configs/flyte-secret
# Update apt packages
RUN apt-get update -y && \
apt-get upgrade -y && \
apt-get install -y curl git gcc
RUN rm -rf /var/lib/apt/lists/*
# Install pip dependencies
COPY requirements.txt .
RUN python -m pip install --upgrade pip
RUN pip install -r requirements.txt --no-cache-dir
# Install flytectl
RUN curl -sL <https://ctl.flyte.org/install> | bash
# Copy Flyte configs
RUN mkdir -p /flyte-configs
COPY config/flyte-dev-config.yaml /flyte-configs/flyte-dev-config.yaml
CMD ["bash"]
ENTRYPOINT [ "" ]
Doubt the problem is here, as when using pyflyte register , I manage to run all workflows.faint-activity-87590
08/03/2023, 9:26 AMCOPY . /root
at some point.
Your workflow and src code need to be in /root
inside the containerquiet-evening-69124
08/03/2023, 9:27 AMfaint-activity-87590
08/03/2023, 9:31 AMquiet-evening-69124
08/03/2023, 9:32 AMfaint-activity-87590
08/03/2023, 9:39 AMquiet-evening-69124
08/03/2023, 9:49 AMpyflyte register -p <PROJECT> -d <DOMAIN> --image myimage:tag src
kind-kite-58745
08/03/2023, 1:23 PMFlyteRemote
class in Python using its register_script
and _`source_path` + copy_all
args. The source_path_ gets copied inside the container and I don’t have to rebuild my image every time. Works with multiple files, it copies the folder recursively so you don’t have to put everything in one filetall-lock-23197
quiet-evening-69124
08/04/2023, 11:15 AM-d /root
in pyflyte package. Thanks for the help.faint-activity-87590
08/04/2023, 1:46 PM