Hi, our team is trying to use fast registration using

pyflyte register

but hitting a wall. The registration passes and it's worklows and launchplans even shows up in Flyte Console however when executing it we get error:

ModuleNotFoundError: No module named 'flyte'

When looking at the resulted generated file on the S3 minio we find that it is completely empty. Our full command is:

pyflyte --config ~/.flyte/cloud-config.yaml register flyte/workflows -p {{.PROJECT}} --domain {{.DOMAIN}} -v {{add .DOCKER_VERSION 1}} -i "{{.CONTAINER_REGISTRY}}/{{.PROJECT}}:{{.DOCKER_VERSION}}"

Folder structure see image Note that using the

pyflyte package

and

flytectl register files

in the same project does work as expected. Are we missing something?

Hi Willem, thanks for reporting the issue.

This is sad we would love to get you unblocked

Hi @freezing-airport-6809 certainly! Dockerfile:

FROM python:3.10-slim-bullseye

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

RUN apt-get update && apt-get install -y build-essential curl gdal-bin libgdal-dev && rm -rf /var/lib/apt/lists/*

ENV VENV /opt/venv
# Virtual environment
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"

# Make sure to install the pygdal version matching the installed gdal version
RUN pip install pygdal=="`gdal-config --version`.*"

# Install Python dependencies
COPY ./requirements.txt /root
RUN pip install --no-cache-dir -r /root/requirements.txt

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag

Requirements.txt

flytekit==1.2.3
jmespath==1.0.1
azure-storage-blob==12.13.1
pyproj==3.4.0
awscli==1.25.9
dataclasses-json==0.5.7
openpyxl==3.0.10

While you asked it did notice we are not adding the Python code in the Dockerfile when we use

pyflyte register

we thought the code would be part of the file uploaded to s3. But maybe that assumption was wrong

Ya you don’t need the code to be there - but maybe the flytekit version

@freezing-airport-6809 I don't see logs in the created pod about downloading the file. Do I have to change a logging level for this? For info these are the full logs of the pod:

Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-execute", line 8, in <module>
    sys.exit(execute_task_cmd())
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd
    _execute_task(
  File "/opt/venv/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 342, in _execute_task
    _task_def = resolver_obj.load_task(loader_args=resolver_args)
  File "/opt/venv/lib/python3.10/site-packages/flytekit/core/python_auto_container.py", line 193, in load_task
    task_module = importlib.import_module(task_module)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'flyte'
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 507, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/REDACTED-ajn6dnhdmzh6gsngtbvt/n1/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/REDACTED-ajn6dnhdmzh6gsngtbvt/n1/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/1l/ajn6dnhdmzh6gsngtbvt-n1-0>', '--checkpoint-path', '<s3://my-s3-bucket/1l/ajn6dnhdmzh6gsngtbvt-n1-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/xw/REDACTED/dev/YFPYVGMNWNNR3W72GM45QXQ6TU======/fast203e87aeba7957fe55641e69aa74fc81.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'flyte.tasks.blob_storage.tasks', 'task-name', 'get_from_blob_storage']' returned non-zero exit status 1.

@flat-monkey-49105, the

flyte

dir is not symlinked, right?