Hi, our team is trying to use fast registration us...
# ask-the-community
w
Hi, our team is trying to use fast registration using
pyflyte register
but hitting a wall. The registration passes and it's worklows and launchplans even shows up in Flyte Console however when executing it we get error:
ModuleNotFoundError: No module named 'flyte'
When looking at the resulted generated file on the S3 minio we find that it is completely empty. Our full command is:
pyflyte --config ~/.flyte/cloud-config.yaml register flyte/workflows -p {{.PROJECT}} --domain {{.DOMAIN}} -v {{add .DOCKER_VERSION 1}} -i "{{.CONTAINER_REGISTRY}}/{{.PROJECT}}:{{.DOCKER_VERSION}}"
Folder structure see image Note that using the
pyflyte package
and
flytectl register files
in the same project does work as expected. Are we missing something?
s
Hi Willem, thanks for reporting the issue.
cc @Kevin Su could it be because of incorrect config issue that s3 file is empty
k
This is sad we would love to get you unblocked
Can you do me a favor and paste the contents of the task tab in the Ui of the execution
Actually before then - can you share your dockerfile
I think it is probably just the wrong install paths
w
Hi @Ketan (kumare3) certainly! Dockerfile:
Copy code
FROM python:3.10-slim-bullseye

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

RUN apt-get update && apt-get install -y build-essential curl gdal-bin libgdal-dev && rm -rf /var/lib/apt/lists/*

ENV VENV /opt/venv
# Virtual environment
RUN python3 -m venv ${VENV}
ENV PATH="${VENV}/bin:$PATH"

# Make sure to install the pygdal version matching the installed gdal version
RUN pip install pygdal=="`gdal-config --version`.*"

# Install Python dependencies
COPY ./requirements.txt /root
RUN pip install --no-cache-dir -r /root/requirements.txt

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag
Requirements.txt
Copy code
flytekit==1.2.3
jmespath==1.0.1
azure-storage-blob==12.13.1
pyproj==3.4.0
awscli==1.25.9
dataclasses-json==0.5.7
openpyxl==3.0.10
While you asked it did notice we are not adding the Python code in the Dockerfile when we use
pyflyte register
we thought the code would be part of the file uploaded to s3. But maybe that assumption was wrong
Also as requested, the contents of the task tab
Copy code
{
  "config": {},
  "id": {
    "resourceType": 1,
    "project": "__REDACTED__",
    "domain": "dev",
    "name": "flyte.tasks.blob_storage.tasks.get_from_blob_storage",
    "version": "155"
  },
  "type": "python-task",
  "metadata": {
    "runtime": {
      "type": 1,
      "version": "1.2.3",
      "flavor": "python"
    },
    "retries": {}
  },
  "interface": {
    "inputs": {
      "variables": {
        "container": {
          "type": {
            "simple": 3
          },
          "description": "container to use on blob storage"
        },
        "file_path": {
          "type": {
            "simple": 3
          },
          "description": "path to the file on blob storage"
        }
      }
    },
    "outputs": {
      "variables": {
        "o0": {
          "type": {
            "blob": {}
          },
          "description": "location of the downloaded file (FlyteFile)"
        }
      }
    }
  },
  "securityContext": {
    "secrets": [
      {
        "group": "azureblob",
        "key": "connectionstring"
      }
    ],
    "tokens": []
  },
  "container": {
    "command": [],
    "args": [
      "pyflyte-fast-execute",
      "--additional-distribution",
      "<s3://my-s3-bucket/4a/__REDACTED__/dev/PI5D2DA6LXTXOVBOIYMIEGZVMY======/fastc04c61d9228f6be2b08faad678cfd364.tar.gz>",
      "--dest-dir",
      "/root",
      "--",
      "pyflyte-execute",
      "--inputs",
      "{{.input}}",
      "--output-prefix",
      "{{.outputPrefix}}",
      "--raw-output-data-prefix",
      "{{.rawOutputDataPrefix}}",
      "--checkpoint-path",
      "{{.checkpointOutputPrefix}}",
      "--prev-checkpoint",
      "{{.prevCheckpointPrefix}}",
      "--resolver",
      "flytekit.core.python_auto_container.default_task_resolver",
      "--",
      "task-module",
      "flyte.tasks.blob_storage.tasks",
      "task-name",
      "get_from_blob_storage"
    ],
    "env": [],
    "config": [],
    "ports": [],
    "image": "__REDACTED__/__REDACTED__:155",
    "resources": {
      "requests": [
        {
          "name": 1,
          "value": "1"
        },
        {
          "name": 3,
          "value": "200Mi"
        }
      ],
      "limits": [
        {
          "name": 1,
          "value": "2"
        },
        {
          "name": 3,
          "value": "250Mi"
        }
      ]
    }
  }
}
k
Ya you don’t need the code to be there - but maybe the flytekit version
Can you check the logs to ensure the s3 file was Downloaded
w
@Ketan (kumare3) I don't see logs in the created pod about downloading the file. Do I have to change a logging level for this? For info these are the full logs of the pod:
Copy code
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-execute", line 8, in <module>
    sys.exit(execute_task_cmd())
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 470, in execute_task_cmd
    _execute_task(
  File "/opt/venv/lib/python3.10/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
    return wrapped(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 342, in _execute_task
    _task_def = resolver_obj.load_task(loader_args=resolver_args)
  File "/opt/venv/lib/python3.10/site-packages/flytekit/core/python_auto_container.py", line 193, in load_task
    task_module = importlib.import_module(task_module)
  File "/usr/local/lib/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'flyte'
Traceback (most recent call last):
  File "/opt/venv/bin/pyflyte-fast-execute", line 8, in <module>
    sys.exit(fast_execute_task_cmd())
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
    return self.main(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
    rv = self.invoke(ctx)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/opt/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
    return __callback(*args, **kwargs)
  File "/opt/venv/lib/python3.10/site-packages/flytekit/bin/entrypoint.py", line 507, in fast_execute_task_cmd
    subprocess.run(cmd, check=True)
  File "/usr/local/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['pyflyte-execute', '--inputs', '<s3://my-s3-bucket/metadata/propeller/REDACTED-ajn6dnhdmzh6gsngtbvt/n1/data/inputs.pb>', '--output-prefix', '<s3://my-s3-bucket/metadata/propeller/REDACTED-ajn6dnhdmzh6gsngtbvt/n1/data/0>', '--raw-output-data-prefix', '<s3://my-s3-bucket/1l/ajn6dnhdmzh6gsngtbvt-n1-0>', '--checkpoint-path', '<s3://my-s3-bucket/1l/ajn6dnhdmzh6gsngtbvt-n1-0/_flytecheckpoints>', '--prev-checkpoint', '""', '--dynamic-addl-distro', '<s3://my-s3-bucket/xw/REDACTED/dev/YFPYVGMNWNNR3W72GM45QXQ6TU======/fast203e87aeba7957fe55641e69aa74fc81.tar.gz>', '--dynamic-dest-dir', '/root', '--resolver', 'flytekit.core.python_auto_container.default_task_resolver', '--', 'task-module', 'flyte.tasks.blob_storage.tasks', 'task-name', 'get_from_blob_storage']' returned non-zero exit status 1.
e
@Willem Gillis, the
flyte
dir is not symlinked, right?
450 Views