Hey I m not having the easiest time with python environment Flyte #flyte-support

Hey, I’m not having the easiest time with python e...

green-agency-30976

10/27/2023, 3:35 PM

Hey, I’m not having the easiest time with python environment. I found out that any custom image needs to set the python path to root. It’s the default, in

pyflyte register

, to which the workflows are shipped, at runtime. I’m just trying the `pyflyte package`+`flytectl register` way now and back to this import issue again. Running the following from the docs

Copy code

pyflyte --pkgs example package --image ghcr.io/flyteorg/flytekit:py3.9-latest
flytectl register files \
    --project flytesnacks \
    --domain development \
    --archive flyte-package.tgz \
    --version "0.1.6"

and triggering the workflow manually from the UI gives the following error:

Copy code

[1/1] currentAttempt done. Last Error: USER::                              │
│ /usr/local/lib/python3.9/site-packages/flytekit/bin/entrypoint.py:365 in     │
│ _execute_task                                                                │
│                                                                              │
│ ❱ 365 │   │   _task_def = resolver_obj.load_task(loader_args=resolver_args)  │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/flytekit/core/utils.py:295 in wrapper │
│                                                                              │
│ ❱ 295 │   │   │   │   return func(*args, **kwargs)                           │
│                                                                              │
│ /usr/local/lib/python3.9/site-packages/flytekit/core/python_auto_container.p │
│ y:235 in load_task                                                           │
│                                                                              │
│ ❱ 235 │   │   task_module = importlib.import_module(name=task_module)  # typ │
│                                                                              │
│ /usr/local/lib/python3.9/importlib/__init__.py:127 in import_module          │
│                                                                              │
│ ❱ 127 │   return _bootstrap._gcd_import(name[level:], package, level)        │
│ in _gcd_import:1030                                                          │
│ in _find_and_load:1007                                                       │
│ in _find_and_load_unlocked:984                                               │
╰──────────────────────────────────────────────────────────────────────────────╯
ModuleNotFoundError: No module named 'example'

Any suggestion wrt how to debug it would be much apreciated 🙂 Running the demo stack on a mac m1 pro with Ventura if that’s of any relevance

green-agency-30976

10/27/2023, 4:00 PM

Looks like I was just a bit duped by the nicety of fast registration!

sparse-advantage-22780

10/27/2023, 4:09 PM

you can actually set PYTHONPATH in containers that aren't at /root

sparse-advantage-22780

10/27/2023, 4:09 PM

Copy code

@task(container_image="localhost:30000/esmfold:latest", environment={"PYTHONPATH": "/root"}, requests=Resources(cpu="7", mem="28Gi"))
def run_esmfold(sequences: list[str]) -> list[FlyteFile]:
    """
    This output should maybe be a FlyteFile instead?
    """

sparse-advantage-22780

10/27/2023, 4:11 PM

sorry I read too fast, I think that's not your problem : p

green-agency-30976

10/27/2023, 4:31 PM

Great to know anyway! Thanks for sharing 🙂

tall-lock-23197

10/30/2023, 4:53 AM

@green-agency-30976, are you still seeing the error?

green-agency-30976

10/30/2023, 10:18 AM

Hi, thanks for asking. All good yes This error is the expected behaviour AFAIK. This does what I want with the

--fast

flag

Copy code

pyflyte --pkgs example package --fast --image <http://ghcr.io/flyteorg/flytekit:py3.9-latest|ghcr.io/flyteorg/flytekit:py3.9-latest>

👍 1

6 Views

Open in Slack

Previous Next