@ancient-wolf-19325 I have a bazel mono repo with a dockerfile where I’m trying to install my pip requirements into the docker container. This docker container is then built inside the /app workerdir. I have two files hello_world.py which has a workflow and hello_world_registry which trys to register my workflow on flyte-core using a kubernetes port forwarded environment. The task is spun up in my PROJECT & DOMAIN and tries to run

pyflyte-fast-execute \
  --additional-distribution <s3://path/to/script_mode.tar.gz> \
  --dest-dir . \
  -- \
  pyflyte-execute \
  --inputs <s3://path/to/metadata/propeller/bucket-name/n0/data/inputs.pb> \
  --output-prefix <s3://path/to/metadata/propeller/bucket-name/n0/data/0> \
  --raw-output-data-prefix <s3://my-s3-bucket/pod-name-n0-0> \
  --checkpoint-path <s3://my-s3-bucket/hi/pod-name-n0-0/_flytecheckpoints> \
  --prev-checkpoint "" \
  --resolver hello_world_registry.rules_python++pip+pypi_311_flytekit.site-packages.flytekit.core.python_auto_container.default_task_resolver \
  -- \
  task-module monorepo/path/to/hello_world \
  task-name generate_sample_data

But I’m getting a module not found error for hello_world_registry? FlyteAssertion: USERAssertionError error=Outputs could not be found because the execution ended in failure. Error message: [3/3] currentAttempt done. Last Error: SYSTEM::Pod failed. No message received from kubernetes. [pod-name-n0-3] terminated with exit code (1). Reason [Error]. Message: _module(“.”.join(class_obj_mod)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “/usr/local/lib/python3.11/importlib/__init__.py”, line 126, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File “<frozen importlib._bootstrap>“, line 1204, in _gcd_import File “<frozen importlib._bootstrap>“, line 1176, in _find_and_load File “<frozen importlib._bootstrap>“, line 1126, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 241, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1204, in _gcd_import File “<frozen importlib._bootstrap>“, line 1176, in _find_and_load File “<frozen importlib._bootstrap>“, line 1126, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 241, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1204, in _gcd_import File “<frozen importlib._bootstrap>“, line 1176, in _find_and_load File “<frozen importlib._bootstrap>“, line 1126, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 241, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1204, in _gcd_import File “<frozen importlib._bootstrap>“, line 1176, in _find_and_load File “<frozen importlib._bootstrap>“, line 1126, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 241, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1204, in _gcd_import File “<frozen importlib._bootstrap>“, line 1176, in _find_and_load File “<frozen importlib._bootstrap>“, line 1126, in _find_and_load_unlocked File “<frozen importlib._bootstrap>“, line 241, in _call_with_frames_removed File “<frozen importlib._bootstrap>“, line 1204, in _gcd_import File “<frozen importlib._bootstrap>“, line 1176, in _find_and_load File “<frozen importlib._bootstrap>“, line 1140, in _find_and_load_unlocked ModuleNotFoundError: No module named ‘hello_world_registry’

The "ModuleNotFoundError: No module named 'hello_world_registry'" typically means that the module is not present in the Python path inside your running container, or it was not included in the fast registration tarball. For Flyte to discover and import your modules, ensure that: - The directory containing

hello_world_registry.py

is included in the tarball created by

pyflyte register

or

pyflyte package

, and that the extraction path matches the container's

WORKDIR

or is on the

PYTHONPATH

. - The directory is a valid Python package (i.e., contains an

__init__.py

file) and your imports are absolute, not relative. - If using fast registration, the code archive is extracted to the correct location (matching

WORKDIR

or

source_root

), and your Dockerfile or ImageSpec sets

WORKDIR

and

PYTHONPATH

accordingly. See fast registration docs and packaging guidance. If you are using Bazel, ensure the build output structure matches what Flyte expects, and that the code is not nested in a way that breaks imports. Also, check that the

--resolver

and

task-module

arguments use the correct Python module path, not a filesystem path. Would you like a step-by-step checklist or example Dockerfile and registration commands? Sources: - Fast registration documentation - Registering workflows and packaging 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 So the module seems to be present, meaning when I run a local shell and import the file explicitly at path.to.module.hello_world_registry I’m able to import it no problem could this be an issue with my PYTHONPATH? What should my PYTHONPATH look like here?

Yes, this is likely a PYTHONPATH issue. In Flyte, the PYTHONPATH inside the container must include the root directory where your module hierarchy (e.g.,

path/to/module/hello_world_registry.py

) begins, so that imports like

import path.to.module.hello_world_registry

work. If your

WORKDIR

is

/app

, and your code is at

/app/path/to/module/hello_world_registry.py

, then

/app

should be in your PYTHONPATH. See Flyte project structure guidance and packaging/registration docs. If PYTHONPATH is not set correctly, Python won’t find your modules even if the files exist. You can set it in your Dockerfile with:

ENV PYTHONPATH=/app:$PYTHONPATH

Would you like more detail on debugging or setting up your Dockerfile and Flyte registration? Sources: - Registering workflows and packaging - Flyte project structure guidance 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 so if I echo my python path what should I see? b/c I ran PYTHONPATH=/app:$PYTHONPATH in my pod and reran pyflyte and got the same error Are there some things I can check for you that can help me debug this?

If you echo your PYTHONPATH inside the pod, you should see /app (or the directory containing your top-level Python package) as one of the entries. For example:

/app:/usr/local/lib/python3.11/site-packages:...

If you still get the error, check: 1. That /app/path/to/module/hello_world_registry.py exists in the container. 2. That /app is in PYTHONPATH. 3. That your import statement matches the directory structure (e.g., import path.to.module.hello_world_registry). 4. That your code archive is extracted to /app (matching WORKDIR and PYTHONPATH). You can debug by running:

ls -lR /app
echo $PYTHONPATH
python -c "import path.to.module.hello_world_registry"

Would you like more detailed troubleshooting steps? Sources: - Registering workflows and packaging - Fast registration documentation 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 I see this

echo $PYTHONPATH
/app:/root

I also see

/app/path/to/module/hello_world_registry.py

and /app/path/to/module/hello_world.py Running python above works just fine

If you can import your module with Python in the pod, but Flyte still fails with ModuleNotFoundError, the issue is likely with how Flyte is invoking the task or with the registration/packaging process, not with PYTHONPATH or file placement. This can happen if the task-module or resolver argument is not using the correct Python module path, or if the code archive is not being extracted as expected during fast registration. See Flyte registration docs. Double-check that: - The

task-module

argument is set to the Python module path (e.g.,

path.to.module.hello_world

) and not a filesystem path. - The code archive is being extracted to

/app

and not to a subdirectory. - The Flyte version in your container matches your local environment. Would you like step-by-step guidance to verify your registration and invocation commands? Sources: - Registering workflows and packaging 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes please give me step by step guidance on how to verify registration and invocation commands, please ask me for any other information that might help you debug this issue.

pyflyte-fast-execute \
  --additional-distribution <s3://path/to/script_mode.tar.gz> \
  --dest-dir . \
  -- \
  pyflyte-execute \
  --inputs <s3://path/to/metadata/propeller/bucket-name/n0/data/inputs.pb> \
  --output-prefix <s3://path/to/metadata/propeller/bucket-name/n0/data/0> \
  --raw-output-data-prefix <s3://my-s3-bucket/pod-name-n0-0> \
  --checkpoint-path <s3://my-s3-bucket/hi/pod-name-n0-0/_flytecheckpoints> \
  --prev-checkpoint "" \
  --resolver hello_world_registry.rules_python++pip+pypi_311_flytekit.site-packages.flytekit.core.python_auto_container.default_task_resolver \
  -- \
  task-module monorepo/path/to/hello_world \
  task-name generate_sample_data

This command appears to • Have a python interpreter where we can import the task module ie python -c “monorepo.path.to.hello_world” and python -c “monorepo.path.to.hello_world_registry” all work fine • I have verified that /app/monorepo/path/to/hello_world* exists • PYTHONPATH when loaded doesn’t appear to have /app however inside a shell before I call pyflyte inside the pod I do PYTHONPATH=/app:$PYTHONPATH • The structure so far is ◦ /app/path/to/module/hello_world.py ◦ /app/path/to/module/hello_world_registry.py • I can confirm that the

script_mode.tar.gz

is being downloaded as expected

It seems likely that your environment is close to correct, but Flyte’s invocation may not be using the same PYTHONPATH or module resolution as your manual shell. Here’s a step-by-step checklist to help debug and verify your registration and invocation: 1. Confirm PYTHONPATH at Execution - Add a print statement or use an init script to print

sys.path

at the top of your task code. This will show the actual PYTHONPATH Flyte uses during execution, not just in your shell. - Ensure

/app

is present in

sys.path

when the task runs. 2. Check task-module and resolver Arguments - The

task-module

argument must be the Python module path, not a filesystem path. For your structure, it should be

monorepo.path.to.hello_world

(not

monorepo/path/to/hello_world

). - The

--resolver

should be the default:

flytekit.core.python_auto_container.default_task_resolver

unless you have a custom resolver. Your current resolver string looks unusual and may be causing issues. 3. Verify Fast Registration Extraction - Confirm that

script_mode.tar.gz

is extracted to

/app

(not a subdirectory like

/app/app

). - After extraction, run

ls -lR /app

to verify the files are in the expected locations. 4. Registration and Packaging - When running

pyflyte register

or

pyflyte package

, ensure you specify the correct package/module and use the

--source

flag if your code is not at the root. - Example:

pyflyte --pkgs monorepo/path/to package --source /app -f

- Register the resulting tarball with

flytectl register files ...

. 5. Dockerfile and Entrypoint - Ensure your Dockerfile sets

WORKDIR /app

and

ENV PYTHONPATH=/app:$PYTHONPATH

. - The entrypoint should not override PYTHONPATH. 6. Test Import in the Same Way as Flyte - In the pod, run:

export PYTHONPATH=/app:$PYTHONPATH
  python -c "import monorepo.path.to.hello_world_registry"

- This should match how Flyte will import the module. 7. Review Flyte Logs - Check the full error traceback in the Flyte UI or pod logs for clues about the import failure. - Look for differences in the import path or missing files. Questions to Help Debug Further: - What is the exact value of

sys.path

when the task fails? - What is the full error traceback from the Flyte logs? - What command did you use to package and register your workflow? - Are you using a custom entrypoint or resolver in your Dockerfile or Flyte config? Would you like a sample Dockerfile and registration command for your structure? Sources: - Registering workflows and packaging 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 So the error seems to be the resolver, what is the resolver and why is it pointing to my bazel runfiles? How should I be registering this (right now I’m registering with bazel)

remote = FlyteRemote(
        # config=Config.for_endpoint(
        #     endpoint=FLYTE_ADMIN_URL,
        #     insecure=True,
        # ),
        config=Config.auto(
            config_file=CONFIG_PATH,
            # platform=PlatformConfig(
            #     endpoint="localhost:8089",
            #     insecure=True,
            #     insecure_skip_verify=True,
            # ),
        ),
        default_project=PROJECT,
        default_domain=DOMAIN,
        data_upload_location=DATA_UPLOAD_LOCATION,
        default_image=CONTAINER_IMAGE,
    )
    # s3_access_credientials()
    if not is_already_registered:
        remote.fast_register_workflow(
            entity=ray_distributed_workflow,
            version=version,
        )

It seems likely that the resolver is a Python class responsible for locating and loading your Flyte tasks at execution time. If the resolver path is pointing to your Bazel runfiles, it suggests that the registration process is capturing the module path as it appears in your Bazel build environment, not as it should be in the container at runtime. The resolver should typically be set to

flytekit.core.python_auto_container.default_task_resolver

for standard Python projects, and the

task-module

should be the importable Python module path (e.g.,

monorepo.path.to.hello_world

), not a Bazel-specific path. See Flyte task resolver docs. When registering with Bazel, ensure that the Python import paths in your packaged code match the structure and PYTHONPATH in your container, and avoid Bazel-specific prefixes in the resolver or module names. If you use

FlyteRemote.fast_register_workflow

, it should handle this for you if your code is structured as a standard Python package. Would you like more detail on how to ensure your resolver and module paths are correct for Flyte? Sources: - Flyte task resolver documentation - Registering workflows and packaging 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 is there a way to set the resolver either by adding an image spec or by modifying the flyte remote registeritation config? b/c as you said there python path is likely referring to my local environment incorrectly

You cannot directly set a custom resolver via ImageSpec or FlyteRemote registration config; Flyte uses the default resolver (

flytekit.core.python_auto_container.default_task_resolver

) unless you explicitly override it in your task or workflow code using the

set_resolver()

method. The resolver is determined at serialization/registration time, and if your local environment (e.g., Bazel runfiles) pollutes the module path, that path will be embedded in the registered entity and used at runtime, causing import errors if it doesn't match the container's environment. See task resolver docs. To avoid this, ensure you register your workflows from an environment that matches your container's module structure and PYTHONPATH. If needed, you can set a custom resolver in code:

from flytekit.core.python_auto_container import default_task_resolver
my_task.set_resolver(default_task_resolver)

But this is rarely necessary if your packaging and registration match your runtime environment. Would you like more detail on aligning your registration and runtime environments? Sources: - https://github.com/flyteorg/flyte/issues/5343 - https://github.com/flyteorg/flyte/issues/5706 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes I’d like more details, specifically I’d like to understand if it’s even possible to use bazel here? I’m not sure why my bazel runfile is overriding the default resolver?

It seems likely that using Bazel is possible, but it requires careful alignment between your Bazel build outputs and the Python module/package structure expected by Flyte at runtime. The issue you’re seeing—where the resolver path is set to your Bazel runfile—suggests that Bazel’s build or execution environment is introducing non-standard module paths, which Flyte then serializes and uses at runtime. This can cause import errors if the runtime container does not mirror the Bazel environment exactly. There is no evidence in the data sources that Bazel itself overrides the default resolver, but rather that the registration process (when run inside Bazel) may serialize the module path as it appears in the Bazel environment, not as it will appear in the container. To use Bazel successfully: - Ensure your Bazel build outputs a directory structure that matches your intended Python package/module layout. - Register your Flyte workflows from a Bazel output directory that mirrors the container’s PYTHONPATH and WORKDIR. - Avoid registering from within a Bazel runfile context that introduces Bazel-specific prefixes or paths. - Always check that the resolver in your registered tasks is

flytekit.core.python_auto_container.default_task_resolver

unless you have a specific reason to override it. If you need to debug further, try registering your workflow outside of Bazel to see if the resolver is set correctly, then compare with the Bazel-based registration. This can help isolate whether Bazel’s environment is introducing the issue. Would you like a step-by-step example of how to structure your Bazel build and registration for Flyte? Sources: - https://github.com/flyteorg/flyte/issues/5343 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 I know that the resolver is incorrect, when I change

--resolver hello_world_registry.rules_python++pip+pypi_311_flytekit.site-packages.flytekit.core.python_auto_container.default_task_resolver

to

--resolver flytekit.core.python_auto_container.default_task_resolver

inside the pod the task executes as expected. Could you help me understand how to properly configure my FlyteRemote in an environment where sitepackages are installed in a dockerfile and the registration is run from my local machine in bazel

The resolver should always be set to

flytekit.core.python_auto_container.default_task_resolver

for standard Flyte Python tasks. If you see a resolver like

hello_world_registry.rules_python++pip+pypi_311_flytekit.site-packages.flytekit.core.python_auto_container.default_task_resolver

, it means your Bazel environment is injecting Bazel-specific prefixes into the module path during registration, which causes import errors at runtime. This is not a FlyteRemote or Dockerfile configuration issue, but a result of how/where you run registration. To fix this: - Always run

FlyteRemote.fast_register_workflow

(or any registration command) from a directory and environment that matches your container’s runtime Python path and module structure. Avoid running registration from within Bazel runfiles or with Bazel-specific PYTHONPATHs. - Ensure your Dockerfile installs your code and dependencies into standard locations (e.g.,

/app

), and sets

WORKDIR /app

and

ENV PYTHONPATH=/app:$PYTHONPATH

. - When using FlyteRemote, you do not need to set the resolver manually; Flyte will use the default unless your registration environment pollutes the module path. Summary: The key is to register from a clean environment (not Bazel runfiles) that matches your container’s structure, so the resolver is set to

flytekit.core.python_auto_container.default_task_resolver

and module paths are importable at runtime. Would you like a sample workflow for registration and Dockerfile setup? Sources: - Registering workflows and packaging 👍 0 button 👎 1 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.