@ancient-wolf-19325 Can you help me run flyte with my custom modules in a hand rolled container where the libraries are managed / installed via Nix? I'm struggling to get the nix packages importable.

It seems likely that running Flyte with custom modules in a hand-rolled container using Nix for dependency management is possible, but there are no explicit instructions for Nix in the official Flyte documentation. The docs and community examples focus on Docker-based workflows, with dependencies managed via pip, Poetry, or requirements.txt, and recommend ensuring your custom modules are on the Python path or installed in the image before running Flyte tasks. For custom images, the key is to set the WORKDIR and PYTHONPATH correctly so Flyte can discover your modules, as described in the Flyte docs and community discussions. If your Nix-built image installs Python libraries in a non-standard location, you must ensure that location is included in the PYTHONPATH inside the container. Also, when using fast registration or

pyflyte run --remote

, the code archive is extracted to the WORKDIR, so your custom modules must be discoverable from there. There is no official Nix-specific guide, so you may need to adapt the Docker-based instructions to your Nix environment. Would you like more detail on how to set up the WORKDIR, PYTHONPATH, and Flyte registration commands for your use case? Sources: - the Flyte docs - community discussions 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 yes please

My understanding is that, to run Flyte with your custom modules in a Nix-built container, you need to ensure two main things: (1) your Python libraries and custom modules are installed in a location that is included in the PYTHONPATH, and (2) the container’s WORKDIR matches where Flyte expects to extract and run your code. Here’s a step-by-step approach based on Flyte’s documented best practices for custom images and community experience: 1. Build your Nix container: Use your Nix expression to install Python, Flytekit, and all required libraries. Make sure your custom modules are either installed as a package (e.g., via setup.py or pip install .) or copied into a directory that will be on the PYTHONPATH. 2. Set WORKDIR and PYTHONPATH: In your Dockerfile (or Nix equivalent), set the WORKDIR to a directory like /root (the Flyte default). Add the directory containing your custom modules to the PYTHONPATH. For example:

WORKDIR /root
   ENV PYTHONPATH=/root:/path/to/your/modules:$PYTHONPATH

If using Nix, ensure the entrypoint shell or wrapper script sets PYTHONPATH accordingly. 3. Register and run with Flyte: When you use

pyflyte run --remote

or

pyflyte register

, Flyte will package your code and extract it to the WORKDIR in the container. Your modules must be importable from that location. If your modules are in a subdirectory (e.g., src/), either add that to PYTHONPATH or use absolute imports in your code. 4. Troubleshooting: If you get ModuleNotFoundError, check the container at runtime to verify the modules are present and PYTHONPATH is set. You can do this by running an interactive shell in the container and trying to import your modules manually. Here’s a minimal Dockerfile-style example (adapt for Nix as needed):

FROM python:3.11-slim
WORKDIR /root
COPY . /root
ENV PYTHONPATH=/root:/root/src:$PYTHONPATH
RUN pip install flytekit  # Or use Nix to install

For Nix, you might use a shell.nix or default.nix that sets up the environment, and a wrapper script to launch with the correct PYTHONPATH. Would you like a more detailed Nix expression or help debugging a specific error message? Sources: - community discussions 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 so I think this is to do with how Flyte runs things. This does not work

pyflyte-fast-execute --additional-distribution  <s3://my-s3-bucket/my-project/development/EGWMREC3XQD5CU7KQUZIBUFMJQ======/fastcdc0ed62711cf79d38ff43a779d24b11.tar.gz>     --dest-dir .     --     pyflyte-execute --inputs <s3://my-s3-bucket/metadata/propeller/my-project-development-apg5ctcpsf7tml9sc48z/load-preset/data/inputs.pb>     --output-prefix <s3://my-s3-bucket/metadata/propeller/my-project-development-apg5ctcpsf7tml9sc48z/load-preset/data/0>     --raw-output-data-prefix <s3://my-s3-bucket/data/qw/apg5ctcpsf7tml9sc48z-load-preset-0>     --checkpoint-path <s3://my-s3-bucket/data/qw/apg5ctcpsf7tml9sc48z-load-preset-0/_flytecheckpoints>     --prev-checkpoint ""     --resolver flytekit.core.python_auto_container.default_task_resolver     -- task-module workflow.compile_llm task-name load_preset

However if I run

pyflyte-fast-execute --additional-distribution  <s3://my-s3-bucket/my-project/development/EGWMREC3XQD5CU7KQUZIBUFMJQ======/fastcdc0ed62711cf79d38ff43a779d24b11.tar.gz>     --dest-dir .

and then separately run

pyflyte-execute --inputs <s3://my-s3-bucket/metadata/propeller/my-project-development-apg5ctcpsf7tml9sc48z/load-preset/data/inputs.pb>     --output-prefix <s3://my-s3-bucket/metadata/propeller/my-project-development-apg5ctcpsf7tml9sc48z/load-preset/data/0>     --raw-output-data-prefix <s3://my-s3-bucket/data/qw/apg5ctcpsf7tml9sc48z-load-preset-0>     --checkpoint-path <s3://my-s3-bucket/data/qw/apg5ctcpsf7tml9sc48z-load-preset-0/_flytecheckpoints>     --prev-checkpoint ""     --resolver flytekit.core.python_auto_container.default_task_resolver     -- task-module workflow.compile_llm task-name load_preset

it works. So running it inside the

--

somehow loses the correct path.

This behavior is due to how

pyflyte-fast-execute

sets up the environment for the subprocess it launches. When you run

pyflyte-fast-execute --additional-distribution ... --dest-dir . -- pyflyte-execute ...

, the code archive is extracted and the subprocess is launched with an updated PYTHONPATH that includes the destination directory. However, if your Nix environment or container setup does not propagate the PYTHONPATH correctly to the subprocess, your modules may not be importable. Running the two commands separately works because your shell session retains the correct environment after extraction. According to the Flyte documentation and community discussions, you must ensure that the extracted code directory is on the PYTHONPATH at the time

pyflyte-execute

runs. If your Nix shell or entrypoint script resets or overrides PYTHONPATH, the subprocess may not see the extracted modules. You may need to explicitly set PYTHONPATH in your container or wrapper script to include the extraction directory (e.g.,

.

or

/root

), or adjust your Nix build to respect the environment variable propagation. Would you like more detail on how to ensure PYTHONPATH is set correctly in this context? Sources: - the Flyte documentation - community discussions 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 It looks like the adding of

/nix/store/4qa2jr1apqzncvd2myhxxzh2yvvp5268-python3.10-protobuf-5.28.3/lib/python3.10/site-packages

via

site.addsitedir

in the nix wrapper for

pyflyte-execute

causes

google.api

to not be importable.

The Flyte's documentation I have ingested does not mention how adding a Nix store path to

site-packages

via

site.addsitedir

in a wrapper for

pyflyte-execute

could interfere with importing modules like

google.api

. However, your observation suggests that manipulating

site-packages

in this way may disrupt Python's import resolution, especially with namespace packages such as

google.api

, which can be sensitive to the order and method by which directories are added to

sys.path

. If possible, please provide more details about your Nix environment, the exact wrapper script, and how your Python environment is being constructed, as this context can help pinpoint the issue. If this doesn't help, please reach out to the Flyte support team for further assistance. 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

@ancient-wolf-19325 does flyte support protobuf 5+?

Yes, Flyte supports protobuf 5+. The Flyte 1.10.7 release notes confirm the adoption of buf for protobuf stub generation and mention ongoing compatibility improvements with newer protobuf versions. However, some issues with Python gRPC and protobuf 5 were reported and addressed in recent updates, so using the latest Flyte and Flytekit versions is recommended for best compatibility (Flyte 1.10.7 Release Notes, Issue #5448). Would you like more detail on protobuf version compatibility or migration steps? Sources: - Issue #5448 - Flyte 1.10.7 Release Notes 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.