Has anyone had any experience with using Nix or cu...
# flyte-support
d
Has anyone had any experience with using Nix or custom python paths with Flyte? We're looking at a PoC and struggling to get our custom libraries loaded in the container. We can run the container locally with the same environment variables and the same python interpreter that runs
pyflyte-execute
and
pyflyte-fast-execute
successfully imports the modules, however when running in k8s under
pyflyte-fast-execute
et al we get import issues saying the modules don't exist. I'm wondering if perhaps these executables manipulate the path in some manner. Any help would be appreciated.
using a custom
PYTHONPATH
wouldn't let
pyflyte-fast-execute
run, as it lost its own import paths for things like
google.api
. The
NIX_PYTHONPATH
which should be shimmed in by the python interpreter built by nix works with a local docker run but for some reason doesn't work in the container. We've tried a variety of custom adjustments to imports similar to how nix updates the import paths (via
site
) but still haven't managed to get any libraries loading using our full module resolution / import path.
w
Pyflyte run has ‘—destination-dir’ command which controls where your local files get copied. By default it’s in /root.
d
Looks like you're talking about
--dest-dir
as part of
pyflyte-fast-execute
? I see that is set to
.
when running in our container, and the workdir is
/
so I think it's downloading to the bare root dir. I've tried adding both
/
and
/root
to the
PYTHONPATH
without success.
Well this was an amazing rabbit hole. If the
PYTHONPATH
includes this dependency when running `pyflyte-fast-execute`:
/nix/store/f793s2a77ahwxfywqw0nkrl7r3jh8xgd-python3.10-jmespath-1.0.1/lib/python3.10/site-packages:
then the nix wrapper for
pyflyte-execute
will break importing
google.api
ONLY IF it runs
site.addsitedir('/nix/store/4qa2jr1apqzncvd2myhxxzh2yvvp5268-python3.10-protobuf-5.28.3/lib/python3.10/site-packages')
(which is already in
PYTHONPATH
, but adding via
site.addsitedir
breaks it). So either removing
jmespath
from the root
PYTHONPATH
or removing
protobuf
from the wrapper, and it successfully imports
google.api
. Absolutely mind blown mind blown I do not understand how these things interplay together but I'm glad we now have a working nix pipeline.
😵 2
f
Ohh man
Can you help us document this
d
I'm not even sure how this happened let alone whether it's going to be a problem for other people lol lolcry