https://flyte.org logo
#ask-the-community
Title
# ask-the-community
g

Geert

11/23/2023, 1:21 PM
Hi! When registering a workflow, is there a convenient way to prevent Flyte wanting to install any Python dependencies locally? We just want to register and run it on remote (where we have all of our dependencies in the Docker environment already). A workaround that we use is this, but it’s not very nice:
Copy code
if "SKIP_HEAVY_IMPORTS" not in os.environ:
    import some_package
y

Yee

11/24/2023, 2:26 AM
there’s a related feature in image spec called
is_container
but i don’t think that will suffice for your purposes. you can always hack things around like moving imports into the body of tasks, but fundamentally task signatures are inspected by flytekit and inspected for typetransformers, etc. i think we to establish a good pattern for this @Thomas Fan @Eduardo Apolinario (eapolinario)
also ties into the discussion around lazy loading internally in flytekit.
g

Geert

11/24/2023, 7:50 AM
Aha, we’ll look into
is_container()
, but I gather this is just something we have to deal with for now then? There is no way to not install task-specific Python requirements when registering. Flyte needs them all installed locally.
y

Yee

11/24/2023, 9:07 AM
the way to do it is by getting python to not load it.
so when you register only the body of workflows are run, not the body of task functions. task functions only get their signatures inspected. if you can figure out a way to delay the loading of the libraries you don’t want until the body of the function is run, that should do it
t

Thomas Fan

11/27/2023, 4:15 PM
Most code I have seen that want to delay the import end up placing the imports into the function body. The other pattern is to use a library such as https://github.com/scientific-python/lazy_loader which implements lazy loading, but it is not well established.
g

Geert

11/27/2023, 5:01 PM
That is a doable fix for us, thanks for the input!
4 Views