This seems like a bug with

pyflyte package

-

pyflyte run

seems to work fine when I add the project root path to

sys.path

I don't think you need to modify

sys.path

. Can you add an

__init__.py

file to

wf

directory and run your pyflyte package command? Let me know if you're seeing any error. Your PYTHONPATH has to be the root directory where you have tasks and workflows.

Yes, there’s

__init__.py

but I’m not able to run

pyflyte package

where wf and tasks are. As you can see in the project structure, I had to run it two levels up from where the wf code is because I couldn’t get it to run otherwise. What would be

--pkgs

value be if I run it in the same dir as the wf code? I tried using

.

for current dir, but that doesn’t work.

Yeah, it has to be one level up. What's the error you're seeing?

OK so I tried running it where the wf file is and got it to package/register, but execution is failing. Here’s my directory structure and pyflyte command

@seunggs, sorry that you've been trying to fix this for two days. I got it working and here's the directory structure, code and commands I ran. project │ ├── Dockerfile │ └── wf │ ├── init.py │ ├── main.py │ ├── src │ │ └── helpers.py │ └── wf.py helpers.py

def sum(a, b):
    return a + b

main.py

from .src.helpers import sum
from flytekit import task


@task
def main_task(a: int, b: int) -> int:
    return sum(a, b)

wf.py

from flytekit import workflow
from .main import main_task


@workflow
def wf(a: int = 10, b: int = 9):
    return main_task(a=a, b=b)

Dockerfile

FROM <http://ghcr.io/flyteorg/flytekit:py3.9-latest|ghcr.io/flyteorg/flytekit:py3.9-latest>

# Copy the actual code
COPY wf /root/project/wf

Ran these two commands in `project`'s parent directory:

pyflyte --pkgs project package --image <http://ghcr.io/samhita-alla/flyte-dir-structure:0.0.1|ghcr.io/samhita-alla/flyte-dir-structure:0.0.1> -f
flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1

Hi @Samhita Alla thanks a lot for taking the time to create this example - much appreciated! A quick question though - we use the flyte packaging process as part of a CI/CD pipeline for our clients and typically our clients create projects with wf.py/main.py at the top of their project dir. This causes issues with relative imports (

ImportError: attempted relative import with no known parent package

) - is there any way to not use relative imports? i.e.

from main import main_task

and

from src.helpers import sum

rather than

from .main…

and

from .src…

?

Can you share your Dockerfile?

Sure (note that the

sidetrek/base-flyte

image this Dockerfile is based on just has flytectl installed and flyte-config file):

# Build stage
FROM sidetrek/base-flyte:latest as build

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

# Make sure to use venv
ENV PATH="$VENV/bin:$PATH"

COPY ./project/requirements.txt /root
# Add --no-cache-dir to prevent OOMKilled
RUN pip install --no-cache-dir -r /root/requirements.txt

# Production stage
FROM sidetrek/base-flyte:latest

WORKDIR /root
ENV VENV /opt/venv
ENV LANG C.UTF-8
ENV LC_ALL C.UTF-8
ENV PYTHONPATH /root

# Add flytectl to PATH
ENV PATH="/bin/flytectl:$PATH"

# Make sure to use venv
ENV PATH="$VENV/bin:$PATH"

# Copy dependencies from build stage
COPY --from=build /opt/venv /opt/venv

# Copy the actual code (again, user's project code)
COPY . /root

# This tag is supplied by the build script and will be used to determine the version
# when registering tasks, workflows, and launch plans
ARG tag
ENV FLYTE_INTERNAL_IMAGE $tag

@Eduardo Apolinario (eapolinario), is it possible to package tasks and workflows without using relative imports, when the code is spread across multiple modules?

@Samhita Alla, sorry, relative imports should work. Also curious about the dir structure. @seunggs, do you have a

___init___.py

file in

src

?

Yes there’s

__init__.py

in

/src

You're packaging

wf_x

but you need to access

src

while executing your code. Try packaging

wf

.

But if you set the --source to

project/wf

, this errors out saying there’s no module named

wf

(probably because you’re already in that dir)

Can you not specify

--source

, and can you run this command in `project`'s parent directory?

pyflyte --pkgs project package --image <your-image> -f

The

__init__.py

file needs to be present in

wf

, not

project

. Ensure the Dockerfile is copying the code in

wf

to

/root/project/wf

.

@Samhita Alla Just tried it and same error:

Loading packages ['project'] under source root /workspace/source
No module named 'src'
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'src'

/workspace/source

is the root folder inside CI/CD where

/project

is. Confirmed that the structure is this inside the CI/CD:

/project
  /wf
    /src
      __init__.py
      helpers.py
    __init__.py
    main.py
    wf.py

And ran

pyflyte --pkgs project package …

without

--source

from parent of

/project

. Using relative import - i.e. in

main.py

,

from .src.helpers import some_fn

It should work! Also if you're using a relative import, the error needs to be no module name ".src". Not sure why you're seeing "src".

Hmm that’s an interesting point - so it’s not packaging /src folder for some reason. But I’m setting --pkgs to

project

- shouldn’t that package all the subdirectories and files?

No, you can specify a directory. I'm just wondering if the CI/CD is picking up the relative import. Can you try it locally and see if that works?

@Samhita Alla Testing relative import locally throws this error:

attempted relative import with no known parent package

Can we connect on a call?

Hi @Samhita Alla - I can chat tomorrow anytime between 8-11am, 3-5pm, 6-8pm PT. But before we get on a call, I just wanted to recap the problem at hand. To recap, due to

attempted relative import with no known parent package

error, I wasn’t able to get relative import working. For various reasons, I don’t think we can use the relative imports for our needs. The only way I could get this to work was to do something like

from <http://project.wf|project.wf>.src.helpers import some_fn

when I run my project in the parent dir of

project

(with no --source and

__init__.py

in

/project

and in

/wf

). This is an acceptable workaround for now, but I’d much rather be able to set

/project/wf

dir as project root for python import purposes. So I’d love to get on a call if you can help me find a way to do this, but otherwise, I am not sure I should waste any more of your time. Let me know if you think it’s still worth it to get on a call. Either way, thank you so much for your help regarding this problem. It’s much appreciated!!

The only way I could get this to work was to do something like

from <http://project.wf|project.wf>.src.helpers import some_fn

when I run my project in the parent dir of

project

(with no --source and

__init__.py

in

/project

and in

/wf

).

Good to know that this is working for you.

This is an acceptable workaround for now, but I’d much rather be able to set

/project/wf

dir as project root for python import purposes.

You can set

wf

as the project root if all your workflows and dependent tasks and libraries are present in the same folder. If not, this won't work. e.g. main.py

from wf.src.helpers import sum
from flytekit import task


@task
def main_task(a: int, b: int) -> int:
    return sum(a, b)

wf.py

from flytekit import workflow
from wf.main import main_task


@workflow
def wf(a: int = 10, b: int = 9):
    return main_task(a=a, b=b)

I'm able to successfully package code when I run

pyflyte package

command in the

project

directory. Also, I agree that relative import isn't a feasible import technique. You should either import from the project root or do relative imports. Let me know if this still isn't clear to you and we can hop on a call!

I think I get it but I’m just curious - why is it not possible to set

/wf

as project root if there are subdirectories? This forces a user to either have a specific dir structure (i.e.

/project/wf

), which requires a prefix for all imports (i.e.

from <http://project.wf|project.wf>.main…

) or if you make

/wf

project root, you have to not have any subdirectories, which prevents the user from having any modularized code. Either way, flyte loses parity with a regular python code.

There definitely can exist subdirectories in

wf

.

Oh I misunderstood you then - “You can set

wf

as the project root if all your workflows and dependent tasks and libraries are present in the same folder. If not, this won’t work.”

You can have any number of folders within

wf

and that should work!

You mean all the project files has to be in

wf

- but I tried this and it doesn’t work

Oh it should!

If you put

/project/wf

as a --src so it can be the project root, then what would --pkgs value be?

You'll need to set project root as

project

not

project/wf

'cause you need to package the whole

wf

directory

In that case,

from main …

doesn’t work

Yeah!

So either way, all imports require prefixes

That's because when you're packaging your code from within a directory, the paths have to be relative to that directory, or relative to all other modules. That's how pyflyte traverses the directory.