General question about python function task (and w...
# flytekit
y
General question about python function task (and workflow) naming and loading conventions: We’re hoping to clean up how this works a bit. Today, they are always determined basically by the path from where the python process was started, to the location of the .py file. For example, if you have
Copy code
:~/dev/my_repo $ tree .
.
├── Dockerfile
└── src
    ├── __init__.py
    ├── __pycache__
    └── parent
        ├── __init__.py
        ├── __pycache__
        └── child
            ├── __init__.py
            ├── __pycache__
            └── hello_world.py
the
__module__
attribute is determined by the path:
Copy code
:~/dev/my_repo $ ipython
In [1]: from src.parent.child.hello_world import my_wf

In [2]: my_wf.__module__
Out[2]: 'src.parent.child.hello_world'


:~/dev/my_repo $ cd src/parent
:~/dev/my_repo/src/parent $ ipython
In [1]: from child.hello_world import my_wf

In [2]: my_wf.__module__
Out[2]: 'child.hello_world'
Note the
__module__
in the second example is shortened. This module name is used for the name of the task, and to load the task at run-time in the container (so your image has to be built correctly of course). We are thinking of changing the behaviour so that, given a task (or launch plan or workflow), we walk up the file system until we find a folder that doesn’t have an
__init__.py
file, and basically assume that the python process was started from there. In the above example, the name in the latter would be the same as in the first example. Obviously this will not work for namespace packages, but we doubt anyone uses those currently for Flyte workflows. Does this sound unreasonable to anyone?
@Miggy and @Ketan (kumare3)
@jeev too maybe when you get a chance
will be settable via switch, and will still default to the current dir in the upcoming release (we’ll default to the go-up-til-no-init-file behaviour at 1.0)
also @Eduardo Apolinario (eapolinario) this is what ketan’s pr is for
111 Views