able-ice-99180
08/01/2022, 5:35 PMbasic_workflow.py
to import a module function instead of have it defined directly. Here is the workflow code:
basic_workflow.py
(modified):
import typing
from typing import Tuple
from example import example_fn
from flytekit import task, workflow
@task
def t1(a: int) -> typing.NamedTuple("OutputsBC", t1_int_output=int, c=str):
# return a + 2, "world"
return example_fn(a)
@task
def t2(a: str, b: str) -> str:
return b + a
@workflow
def module_wf(a: int, b: str) -> Tuple[int, str]:
x, y = t1(a=a)
d = t2(a=y, b=b)
return x, d
(notice from example import example_fn
)
example.py
(the imported module):
from typing import Tuple
def example_fn(a: int) -> Tuple[int, str]:
return a + 2, "world"
When I execute:
pyflyte run --remote basic_workflow.py module_wf --a 5 --b hello
I get the error:
ModuleNotFoundError: No module named 'example'
broad-monitor-993
08/01/2022, 6:42 PMable-ice-99180
08/01/2022, 6:43 PM~/workspace/flyte/experiments/module_test
$ l
total 16
-rw-r--r-- 1 szumowskit1 288552604 0 Aug 1 13:18 __init__.py
drwxr-xr-x 15 szumowskit1 288552604 480 Aug 1 13:19 ..
-rw-r--r-- 1 szumowskit1 288552604 94 Aug 1 13:25 example.py
-rw-r--r-- 1 szumowskit1 288552604 418 Aug 1 13:25 basic_workflow.py
drwxr-xr-x 5 szumowskit1 288552604 160 Aug 1 14:43 .
able-ice-99180
08/01/2022, 6:43 PMpyflyte
was executed from that directorythankful-minister-83577
thankful-minister-83577
run
command, was that it was a light-weight thing that would help people quickly iterate on small self-contained scriptsthankful-minister-83577
run
something, that script gets zipped up and put somewhere, and when the task runs on the backend, it gets pulled by downthankful-minister-83577
thankful-minister-83577
pyflyte register
instead?thankful-minister-83577
thankful-minister-83577
my_wf --input_a foo
component to the command, just leave that offable-ice-99180
08/01/2022, 7:25 PMflytectl register
?able-ice-99180
08/01/2022, 7:26 PM$ flytectl register files --project=flytesnacks --domain=development basic_workflow.py
Error: input package have some invalid files. try to run pyflyte package again [basic_workflow.py]
able-ice-99180
08/01/2022, 7:27 PMyou’ve been running Flyte workflows as one-off scripts, which is useful for quick prototyping and iteration of small ideas on a Flyte cluster
if you need to build a larger Flyte app with sub-modules or sub-packages to organize the logic of your tasks and workflows
...
thankful-minister-83577
pyflyte register
thankful-minister-83577
able-ice-99180
08/01/2022, 7:30 PM$ pyflyte register basic_workflow.py
Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
Try 'pyflyte --help' for help.
Error: No such command 'register'.
able-ice-99180
08/01/2022, 8:14 PM$ pyflyte --pkgs module_test package --image <http://gcr.io/urbn-data-science/flytekit-test-wrapper:latest|gcr.io/urbn-data-science/flytekit-test-wrapper:latest>
Register:
$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
where module_test
is the directory listed above.
Then after kicking off a run of that workflow in the console, I get the same error:
ModuleNotFoundError: No module named 'example'
Granted, I can package up the code in the Docker build. But was curious if there was a way to serialize module imports instead of having to bake it in the Docker image.broad-monitor-993
08/01/2022, 8:38 PMable-ice-99180
08/01/2022, 8:40 PMFROM <http://gcr.io/deeplearning-platform-release/pytorch-gpu:latest|gcr.io/deeplearning-platform-release/pytorch-gpu:latest>
# FROM python:3.8
# FROM <http://gcr.io/urbn-data-science/flytekit-test-wrapper:latest|gcr.io/urbn-data-science/flytekit-test-wrapper:latest>
WORKDIR /root
ENV PYTHONPATH /root
RUN pip install gsutil
# Pod tasks should be exposed in the default image
RUN pip install -U flytekit flytekitplugins-pod
# Required for gsutil to work with workload-identity
RUN echo '[GoogleCompute]\nservice_account = default' > /etc/boto.cfg
ENV FLYTE_INTERNAL_IMAGE "FOOBAR"
It's a copy/paste of the flytekit Dockerfile.python3.8 file, but layered on a GCP deep learning GPU image.
This I am able to use running single-script pyflyte
calls, such as:
pyflyte run --image <http://gcr.io/urbn-data-science/flytekit-test-wrapper:latest|gcr.io/urbn-data-science/flytekit-test-wrapper:latest> --remote basic_workflow.py module_wf --a 5 --b hello
broad-monitor-993
08/01/2022, 8:41 PMable-ice-99180
08/01/2022, 8:42 PMpyflyte run
runs, it does not need to be. But for ones that import modules, I suspect it needs to be baked into the Docker image build?
Following the docs, I noticed the Dockerfile auto-generated here copies the code in. But wasn't sure if that was always needed or not.broad-monitor-993
08/01/2022, 8:46 PMpyflyte package
then register with flytectl register
) you need to bake in the source code into the Docker container.
For pyflyte run
I don’t think this is the case … it uses “fast serialization” to load your source code from blob store when the task is executed.
Not sure about `pyflyte register`… I’m guessing source code needs to be included in the Docker container.broad-monitor-993
08/01/2022, 8:48 PMForI suspect this is why you originally got theI don’t think this is the case … it uses “fast serialization” to load your source code from blob store when the task is executed.pyflyte run
No module named 'example'
error… pyflyte run
doesn’t support user-defined importsable-ice-99180
08/01/2022, 8:49 PMbroad-monitor-993
08/01/2022, 8:52 PMpyflyte run
is for fast prototyping and demo use cases… for anything more serious/complex pyflyte register
or pyflyte package + flytectl register
is probably the way to go.broad-monitor-993
08/01/2022, 8:52 PM```$ pyflyte register basic_workflow.py
Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
Try 'pyflyte --help' for help.
Error: No such command 'register'.```I think you need
flytekit >= 1.1.0
for that commandable-ice-99180
08/01/2022, 9:15 PMpyflyte register
worked without needing to bake the code into the Docker image. 🎉
Thank you both!