Tom Szumowski
08/01/2022, 5:35 PMbasic_workflow.py
to import a module function instead of have it defined directly. Here is the workflow code:
basic_workflow.py
(modified):
import typing
from typing import Tuple
from example import example_fn
from flytekit import task, workflow
@task
def t1(a: int) -> typing.NamedTuple("OutputsBC", t1_int_output=int, c=str):
# return a + 2, "world"
return example_fn(a)
@task
def t2(a: str, b: str) -> str:
return b + a
@workflow
def module_wf(a: int, b: str) -> Tuple[int, str]:
x, y = t1(a=a)
d = t2(a=y, b=b)
return x, d
(notice from example import example_fn
)
example.py
(the imported module):
from typing import Tuple
def example_fn(a: int) -> Tuple[int, str]:
return a + 2, "world"
When I execute:
pyflyte run --remote basic_workflow.py module_wf --a 5 --b hello
I get the error:
ModuleNotFoundError: No module named 'example'
Niels Bantilan
08/01/2022, 6:42 PMTom Szumowski
08/01/2022, 6:43 PM~/workspace/flyte/experiments/module_test
$ l
total 16
-rw-r--r-- 1 szumowskit1 288552604 0 Aug 1 13:18 __init__.py
drwxr-xr-x 15 szumowskit1 288552604 480 Aug 1 13:19 ..
-rw-r--r-- 1 szumowskit1 288552604 94 Aug 1 13:25 example.py
-rw-r--r-- 1 szumowskit1 288552604 418 Aug 1 13:25 basic_workflow.py
drwxr-xr-x 5 szumowskit1 288552604 160 Aug 1 14:43 .
Tom Szumowski
08/01/2022, 6:43 PMpyflyte
was executed from that directoryYee
Yee
run
command, was that it was a light-weight thing that would help people quickly iterate on small self-contained scriptsYee
run
something, that script gets zipped up and put somewhere, and when the task runs on the backend, it gets pulled by downYee
Yee
pyflyte register
instead?Yee
Yee
my_wf --input_a foo
component to the command, just leave that offTom Szumowski
08/01/2022, 7:25 PMflytectl register
?Tom Szumowski
08/01/2022, 7:26 PM$ flytectl register files --project=flytesnacks --domain=development basic_workflow.py
Error: input package have some invalid files. try to run pyflyte package again [basic_workflow.py]
Tom Szumowski
08/01/2022, 7:27 PMyou’ve been running Flyte workflows as one-off scripts, which is useful for quick prototyping and iteration of small ideas on a Flyte cluster
if you need to build a larger Flyte app with sub-modules or sub-packages to organize the logic of your tasks and workflows
...
Yee
pyflyte register
Yee
Tom Szumowski
08/01/2022, 7:30 PM$ pyflyte register basic_workflow.py
Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
Try 'pyflyte --help' for help.
Error: No such command 'register'.
Tom Szumowski
08/01/2022, 8:14 PM$ pyflyte --pkgs module_test package --image <http://gcr.io/urbn-data-science/flytekit-test-wrapper:latest|gcr.io/urbn-data-science/flytekit-test-wrapper:latest>
Register:
$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
where module_test
is the directory listed above.
Then after kicking off a run of that workflow in the console, I get the same error:
ModuleNotFoundError: No module named 'example'
Granted, I can package up the code in the Docker build. But was curious if there was a way to serialize module imports instead of having to bake it in the Docker image.Niels Bantilan
08/01/2022, 8:38 PMTom Szumowski
08/01/2022, 8:40 PMFROM <http://gcr.io/deeplearning-platform-release/pytorch-gpu:latest|gcr.io/deeplearning-platform-release/pytorch-gpu:latest>
# FROM python:3.8
# FROM <http://gcr.io/urbn-data-science/flytekit-test-wrapper:latest|gcr.io/urbn-data-science/flytekit-test-wrapper:latest>
WORKDIR /root
ENV PYTHONPATH /root
RUN pip install gsutil
# Pod tasks should be exposed in the default image
RUN pip install -U flytekit flytekitplugins-pod
# Required for gsutil to work with workload-identity
RUN echo '[GoogleCompute]\nservice_account = default' > /etc/boto.cfg
ENV FLYTE_INTERNAL_IMAGE "FOOBAR"
It's a copy/paste of the flytekit Dockerfile.python3.8 file, but layered on a GCP deep learning GPU image.
This I am able to use running single-script pyflyte
calls, such as:
pyflyte run --image <http://gcr.io/urbn-data-science/flytekit-test-wrapper:latest|gcr.io/urbn-data-science/flytekit-test-wrapper:latest> --remote basic_workflow.py module_wf --a 5 --b hello
Niels Bantilan
08/01/2022, 8:41 PMTom Szumowski
08/01/2022, 8:42 PMpyflyte run
runs, it does not need to be. But for ones that import modules, I suspect it needs to be baked into the Docker image build?
Following the docs, I noticed the Dockerfile auto-generated here copies the code in. But wasn't sure if that was always needed or not.Niels Bantilan
08/01/2022, 8:46 PMpyflyte package
then register with flytectl register
) you need to bake in the source code into the Docker container.
For pyflyte run
I don’t think this is the case … it uses “fast serialization” to load your source code from blob store when the task is executed.
Not sure about `pyflyte register`… I’m guessing source code needs to be included in the Docker container.Niels Bantilan
08/01/2022, 8:48 PMForI suspect this is why you originally got theI don’t think this is the case … it uses “fast serialization” to load your source code from blob store when the task is executed.pyflyte run
No module named 'example'
error… pyflyte run
doesn’t support user-defined importsTom Szumowski
08/01/2022, 8:49 PMNiels Bantilan
08/01/2022, 8:52 PMpyflyte run
is for fast prototyping and demo use cases… for anything more serious/complex pyflyte register
or pyflyte package + flytectl register
is probably the way to go.Niels Bantilan
08/01/2022, 8:52 PM```$ pyflyte register basic_workflow.py
Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
Try 'pyflyte --help' for help.
Error: No such command 'register'.```I think you need
flytekit >= 1.1.0
for that commandTom Szumowski
08/01/2022, 9:15 PMpyflyte register
worked without needing to bake the code into the Docker image. 🎉
Thank you both!