Tom Szumowski
08/01/2022, 5:35 PMbasic_workflow.py
to import a module function instead of have it defined directly. Here is the workflow code:
basic_workflow.py
(modified):
import typing
from typing import Tuple
from example import example_fn
from flytekit import task, workflow
@task
def t1(a: int) -> typing.NamedTuple("OutputsBC", t1_int_output=int, c=str):
# return a + 2, "world"
return example_fn(a)
@task
def t2(a: str, b: str) -> str:
return b + a
@workflow
def module_wf(a: int, b: str) -> Tuple[int, str]:
x, y = t1(a=a)
d = t2(a=y, b=b)
return x, d
(notice from example import example_fn
)
example.py
(the imported module):
from typing import Tuple
def example_fn(a: int) -> Tuple[int, str]:
return a + 2, "world"
When I execute:
pyflyte run --remote basic_workflow.py module_wf --a 5 --b hello
I get the error:
ModuleNotFoundError: No module named 'example'
Niels Bantilan
08/01/2022, 6:42 PMTom Szumowski
08/01/2022, 6:43 PM~/workspace/flyte/experiments/module_test
$ l
total 16
-rw-r--r-- 1 szumowskit1 288552604 0 Aug 1 13:18 __init__.py
drwxr-xr-x 15 szumowskit1 288552604 480 Aug 1 13:19 ..
-rw-r--r-- 1 szumowskit1 288552604 94 Aug 1 13:25 example.py
-rw-r--r-- 1 szumowskit1 288552604 418 Aug 1 13:25 basic_workflow.py
drwxr-xr-x 5 szumowskit1 288552604 160 Aug 1 14:43 .
pyflyte
was executed from that directoryYee
run
command, was that it was a light-weight thing that would help people quickly iterate on small self-contained scriptsrun
something, that script gets zipped up and put somewhere, and when the task runs on the backend, it gets pulled by downpyflyte register
instead?my_wf --input_a foo
component to the command, just leave that offTom Szumowski
08/01/2022, 7:25 PMflytectl register
?$ flytectl register files --project=flytesnacks --domain=development basic_workflow.py
Error: input package have some invalid files. try to run pyflyte package again [basic_workflow.py]
you’ve been running Flyte workflows as one-off scripts, which is useful for quick prototyping and iteration of small ideas on a Flyte cluster
if you need to build a larger Flyte app with sub-modules or sub-packages to organize the logic of your tasks and workflows
...
Yee
pyflyte register
Tom Szumowski
08/01/2022, 7:30 PM$ pyflyte register basic_workflow.py
Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
Try 'pyflyte --help' for help.
Error: No such command 'register'.
$ pyflyte --pkgs module_test package --image <http://gcr.io/urbn-data-science/flytekit-test-wrapper:latest|gcr.io/urbn-data-science/flytekit-test-wrapper:latest>
Register:
$ flytectl register files --project flytesnacks --domain development --archive flyte-package.tgz --version v1
where module_test
is the directory listed above.
Then after kicking off a run of that workflow in the console, I get the same error:
ModuleNotFoundError: No module named 'example'
Granted, I can package up the code in the Docker build. But was curious if there was a way to serialize module imports instead of having to bake it in the Docker image.Niels Bantilan
08/01/2022, 8:38 PMTom Szumowski
08/01/2022, 8:40 PMFROM <http://gcr.io/deeplearning-platform-release/pytorch-gpu:latest|gcr.io/deeplearning-platform-release/pytorch-gpu:latest>
# FROM python:3.8
# FROM <http://gcr.io/urbn-data-science/flytekit-test-wrapper:latest|gcr.io/urbn-data-science/flytekit-test-wrapper:latest>
WORKDIR /root
ENV PYTHONPATH /root
RUN pip install gsutil
# Pod tasks should be exposed in the default image
RUN pip install -U flytekit flytekitplugins-pod
# Required for gsutil to work with workload-identity
RUN echo '[GoogleCompute]\nservice_account = default' > /etc/boto.cfg
ENV FLYTE_INTERNAL_IMAGE "FOOBAR"
It's a copy/paste of the flytekit Dockerfile.python3.8 file, but layered on a GCP deep learning GPU image.
This I am able to use running single-script pyflyte
calls, such as:
pyflyte run --image <http://gcr.io/urbn-data-science/flytekit-test-wrapper:latest|gcr.io/urbn-data-science/flytekit-test-wrapper:latest> --remote basic_workflow.py module_wf --a 5 --b hello
Niels Bantilan
08/01/2022, 8:41 PMTom Szumowski
08/01/2022, 8:42 PMpyflyte run
runs, it does not need to be. But for ones that import modules, I suspect it needs to be baked into the Docker image build?
Following the docs, I noticed the Dockerfile auto-generated here copies the code in. But wasn't sure if that was always needed or not.Niels Bantilan
08/01/2022, 8:46 PMpyflyte package
then register with flytectl register
) you need to bake in the source code into the Docker container.
For pyflyte run
I don’t think this is the case … it uses “fast serialization” to load your source code from blob store when the task is executed.
Not sure about `pyflyte register`… I’m guessing source code needs to be included in the Docker container.ForI suspect this is why you originally got theI don’t think this is the case … it uses “fast serialization” to load your source code from blob store when the task is executed.pyflyte run
No module named 'example'
error… pyflyte run
doesn’t support user-defined importsTom Szumowski
08/01/2022, 8:49 PMNiels Bantilan
08/01/2022, 8:52 PMpyflyte run
is for fast prototyping and demo use cases… for anything more serious/complex pyflyte register
or pyflyte package + flytectl register
is probably the way to go.```$ pyflyte register basic_workflow.py
Usage: pyflyte [OPTIONS] COMMAND [ARGS]...
Try 'pyflyte --help' for help.
Error: No such command 'register'.```I think you need
flytekit >= 1.1.0
for that commandTom Szumowski
08/01/2022, 9:15 PMpyflyte register
worked without needing to bake the code into the Docker image. 🎉
Thank you both!