average-winter-43654
07/19/2023, 3:04 PMexample.py
. If I run python example.py
, it will work correctly. If I run pyflyte run (--remote) example.py wf
, I will get this error message Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'mike'
.
• I also tried to move these files to the my_project
directory and use setup.py
to build and install the package in the Dockerfile, and I got the same error.
• Plus, I did not see docker_build.sh
in the directory after executing pyflyte init my_project
• Please check more details in the thread. Thanks for your guidance.average-winter-43654
07/19/2023, 3:05 PM(my_project) ➜ my_project git:(main) ✗ tree
.
├── Dockerfile
├── LICENSE
├── README.md
├── requirements.txt
└── workflows
├── __init__.py
├── __pycache__
│ ├── __init__.cpython-311.pyc
│ └── example.cpython-311.pyc
├── example.py
└── mike
├── __pycache__
│ └── demo.cpython-311.pyc
└── demo.py
average-winter-43654
07/19/2023, 3:06 PM(my_project) ➜ my_project git:(main) ✗ cat workflows/mike/demo.py
def add(a, b):
return a + b
def minus(a, b):
return a - b
average-winter-43654
07/19/2023, 3:06 PM"""A simple Flyte example."""
import typing
from flytekit import task, workflow
from mike.demo import add, minus
@task
def say_hello(name: str) -> str:
"""A simple Flyte task to say "hello".
The @task decorator allows Flyte to use this function as a Flyte task, which
is executed as an isolated, containerized unit of compute.
"""
print(f"1 + 2 = {add(1, 2)}")
return f"hello {name}!"
@task
def greeting_length(greeting: str) -> int:
"""A task the counts the length of a greeting."""
print(f"3 - 1 = {minus(3, 1)}")
return len(greeting)
@workflow
def wf(name: str = "union") -> typing.Tuple[str, int]:
"""Declare workflow called `wf`.
The @workflow decorator defines an execution graph that is composed of tasks
and potentially sub-workflows. In this simple example, the workflow is
composed of just one task.
There are a few important things to note about workflows:
- Workflows are a domain-specific language (DSL) for creating execution
graphs and therefore only support a subset of Python's behavior.
- Tasks must be invoked with keyword arguments
- The output variables of tasks are Promises, which are placeholders for
values that are yet to be materialized, not the actual values.
"""
greeting = say_hello(name=name)
greeting_len = greeting_length(greeting=greeting)
return greeting, greeting_len
if __name__ == "__main__":
# Execute the workflow, simply by invoking it like a function and passing in
# the necessary parameters
print(f"Running wf() { wf(name='passengers') }")
average-winter-43654
07/19/2023, 3:08 PMDockerfile
average-winter-43654
07/19/2023, 3:11 PM(my_project) ➜ my_project git:(main) ✗ python workflows/example.py
1 + 2 = 3
3 - 1 = 2
Running wf() DefaultNamedTupleOutput(o0='hello passengers!', o1=17)
(my_project) ➜ my_project git:(main) ✗ pyflyte run workflows/example.py wf
Failed with Unknown Exception <class 'ModuleNotFoundError'> Reason: No module named 'mike'
No module named 'mike'
kind-kite-58745
07/19/2023, 4:20 PMworkflows
folder?
I also set up a flyte PoC last month, I’ll share an alternative way to handle these issues which worked for me. I don’t work with pyflyte cli, instead I do everything from python
There’s a FlyteRemote class that lets you interact with the flyte remote from python. I also had some import issues while I was setting up my PoC, and in the end I settled on using FlyteRemote.register_script
in my class that wraps around FlyteRemote:
class FlyteClient:
def __init__(self, endpoint: str, project: str, **kwargs):
...
self.remote = FlyteRemote(
config=Config.for_endpoint(endpoint=self.endpoint),
default_project=self.project,
default_domain=self.domain,
)
...
def register_script(self, workflow: FlyteWorkflow) -> FlyteWorkflow:
main_script_path = os.path.abspath(__main__.__file__)
source_dir = os.path.dirname(main_script_path)
# some custom use-case-specific logic
if os.path.basename(source_dir) == "scripts":
source_path = os.path.dirname(source_dir)
else:
source_path = source_dir
return self.remote.register_script(
entity=workflow,
version=self._uid,
#module_name=workflow.name,
source_path=source_path,
copy_all=True
)
In this example source_path lets you decide which folder of python code files will be copied into the remote container that will execute the task. The files will be available for importing and running during the execution. This specific example lets you run python example.py from anywhere and it will rely on the file’s path instead of your local path where you are currently cd’ed to in your command line session, so you can have more consistent behavior. This example doesn’t take into account the case when you are running directly from python console (where there is no __main__.___file___
but it can be adjusted to handle that case also if needed
register_script is cool because if you pass a FlyteWorkflow to it, it will register both the workflow and all its children tasks with the flyte remote in one operation, instead of doing register_workflow and then register_task multiple times.
After the workflow is registered, register_script returns it as its return value, and it can then be used with FlyteRemote.execute() to start an execution
Running cli commands seems to be the preferred way to interact with the flyte remote from what I’ve seen, not as many users use flyteremote compared to pyflyte based on the discussions in this channel so far. I hope someone who is experienced with the pyflyte cli gives you some guidance on that, but I wanted to let you know that this can also be done from python which may be preferred for some use cases, for example in my specific case it is much more suitable than cli commands
So if you prefer working with flyte remote programmatically, that’s an option and I can help you get started with it because I’ve already set it up for myself. Whether or not you should use cli commands or FlyteRemote class is up to your preferences and specific use case
Read more: https://docs.flyte.org/projects/flytekit/en/latest/design/control_plane.htmlaverage-winter-43654
07/19/2023, 5:01 PMkind-kite-58745
07/19/2023, 5:05 PMbroad-monitor-993
07/19/2023, 6:34 PMNo module named 'mike'
You’ll need to provide the fully qualified module name in your import:
from workflows.mike.demo import ...
broad-monitor-993
07/19/2023, 6:42 PMsimple-example
directory contains the cookie cutter flyte project that you get when you do pyflyte init …
I just added back the docker_build.sh
script in there.broad-monitor-993
07/19/2023, 6:43 PMbroad-monitor-993
07/19/2023, 6:47 PMBTW, do you have any idea how good Flyte is if we run it locally? I mean can it still orchestrate a fairly complex workflow without using K8s and docker container?You can use the local sandbox for initial testing and debugging, and you can certainly do things like schedule workflows on your local computer… but you won’t get a lot of the benefits of scale and reproducibility doing that.
average-winter-43654
07/21/2023, 9:19 AM