Hi all!
Can you please advise me an example with hello world workflow on Flyte where at least two tasks executed on their own containers?
With single python workflow code on flyte/workflow/example.py and such task option:
@task(container_image="<http://registry.name.com/project/image:tag|registry.name.com/project/image:tag>")
I am trying to build this example by myself on my local Flyte sandbox, but getting error messages:
ModuleNotFoundError: No module named 'flyte'
My workflow from example.py
@task(container_image="<http://registry.name.com/project/image:tag|registry.name.com/project/image:tag>")
def some_data_generation() -> PythonPickledFile:
with open(BASE_FILE_PATH) as file:
some_descriptors = json.load(file)
some_set = generate_some_set(some_descriptors[0])
with open(PICKLE_PATH, 'wb') as handle:
pickle.dump(some_set, handle)
return PICKLE_PATH
@task(container_image="<http://registry.name.com/project/image2:tag|registry.name.com/project/image2:tag>")
def load_pickle_dump(dump_file_path: PythonPickledFile) -> set:
with open(dump_file_path, 'rb') as handle:
return pickle.load(handle)
@workflow
def my_wf() -> set:
dump_file_path = some_data_generation()
return load_pickle_dump(dump_file_path=dump_file_path)
if __name__ == "__main__":
a = my_wf()
print(f"Running my_wf() {a}")
What was localised:
1. Workflow works fine without containers (without @task(container_image= )
2. Images works fine if workflow contain only one task with Docker image where included Flyte Workflow folder with this file (example.py)
3. Problem appears if i build Docker Image for first task without Flyte workflow files inside (but with initial data for first files to skip downloading).
4. I am sure - i can skip second task at all from this test case, problem should appear. Root cause - i have Docker image without Flute workflow (example.py) but it seems that this code is required inside the container to be executed. I do not understand how can i split example.py between two tasks if it should be executed actually outside the tasks (because this is a workflow, if should contain tasks inside it according to example)
Error:
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[j3nl8bafcr-n0-0] terminated with exit code (1). Reason [Error]. Message:
thon3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 460, in execute_task_cmd
_execute_task(
File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
return wrapped(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 327, in _execute_task
_task_def = resolver_obj.load_task(loader_args=resolver_args)
File "/opt/venv/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 189, in load_task
task_module = importlib.import_module(task_module)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'flyte'
As i see - it caused by Docker Container without a Flyte workflow. But if i will place workflow in first task - how it should be correctly splitted between two tasks? Should not include code for first task inside the second task image/container?
Am i on correct way at all?
Can you please advise a simple short workflow example on Python with two tasks on separate containers?
