astonishing-airport-86417
04/02/2022, 11:42 AM@task(container_image="<http://registry.name.com/project/image:tag|registry.name.com/project/image:tag>")
I am trying to build this example by myself on my local Flyte sandbox, but getting error messages:
ModuleNotFoundError: No module named 'flyte'
My workflow from example.py
@task(container_image="<http://registry.name.com/project/image:tag|registry.name.com/project/image:tag>")
def some_data_generation() -> PythonPickledFile:
with open(BASE_FILE_PATH) as file:
some_descriptors = json.load(file)
some_set = generate_some_set(some_descriptors[0])
with open(PICKLE_PATH, 'wb') as handle:
pickle.dump(some_set, handle)
return PICKLE_PATH
@task(container_image="<http://registry.name.com/project/image2:tag|registry.name.com/project/image2:tag>")
def load_pickle_dump(dump_file_path: PythonPickledFile) -> set:
with open(dump_file_path, 'rb') as handle:
return pickle.load(handle)
@workflow
def my_wf() -> set:
dump_file_path = some_data_generation()
return load_pickle_dump(dump_file_path=dump_file_path)
if __name__ == "__main__":
a = my_wf()
print(f"Running my_wf() {a}")
What was localised:
1. Workflow works fine without containers (without @task(container_image= )
2. Images works fine if workflow contain only one task with Docker image where included Flyte Workflow folder with this file (example.py)
3. Problem appears if i build Docker Image for first task without Flyte workflow files inside (but with initial data for first files to skip downloading).
4. I am sure - i can skip second task at all from this test case, problem should appear. Root cause - i have Docker image without Flute workflow (example.py) but it seems that this code is required inside the container to be executed. I do not understand how can i split example.py between two tasks if it should be executed actually outside the tasks (because this is a workflow, if should contain tasks inside it according to example)
Error:
[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[j3nl8bafcr-n0-0] terminated with exit code (1). Reason [Error]. Message:
thon3.8/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/opt/venv/lib/python3.8/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 460, in execute_task_cmd
_execute_task(
File "/opt/venv/lib/python3.8/site-packages/flytekit/exceptions/scopes.py", line 160, in system_entry_point
return wrapped(*args, **kwargs)
File "/opt/venv/lib/python3.8/site-packages/flytekit/bin/entrypoint.py", line 327, in _execute_task
_task_def = resolver_obj.load_task(loader_args=resolver_args)
File "/opt/venv/lib/python3.8/site-packages/flytekit/core/python_auto_container.py", line 189, in load_task
task_module = importlib.import_module(task_module)
File "/usr/local/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 961, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
File "<frozen importlib._bootstrap>", line 991, in _find_and_load
File "<frozen importlib._bootstrap>", line 973, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'flyte'
As i see - it caused by Docker Container without a Flyte workflow. But if i will place workflow in first task - how it should be correctly splitted between two tasks? Should not include code for first task inside the second task image/container?
Am i on correct way at all?
Can you please advise a simple short workflow example on Python with two tasks on separate containers?freezing-airport-6809
astonishing-airport-86417
04/02/2022, 4:34 PMfreezing-airport-6809
glamorous-carpet-83516
04/02/2022, 6:02 PMglamorous-carpet-83516
04/02/2022, 6:27 PMmyflyteapp
โโโ Dockerfile
โโโ docker_build_and_tag.sh
โโโ flyte
โ โโโ __init__.py
โ โโโ workflows
โ โโโ __init__.py
โ โโโ example.py
โโโ requirements.txt
astonishing-airport-86417
04/02/2022, 6:33 PMastonishing-airport-86417
04/02/2022, 6:35 PMHow should looks example.py content in container1 and container2?
glamorous-carpet-83516
04/02/2022, 6:43 PMhow it should be correctly splitted between two tasks? Should not include code for first task inside the second task image/container?In most of time, tasks will use same image, which means your image will contain code for first task and second task. although we use same image, we will run those tasks in different container (or pods in k8s). In each container, flyte will only execute a function (your task). your donโt need two different dockerfile, one for first task, and another on for second task. The tasks and workflow should be in a single Docker file.
glamorous-carpet-83516
04/02/2022, 6:46 PMglamorous-carpet-83516
04/02/2022, 6:50 PMmyflyteapp
โโโ Dockerfile1
|โโ Dockerfile2
โโโ docker_build_and_tag.sh
โโโ flyte
โ โโโ __init__.py
โ โโโ workflows
โ โโโ __init__.py
โ โโโ example.py
โโโ requirements.txt
astonishing-airport-86417
04/02/2022, 6:51 PMastonishing-airport-86417
04/02/2022, 6:53 PMglamorous-carpet-83516
04/02/2022, 6:53 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
astonishing-airport-86417
04/02/2022, 6:58 PMastonishing-airport-86417
04/02/2022, 7:04 PMglamorous-carpet-83516
04/02/2022, 7:14 PMpython example.py
in the container, we will run pyflyte-execute --task-module flyte.workflow.load_pickle_dump
. Therefore, we can only execute a task in each containerastonishing-airport-86417
04/02/2022, 7:20 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
freezing-airport-6809
astonishing-airport-86417
04/02/2022, 8:12 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809