<#4245 [BUG] Dynamic workflows: Module from file /...
# flyte-github
a
#4245 [BUG] Dynamic workflows: Module from file /usr/local/bin/pyflyte-execute cannot be loaded Issue created by iirekm Describe the bug
Copy code
import sys
from typing import List
from flytekit import task, workflow, dynamic, Resources
from flytekit.remote import *
from flytekit.configuration import Config

@task(requests=Resources(mem="500Mi"))
def add1(x: int) -> int:
    return x + 1

@dynamic(requests=Resources(mem="500Mi"))
def q(n: int) -> int:
    ctr = 0
    for _ in range(n):
        ctr = add1(x=ctr)
    return ctr

@task
def fin(x: int):
    print(x)

@workflow
def workflow():
    fin(x=q(n=5))
• calling
workflow()
locally - works (prints 5) •
pyflyte run .../script.py workflow
fails with
Module from file /home/.../.venv/bin/pyflyte cannot be loaded
•
pyflyte run --remote --image ...
fails with similar
Module from file /usr/local/bin/pyflyte-execute cannot be loaded
• however the following (copied from pyflyte script) works (!!!):
Copy code
if __name__ == "__main__":
    from flytekit.clis.sdk_in_container.pyflyte import main
    sys.argv = "pyflyte run my.package.dynamic_demo. workflow".split(" ")
    main()
• however when I add
--remote --image ...
here - doesn't work again. What's going on here?
@task
,
@workflow
work as expected. There's only problem with
@dynamic
. Python: 3.11 Flytekit: 1.9.1 Full stack trace: ``` 2023-10-16 195034,538925 ERROR 2023-10-16 195034,538 flytekit ERROR Exception when executing Module from file base_task.py:612 /home/.../.venv/bin/pyflyte cannot be loaded ╭──────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────╮ │ /home/.../demos/nnets/.venv/lib/python3.11/site-packages/flytekit/core/tracker.py:18 │ │ in import_module_from_file │ │ │ │ 15 def import_module_from_file(module_name, file): │ │ 16 │ try: │ │ 17 │ │ spec = importlib.util.spec_from_file_location(module_name, file) │ │ ❱ 18 │ │ module = importlib.util.module_from_spec(spec) │ │ 19 │ │ spec.loader.exec_module(module) │ │ 20 │ │ return module │ │ 21 │ except AssertionError: │ │ in module_from_spec:570 │ ╰────────────────────────────────────────────────────────────────────────────────────────────────────────────╯ AttributeError: 'NoneType' object has no attribute 'loader' The above exception was the direct cause of the following exception: ╭──────────────────────────────────── Traceback (most recent call last) ─────────────────────────────────────╮ │ /home/.../.venv/lib/python3.11/site-packages/flytekit/core/base_task.py:6 │ │ 10 in dispatch_execute │ │ │ │ 607 │ │ │ logger.info(f"Invoking {self.name} with inputs: {native_inputs}") │ │ 608 │ │ │ try: │ │ 609 │ │ │ │ with timeit("Execute user level code"): │ │ ❱ 610 │ │ │ │ │ native_outputs = self.execute(**native_inputs) │ │ 611 │ │ │ except Exception as e: │ │ 612 │ │ │ │ logger.exception(f"Exception when executing {e}") │ │ 613 │ │ │ │ raise e │ │ │ │ /home/.../.venv/lib/python3.11/site-packages/flytekit/core/python_functio │ │ n_task.py:180 in execute │ │ │ │ 177 │ │ │ kwargs["async_ctx"] = FlyteContextManager.current_context() │ │ 178 │ │ │ return exception_scopes.user_entry_point(self._task_function)(**kwargs) │ │ 179 │ │ elif self.execution_mode == self.ExecutionBehavior.DYNAMIC: │ │ ❱ 180 │ │ │ return self.dynamic_execute(self._task_function, **kwargs) │ │ 181 │ │ │ 182 │ def _create_and_cache_dynamic_workflow(self): │ │ 183 │ │ if self._wf is None: │ │ … flyteorg/flyte