@ancient-wolf-19325 I got the following error when running a workflow with

--remote

but not without this flag:

RecursionError: maximum recursion depth exceeded

. Here is the detailed traceback:

Trace:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.12/pathlib.py", line 441, in __str__
        return self._str
               ^^^^^^^^^
    AttributeError: 'PosixPath' object has no attribute '_str'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/usr/local/lib/python3.12/pathlib.py", line 555, in drive
        return self._drv
               ^^^^^^^^^
    AttributeError: 'PosixPath' object has no attribute '_drv'

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/.venv/lib/python3.12/site-packages/flytekit/bin/entrypoint.py", line 164, in _dispatch_execute
        task_def = load_task()
                   ^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/bin/entrypoint.py", line 583, in load_task
        return resolver_obj.load_task(loader_args=resolver_args)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/utils.py", line 312, in wrapper
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/python_auto_container.py", line 271, in load_task
        task_module = importlib.import_module(name=task_module)  # type: ignore
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/importlib/__init__.py", line 90, in import_module
        return _bootstrap._gcd_import(name[level:], package, level)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
      File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
      File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
      File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
      File "<frozen importlib._bootstrap_external>", line 999, in exec_module
      File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
      File "/workflows/pipeline.py", line 12, in <module>
        @task()
         ^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/task.py", line 359, in wrapper
        task_instance = TaskPlugins.find_pythontask_plugin(type(task_config))(
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/tracker.py", line 82, in __call__
        o = super(InstanceTrackingMeta, cls).__call__(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/python_function_task.py", line 139, in __init__
        name, _, _, _ = extract_task_module(task_function)
                        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/tracker.py", line 382, in extract_task_module
        mod_name = get_full_module_path(mod, mod_name)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/tracker.py", line 391, in get_full_module_path
        new_mod_name = _mod_sanitizer.get_absolute_module_name(inspect.getabsfile(mod), package_root)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/tracker.py", line 328, in get_absolute_module_name
        return self._resolve_abs_module_name(path, package_root)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/tracker.py", line 318, in _resolve_abs_module_name
        mod_name = self._resolve_abs_module_name(dirname, package_root)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/tracker.py", line 318, in _resolve_abs_module_name
        mod_name = self._resolve_abs_module_name(dirname, package_root)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/.venv/lib/python3.12/site-packages/flytekit/core/tracker.py", line 318, in _resolve_abs_module_name
        mod_name = self._resolve_abs_module_name(dirname, package_root)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      [Previous line repeated 964 more times]
      File "/.venv/lib/python3.12/site-packages/flytekit/core/tracker.py", line 294, in _resolve_abs_module_name
        if not Path(dirname).is_dir():
               ^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/pathlib.py", line 875, in is_dir
        return S_ISDIR(self.stat().st_mode)
                       ^^^^^^^^^^^
      File "/usr/local/lib/python3.12/pathlib.py", line 840, in stat
        return os.stat(self, follow_symlinks=follow_symlinks)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/pathlib.py", line 448, in __fspath__
        return str(self)
               ^^^^^^^^^
      File "/usr/local/lib/python3.12/pathlib.py", line 443, in __str__
        self._str = self._format_parsed_parts(self.drive, self.root,
                                              ^^^^^^^^^^
      File "/usr/local/lib/python3.12/pathlib.py", line 557, in drive
        self._load_parts()
      File "/usr/local/lib/python3.12/pathlib.py", line 415, in _load_parts
        drv, root, tail = self._parse_path(path)
                          ^^^^^^^^^^^^^^^^^^^^^^
      File "/usr/local/lib/python3.12/pathlib.py", line 395, in _parse_path
        drv, root, rel = cls._flavour.splitroot(path)
                         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    RecursionError: maximum recursion depth exceeded

Message:

    RecursionError: maximum recursion depth exceeded

What can I do to make it work?

The

RecursionError

you're encountering when running the workflow with the

--remote

flag suggests an issue with module path resolution in Flyte's task tracking system. This error often occurs due to an infinite loop in resolving module paths, which can be influenced by the environment or the way modules are structured. To address this issue, consider the following steps: 1. Check Module Structure: Ensure that your Python modules and packages are correctly structured and contain

__init__.py

files where necessary. This helps Python recognize directories as packages. 2. Verify Python Path: Make sure that the Python path is correctly set up in your environment. You can print the Python path using

import sys; print(sys.path)

to verify that all necessary directories are included. 3. Environment Consistency: Ensure that the local and remote environments are consistent in terms of Python version and package dependencies. Differences in these environments can lead to unexpected behavior. 4. Increase Recursion Limit: As a temporary workaround, you can increase the recursion limit using

sys.setrecursionlimit(new_limit)

, but this is not a recommended long-term solution as it may mask underlying issues. Would you like more detailed guidance on any of these steps? Sources: - Flyte GitHub Issue #4853 👍 0 button 👎 0 button 💡 Hint: Mention @runllm in the thread for followups.

cc @thankful-minister-83577 this is because maybe the object is recursive somehow?

is this happening in the container that’s being run? or on the host where pyflyte run --remote is being called?