hi flyte folks so i am upgrading from 1 4 to 1 9 in flytekit Flyte #ask-the-community

hi flyte folks, so i am upgrading from 1.4 to 1.9 ...

Jay Ganbat

10/19/2023, 4:30 PM

hi flyte folks, so i am upgrading from 1.4 to 1.9 in flytekit. We have bunch of local execution e2e tests. However when i try to use pytest with parallel option it crashes with the following error 🧵 but when i ran it stand-alone it works just fine, any idea what could be the issue or what config to modify

Jay Ganbat

10/19/2023, 4:30 PM

Copy code

../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/base_task.py:312: in __call__
    return flyte_entity_call_handler(self, *args, **kwargs)  # type: ignore
../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/promise.py:1023: in flyte_entity_call_handler
    return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/base_task.py:276: in local_execute
    outputs_literal_map = self.sandbox_execute(ctx, input_literal_map)
../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/base_task.py:363: in sandbox_execute
    return self.dispatch_execute(ctx, input_literal_map)
../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/base_task.py:613: in dispatch_execute
    raise e
../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/base_task.py:610: in dispatch_execute
    native_outputs = self.execute(**native_inputs)
../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/python_function_task.py:180: in execute
    return self.dynamic_execute(self._task_function, **kwargs)
../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/python_function_task.py:279: in dynamic_execute
    self._create_and_cache_dynamic_workflow()
../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/python_function_task.py:188: in _create_and_cache_dynamic_workflow
    self._wf = PythonFunctionWorkflow(self._task_function, metadata=workflow_meta, default_metadata=defaults)
../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/tracker.py:78: in __call__
    mod_name, mod_file = InstanceTrackingMeta._find_instance_module()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

    @staticmethod
    def _find_instance_module():
        frame = inspect.currentframe()
        while frame:
            if frame.f_code.co_name == "<module>" and "__name__" in frame.f_globals:
                if frame.f_globals["__name__"] != "__main__":
>                   return frame.f_globals["__name__"], frame.f_globals["__file__"]
E                   KeyError: '__file__'

../../../.virtualenvs/myenv/lib/python3.10/site-packages/flytekit/core/tracker.py:66: KeyError

During handling of the above exception, another exception occurred:

Jay Ganbat

10/19/2023, 5:42 PM

ok i think i got the issue, when i use

pytest -n

it spawns multiple workers and somehow this workers do not have

__file__

key in them. i changed key retrieval with

get

as a quick hack and that seem to be working 😆 but dont think this is a correct approach @Niels Bantilan do yo have a quick suggestion on this

Niels Bantilan

10/19/2023, 6:17 PM

hi @Jay Ganbat we’re working on a fix for this, in the mean time can you use

1.9.0

Jay Ganbat

10/19/2023, 6:22 PM

nice, yeah i will try that one

Open in Slack

Previous Next