most-insurance-86484
05/09/2023, 12:39 PMHashMethod
, which for consistency's sake I'd like to also use as the input parameter for a second task. However doing so results in an error message that looks quite similar to the behavior I had described in my original post above (and probably has the same cause - if that's the case, I'm happy to provide a PR).
I can see why allowing annotations on both an task output and a task input may lead to weird semantics questions (what function would the HashMethod
on the input argument serve?), but I like the consistent typing between outputs and inputs and could see the data scientists in our org stumble across a similar problem in the future.
Thanks!most-insurance-86484
05/09/2023, 12:41 PMHashedFile
is used as the type annotation for the argument to the do_work
task - replacing it with FlyteFile
solves the error):
from pathlib import Path
from typing import Annotated
from flytekit import HashMethod, current_context, task, workflow
from flytekit.types.file import FlyteFile
def hash_fn(ff: FlyteFile) -> str:
return ff.path
HashedFile = Annotated[FlyteFile, HashMethod(hash_fn)]
@task
def create_file() -> HashedFile:
f = Path(current_context().working_directory) / "data.txt"
f.write_text("Hello World")
return HashedFile(str(f))
@task
def do_work(ff: HashedFile) -> None:
print(ff)
@workflow
def wf() -> None:
data_file = create_file()
do_work(ff=data_file)
if __name__ == "__main__":
wf()
Output:
Traceback (most recent call last):
File "/work/mlops/splat-ml/.sandbox/flyte_annotated_task_input.py", line 34, in <module>
wf()
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/workflow.py", line 263, in __call__
return flyte_entity_call_handler(self, *args, **input_kwargs)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/promise.py", line 1093, in flyte_entity_call_handler
result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/workflow.py", line 282, in local_execute
function_outputs = self.execute(**kwargs)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/workflow.py", line 722, in execute
return exception_scopes.user_entry_point(self._workflow_function)(**kwargs)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 198, in user_entry_point
return wrapped(*args, **kwargs)
File "/work/mlops/splat-ml/.sandbox/flyte_annotated_task_input.py", line 30, in wf
do_work(ff=data_file)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 299, in __call__
return flyte_entity_call_handler(self, *args, **kwargs) # type: ignore
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/promise.py", line 1085, in flyte_entity_call_handler
return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 280, in local_execute
outputs_literal_map = self.sandbox_execute(ctx, input_literal_map)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 346, in sandbox_execute
return self.dispatch_execute(ctx, input_literal_map)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 518, in dispatch_execute
native_inputs = TypeEngine.literal_map_to_kwargs(exec_ctx, input_literal_map, self.python_interface.inputs)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 867, in literal_map_to_kwargs
return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 867, in <dictcomp>
return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 831, in to_python_value
return transformer.to_python_value(ctx, lv, expected_python_type)
File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/types/file/file.py", line 417, in to_python_value
if not issubclass(expected_python_type, FlyteFile): # type: ignore
File "/home/adriano/.pyenv/versions/3.9.16/lib/python3.9/abc.py", line 123, in __subclasscheck__
return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class
The offending line in `file.py`:
if not issubclass(expected_python_type, FlyteFile): # type: ignore
raise TypeError(f"Neither os.PathLike nor FlyteFile specified {expected_python_type}")
tall-lock-23197
to_python_value
method for this to work. cc @high-accountant-32689high-accountant-32689
05/10/2023, 5:39 PMto_python_value
). You can link to the same gh issue (and this time we can close it for good 🙂 )most-insurance-86484
05/11/2023, 6:04 AMmost-insurance-86484
05/11/2023, 8:15 AM