This is a potential follow-up to my earlier questi...
# ask-the-community
a
This is a potential follow-up to my earlier question: https://flyte-org.slack.com/archives/CP2HDHKE1/p1678183814829309 Are annotated `FlyteFile`s supposed to be used as input parameters of a task? I have attached an example workflow in the thread, where I'm defining a type alias for an annotated FlyteFile with a
HashMethod
, which for consistency's sake I'd like to also use as the input parameter for a second task. However doing so results in an error message that looks quite similar to the behavior I had described in my original post above (and probably has the same cause - if that's the case, I'm happy to provide a PR). I can see why allowing annotations on both an task output and a task input may lead to weird semantics questions (what function would the
HashMethod
on the input argument serve?), but I like the consistent typing between outputs and inputs and could see the data scientists in our org stumble across a similar problem in the future. Thanks!
Example workflow (note that
HashedFile
is used as the type annotation for the argument to the
do_work
task - replacing it with
FlyteFile
solves the error):
Copy code
from pathlib import Path
from typing import Annotated

from flytekit import HashMethod, current_context, task, workflow
from flytekit.types.file import FlyteFile


def hash_fn(ff: FlyteFile) -> str:
    return ff.path


HashedFile = Annotated[FlyteFile, HashMethod(hash_fn)]


@task
def create_file() -> HashedFile:
    f = Path(current_context().working_directory) / "data.txt"
    f.write_text("Hello World")
    return HashedFile(str(f))


@task
def do_work(ff: HashedFile) -> None:
    print(ff)


@workflow
def wf() -> None:
    data_file = create_file()
    do_work(ff=data_file)


if __name__ == "__main__":
    wf()
Output:
Copy code
Traceback (most recent call last):
  File "/work/mlops/splat-ml/.sandbox/flyte_annotated_task_input.py", line 34, in <module>
    wf()
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/workflow.py", line 263, in __call__
    return flyte_entity_call_handler(self, *args, **input_kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/promise.py", line 1093, in flyte_entity_call_handler
    result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/workflow.py", line 282, in local_execute
    function_outputs = self.execute(**kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/workflow.py", line 722, in execute
    return exception_scopes.user_entry_point(self._workflow_function)(**kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 198, in user_entry_point
    return wrapped(*args, **kwargs)
  File "/work/mlops/splat-ml/.sandbox/flyte_annotated_task_input.py", line 30, in wf
    do_work(ff=data_file)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 299, in __call__
    return flyte_entity_call_handler(self, *args, **kwargs)  # type: ignore
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/promise.py", line 1085, in flyte_entity_call_handler
    return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 280, in local_execute
    outputs_literal_map = self.sandbox_execute(ctx, input_literal_map)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 346, in sandbox_execute
    return self.dispatch_execute(ctx, input_literal_map)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 518, in dispatch_execute
    native_inputs = TypeEngine.literal_map_to_kwargs(exec_ctx, input_literal_map, self.python_interface.inputs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 867, in literal_map_to_kwargs
    return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 867, in <dictcomp>
    return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 831, in to_python_value
    return transformer.to_python_value(ctx, lv, expected_python_type)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/types/file/file.py", line 417, in to_python_value
    if not issubclass(expected_python_type, FlyteFile):  # type: ignore
  File "/home/adriano/.pyenv/versions/3.9.16/lib/python3.9/abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class
The offending line in `file.py`:
Copy code
if not issubclass(expected_python_type, FlyteFile):  # type: ignore
    raise TypeError(f"Neither os.PathLike nor FlyteFile specified {expected_python_type}")
s
You may need to add the annotated check to the
to_python_value
method for this to work. cc @Eduardo Apolinario (eapolinario)
e
@Adrian Rumpold, this seems like a reasonable extension. Feel free to open a PR (should be as simple as adding this check to
to_python_value
). You can link to the same gh issue (and this time we can close it for good 🙂 )
a
Sounds good, I'll prepare another MR! Does this warrant any documentation notes or extra test cases regarding the semantics (as mentioned in the OP)?
121 Views