https://flyte.org logo
Join Slack
Powered by
This is a potential follow-up to my earlier questi...
# ask-the-community
a
This is a potential follow-up to my earlier question: https://flyte-org.slack.com/archives/CP2HDHKE1/p1678183814829309 Are annotated `FlyteFile`s supposed to be used as input parameters of a task? I have attached an example workflow in the thread, where I'm defining a type alias for an annotated FlyteFile with a 
HashMethod
, which for consistency's sake I'd like to also use as the input parameter for a second task. However doing so results in an error message that looks quite similar to the behavior I had described in my original post above (and probably has the same cause - if that's the case, I'm happy to provide a PR). I can see why allowing annotations on both an task output and a task input may lead to weird semantics questions (what function would the 
HashMethod
on the input argument serve?), but I like the consistent typing between outputs and inputs and could see the data scientists in our org stumble across a similar problem in the future. Thanks!
Example workflow (note that 
HashedFile
is used as the type annotation for the argument to the 
do_work
task - replacing it with 
FlyteFile
solves the error):
Copy code
from pathlib import Path
from typing import Annotated

from flytekit import HashMethod, current_context, task, workflow
from flytekit.types.file import FlyteFile


def hash_fn(ff: FlyteFile) -> str:
    return ff.path


HashedFile = Annotated[FlyteFile, HashMethod(hash_fn)]


@task
def create_file() -> HashedFile:
    f = Path(current_context().working_directory) / "data.txt"
    f.write_text("Hello World")
    return HashedFile(str(f))


@task
def do_work(ff: HashedFile) -> None:
    print(ff)


@workflow
def wf() -> None:
    data_file = create_file()
    do_work(ff=data_file)


if __name__ == "__main__":
    wf()
Output:
Copy code
Traceback (most recent call last):
  File "/work/mlops/splat-ml/.sandbox/flyte_annotated_task_input.py", line 34, in <module>
    wf()
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/workflow.py", line 263, in __call__
    return flyte_entity_call_handler(self, *args, **input_kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/promise.py", line 1093, in flyte_entity_call_handler
    result = cast(LocallyExecutable, entity).local_execute(child_ctx, **kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/workflow.py", line 282, in local_execute
    function_outputs = self.execute(**kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/workflow.py", line 722, in execute
    return exception_scopes.user_entry_point(self._workflow_function)(**kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/exceptions/scopes.py", line 198, in user_entry_point
    return wrapped(*args, **kwargs)
  File "/work/mlops/splat-ml/.sandbox/flyte_annotated_task_input.py", line 30, in wf
    do_work(ff=data_file)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 299, in __call__
    return flyte_entity_call_handler(self, *args, **kwargs)  # type: ignore
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/promise.py", line 1085, in flyte_entity_call_handler
    return cast(LocallyExecutable, entity).local_execute(ctx, **kwargs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 280, in local_execute
    outputs_literal_map = self.sandbox_execute(ctx, input_literal_map)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 346, in sandbox_execute
    return self.dispatch_execute(ctx, input_literal_map)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/base_task.py", line 518, in dispatch_execute
    native_inputs = TypeEngine.literal_map_to_kwargs(exec_ctx, input_literal_map, self.python_interface.inputs)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 867, in literal_map_to_kwargs
    return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 867, in <dictcomp>
    return {k: TypeEngine.to_python_value(ctx, lm.literals[k], python_types[k]) for k, v in lm.literals.items()}
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/core/type_engine.py", line 831, in to_python_value
    return transformer.to_python_value(ctx, lv, expected_python_type)
  File "/work/mlops/splat-ml/.venv/lib/python3.9/site-packages/flytekit/types/file/file.py", line 417, in to_python_value
    if not issubclass(expected_python_type, FlyteFile):  # type: ignore
  File "/home/adriano/.pyenv/versions/3.9.16/lib/python3.9/abc.py", line 123, in __subclasscheck__
    return _abc_subclasscheck(cls, subclass)
TypeError: issubclass() arg 1 must be a class
The offending line in `file.py`:
Copy code
if not issubclass(expected_python_type, FlyteFile):  # type: ignore
    raise TypeError(f"Neither os.PathLike nor FlyteFile specified {expected_python_type}")
s
You may need to add the annotated check to the 
to_python_value
method for this to work. cc @Eduardo Apolinario (eapolinario)
e
@Adrian Rumpold, this seems like a reasonable extension. Feel free to open a PR (should be as simple as adding this check to 
to_python_value
). You can link to the same gh issue (and this time we can close it for good 🙂 )
a
Sounds good, I'll prepare another MR! Does this warrant any documentation notes or extra test cases regarding the semantics (as mentioned in the OP)?
154 Views
Open in Slack
PreviousNext
Powered by