sticky-angle-28419
11/19/2022, 3:39 AMflytekit.extras
types, but it doesn’t seem to exist in flytekit
(v1.2.3) - do I need to install a separate module for this?glamorous-carpet-83516
11/19/2022, 4:01 AMsticky-angle-28419
11/19/2022, 4:01 AMflytekit.extras.pytorch.PyTorchCheckpoint
glamorous-carpet-83516
11/19/2022, 5:54 AMbroad-monitor-993
11/19/2022, 2:06 PMtorch
installed? The flytekit.extras.pytorch
module transformers are only registered if the torch
package is installed: https://github.com/flyteorg/flytekit/blob/dd3fcaea3604f606c98b05ce05e3a8fc7d520906/flytekit/extras/pytorch/__init__.py#L25-L31sticky-angle-28419
11/19/2022, 7:21 PMsticky-angle-28419
11/19/2022, 7:21 PMAttributeError: module 'flytekit.extras' has no attribute 'pytorch'
tall-lock-23197
from flytekit.extras.pytorch import PyTorchCheckpoint
This import should work. What’s your flytekit version?sticky-angle-28419
11/21/2022, 4:31 AM1.2.3
tall-lock-23197
tall-lock-23197
sticky-angle-28419
11/21/2022, 4:33 AMsticky-angle-28419
11/21/2022, 4:33 AMtall-lock-23197
sticky-angle-28419
11/21/2022, 5:30 PMflytekit.types.file.PythonPickledFile = <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>
sticky-angle-28419
11/21/2022, 5:30 PMflytekit.types.file.PythonPickledFile
as the return type hint be the same as flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
?sticky-angle-28419
11/21/2022, 5:31 PMpylint
and also in the editor with parsing errorsticky-angle-28419
11/21/2022, 5:31 PM__class_getitem__.<locals>._SpecificFormatClass
- would you mind giving me a bit of context as to what this is?sticky-angle-28419
11/21/2022, 5:32 PMflytekit.types.file.PythonPickledFile
and then running typing.get_type_hints
returns flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
tall-lock-23197
PythonPickledFile
can be thought of as FlyteFile[Annotated[str, "pickle"]]
and hence you get that type. __class_getitem__
is to capture the item in […]
in FlyteFile. PythonPickledFile
shouldn’t be used externally; it’s automatically used by Flytekit when the type isn’t recognizable. So you needn’t worry about it.sticky-angle-28419
11/21/2022, 5:39 PM<locals>
here?sticky-angle-28419
11/21/2022, 5:39 PMhigh-accountant-32689
11/21/2022, 9:38 PM_class_getitem__
( https://github.com/flyteorg/flytekit/blob/f2dd4aa146fa2886b469c31a6eda0c392567fb56/flytekit/types/file/file.py#L163-L169 )high-accountant-32689
11/21/2022, 9:38 PMsticky-angle-28419
11/22/2022, 12:01 AMdef test_fn() -> flytekit.types.file.FlyteFile["csv"]:
pass
sticky-angle-28419
11/22/2022, 12:01 AMtyping.get_type_hints(test_fn)
sticky-angle-28419
11/22/2022, 12:01 AMsticky-angle-28419
11/22/2022, 12:01 AM{'return': <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>}
sticky-angle-28419
11/22/2022, 12:03 AMsticky-angle-28419
11/22/2022, 12:03 AMtall-lock-23197
FlyteFile
as the return type, then you wouldn’t be getting that.sticky-angle-28419
11/22/2022, 4:58 AMFlyteFile
works fine - FlyteFile["csv"]
does nottall-lock-23197
sticky-angle-28419
11/22/2022, 4:59 AMtyping.get_type_hint
returns on a function that uses FlyteFile["csv"]
sticky-angle-28419
11/22/2022, 5:00 AMtyping.get_type_hint
doesn’t know how to deal with _class_getitem__
tall-lock-23197
_SpecificFormatClass
subclasses FlyteFile
, so it’s returning the right type.sticky-angle-28419
11/22/2022, 2:15 PMFlyteFile["csv"]
and FlyteFile["png"]
are the same type but with different metadata using typing_extensions.Annotated
? So there’s no way to differentiate between the two if I were to use typing.get_type_hint
to get types? For example, if I were to run typing.get_type_hint
for FlyteFile["csv"]
and FlyteFile["png"]
, both would return the same _SpecificFormatClass
- any way to find out which is which?high-accountant-32689
11/22/2022, 6:46 PMFlyteFile["csv"]
does not match FlyteFile["png"]
(similarly FlyteFile[Annotated[str, FileExt("abc")]]
is not going to match FlyteFile[Annotated[str, FileExt("def")]]
).
If I understand your concern correctly, you want better typehints, in that case I agree, flytekit could use some work to provide better type hints to static type checkers like mypy or pyright. Can you open a github issue to track this? The support for types is way more mature now than when FlyteFile
was initially written, so possibly we'll be able to improve the ux there.sticky-angle-28419
11/22/2022, 6:47 PMsticky-angle-28419
11/22/2022, 6:47 PMhigh-accountant-32689
11/22/2022, 6:47 PM