seunggs
11/19/2022, 3:39 AMflytekit.extras
types, but it doesn’t seem to exist in flytekit
(v1.2.3) - do I need to install a separate module for this?Kevin Su
11/19/2022, 4:01 AMseunggs
11/19/2022, 4:01 AMflytekit.extras.pytorch.PyTorchCheckpoint
Kevin Su
11/19/2022, 5:54 AMNiels Bantilan
11/19/2022, 2:06 PMtorch
installed? The flytekit.extras.pytorch
module transformers are only registered if the torch
package is installed: https://github.com/flyteorg/flytekit/blob/dd3fcaea3604f606c98b05ce05e3a8fc7d520906/flytekit/extras/pytorch/__init__.py#L25-L31seunggs
11/19/2022, 7:21 PMseunggs
11/19/2022, 7:21 PMAttributeError: module 'flytekit.extras' has no attribute 'pytorch'
Samhita Alla
from flytekit.extras.pytorch import PyTorchCheckpoint
This import should work. What’s your flytekit version?seunggs
11/21/2022, 4:31 AM1.2.3
Samhita Alla
Samhita Alla
seunggs
11/21/2022, 4:33 AMseunggs
11/21/2022, 4:33 AMSamhita Alla
seunggs
11/21/2022, 5:30 PMflytekit.types.file.PythonPickledFile = <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>
seunggs
11/21/2022, 5:30 PMflytekit.types.file.PythonPickledFile
as the return type hint be the same as flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
?seunggs
11/21/2022, 5:31 PMpylint
and also in the editor with parsing errorseunggs
11/21/2022, 5:31 PM__class_getitem__.<locals>._SpecificFormatClass
- would you mind giving me a bit of context as to what this is?seunggs
11/21/2022, 5:32 PMflytekit.types.file.PythonPickledFile
and then running typing.get_type_hints
returns flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
Samhita Alla
PythonPickledFile
can be thought of as FlyteFile[Annotated[str, "pickle"]]
and hence you get that type. __class_getitem__
is to capture the item in […]
in FlyteFile. PythonPickledFile
shouldn’t be used externally; it’s automatically used by Flytekit when the type isn’t recognizable. So you needn’t worry about it.seunggs
11/21/2022, 5:39 PM<locals>
here?seunggs
11/21/2022, 5:39 PMEduardo Apolinario (eapolinario)
11/21/2022, 9:38 PM_class_getitem__
( https://github.com/flyteorg/flytekit/blob/f2dd4aa146fa2886b469c31a6eda0c392567fb56/flytekit/types/file/file.py#L163-L169 )Eduardo Apolinario (eapolinario)
11/21/2022, 9:38 PMseunggs
11/22/2022, 12:01 AMdef test_fn() -> flytekit.types.file.FlyteFile["csv"]:
pass
seunggs
11/22/2022, 12:01 AMtyping.get_type_hints(test_fn)
seunggs
11/22/2022, 12:01 AMseunggs
11/22/2022, 12:01 AM{'return': <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>}
seunggs
11/22/2022, 12:03 AMseunggs
11/22/2022, 12:03 AMSamhita Alla
FlyteFile
as the return type, then you wouldn’t be getting that.seunggs
11/22/2022, 4:58 AMFlyteFile
works fine - FlyteFile["csv"]
does notSamhita Alla
seunggs
11/22/2022, 4:59 AMtyping.get_type_hint
returns on a function that uses FlyteFile["csv"]
seunggs
11/22/2022, 5:00 AMtyping.get_type_hint
doesn’t know how to deal with _class_getitem__
Samhita Alla
_SpecificFormatClass
subclasses FlyteFile
, so it’s returning the right type.seunggs
11/22/2022, 2:15 PMFlyteFile["csv"]
and FlyteFile["png"]
are the same type but with different metadata using typing_extensions.Annotated
? So there’s no way to differentiate between the two if I were to use typing.get_type_hint
to get types? For example, if I were to run typing.get_type_hint
for FlyteFile["csv"]
and FlyteFile["png"]
, both would return the same _SpecificFormatClass
- any way to find out which is which?Eduardo Apolinario (eapolinario)
11/22/2022, 6:46 PMFlyteFile["csv"]
does not match FlyteFile["png"]
(similarly FlyteFile[Annotated[str, FileExt("abc")]]
is not going to match FlyteFile[Annotated[str, FileExt("def")]]
).
If I understand your concern correctly, you want better typehints, in that case I agree, flytekit could use some work to provide better type hints to static type checkers like mypy or pyright. Can you open a github issue to track this? The support for types is way more mature now than when FlyteFile
was initially written, so possibly we'll be able to improve the ux there.seunggs
11/22/2022, 6:47 PMseunggs
11/22/2022, 6:47 PMEduardo Apolinario (eapolinario)
11/22/2022, 6:47 PM