seunggs
11/19/2022, 3:39 AMflytekit.extras
types, but it doesn’t seem to exist in flytekit
(v1.2.3) - do I need to install a separate module for this?Kevin Su
11/19/2022, 4:01 AMseunggs
11/19/2022, 4:01 AMflytekit.extras.pytorch.PyTorchCheckpoint
Kevin Su
11/19/2022, 5:54 AMNiels Bantilan
11/19/2022, 2:06 PMtorch
installed? The flytekit.extras.pytorch
module transformers are only registered if the torch
package is installed: https://github.com/flyteorg/flytekit/blob/dd3fcaea3604f606c98b05ce05e3a8fc7d520906/flytekit/extras/pytorch/__init__.py#L25-L31seunggs
11/19/2022, 7:21 PMAttributeError: module 'flytekit.extras' has no attribute 'pytorch'
Samhita Alla
11/21/2022, 4:30 AMfrom flytekit.extras.pytorch import PyTorchCheckpoint
This import should work. What’s your flytekit version?seunggs
11/21/2022, 4:31 AM1.2.3
Samhita Alla
11/21/2022, 4:32 AMseunggs
11/21/2022, 4:33 AMSamhita Alla
11/21/2022, 4:33 AMseunggs
11/21/2022, 5:30 PMflytekit.types.file.PythonPickledFile = <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>
flytekit.types.file.PythonPickledFile
as the return type hint be the same as flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
?pylint
and also in the editor with parsing error__class_getitem__.<locals>._SpecificFormatClass
- would you mind giving me a bit of context as to what this is?flytekit.types.file.PythonPickledFile
and then running typing.get_type_hints
returns flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
Samhita Alla
11/21/2022, 5:36 PMPythonPickledFile
can be thought of as FlyteFile[Annotated[str, "pickle"]]
and hence you get that type. __class_getitem__
is to capture the item in […]
in FlyteFile. PythonPickledFile
shouldn’t be used externally; it’s automatically used by Flytekit when the type isn’t recognizable. So you needn’t worry about it.seunggs
11/21/2022, 5:39 PM<locals>
here?Eduardo Apolinario (eapolinario)
11/21/2022, 9:38 PM_class_getitem__
( https://github.com/flyteorg/flytekit/blob/f2dd4aa146fa2886b469c31a6eda0c392567fb56/flytekit/types/file/file.py#L163-L169 )seunggs
11/22/2022, 12:01 AMdef test_fn() -> flytekit.types.file.FlyteFile["csv"]:
pass
typing.get_type_hints(test_fn)
{'return': <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>}
Samhita Alla
11/22/2022, 4:10 AMFlyteFile
as the return type, then you wouldn’t be getting that.seunggs
11/22/2022, 4:58 AMFlyteFile
works fine - FlyteFile["csv"]
does notSamhita Alla
11/22/2022, 4:59 AMseunggs
11/22/2022, 4:59 AMtyping.get_type_hint
returns on a function that uses FlyteFile["csv"]
typing.get_type_hint
doesn’t know how to deal with _class_getitem__
Samhita Alla
11/22/2022, 5:57 AM_SpecificFormatClass
subclasses FlyteFile
, so it’s returning the right type.seunggs
11/22/2022, 2:15 PMFlyteFile["csv"]
and FlyteFile["png"]
are the same type but with different metadata using typing_extensions.Annotated
? So there’s no way to differentiate between the two if I were to use typing.get_type_hint
to get types? For example, if I were to run typing.get_type_hint
for FlyteFile["csv"]
and FlyteFile["png"]
, both would return the same _SpecificFormatClass
- any way to find out which is which?Eduardo Apolinario (eapolinario)
11/22/2022, 6:46 PMFlyteFile["csv"]
does not match FlyteFile["png"]
(similarly FlyteFile[Annotated[str, FileExt("abc")]]
is not going to match FlyteFile[Annotated[str, FileExt("def")]]
).
If I understand your concern correctly, you want better typehints, in that case I agree, flytekit could use some work to provide better type hints to static type checkers like mypy or pyright. Can you open a github issue to track this? The support for types is way more mature now than when FlyteFile
was initially written, so possibly we'll be able to improve the ux there.seunggs
11/22/2022, 6:47 PMEduardo Apolinario (eapolinario)
11/22/2022, 6:47 PM