https://flyte.org logo
Title
s

seunggs

11/19/2022, 3:39 AM
Hi, tyring out
flytekit.extras
types, but it doesn’t seem to exist in
flytekit
(v1.2.3) - do I need to install a separate module for this?
k

Kevin Su

11/19/2022, 4:01 AM
What kind of type you want to use in extras
s

seunggs

11/19/2022, 4:01 AM
flytekit.extras.pytorch.PyTorchCheckpoint
n

Niels Bantilan

11/19/2022, 2:06 PM
do you have
torch
installed? The
flytekit.extras.pytorch
module transformers are only registered if the
torch
package is installed: https://github.com/flyteorg/flytekit/blob/dd3fcaea3604f606c98b05ce05e3a8fc7d520906/flytekit/extras/pytorch/__init__.py#L25-L31
s

seunggs

11/19/2022, 7:21 PM
Yes I do have torch installed and imported
Still getting this error:
AttributeError: module 'flytekit.extras' has no attribute 'pytorch'
s

Samhita Alla

11/21/2022, 4:30 AM
from flytekit.extras.pytorch import PyTorchCheckpoint
This import should work. What’s your flytekit version?
s

seunggs

11/21/2022, 4:31 AM
1.2.3
s

Samhita Alla

11/21/2022, 4:32 AM
I have the same version and it works for me.
Can you re-install flytekit and verify?
s

seunggs

11/21/2022, 4:33 AM
I can’t try it now but I will in the next couple of days and report back
Thanks for looking into it
s

Samhita Alla

11/21/2022, 4:33 AM
Sure, no problem.
s

seunggs

11/21/2022, 5:30 PM
@Samhita Alla this is just out of curiosity since I’m still trying to get my head around Python, but looking at the docs,
flytekit.types.file.PythonPickledFile = <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>
So shouldn’t specifying
flytekit.types.file.PythonPickledFile
as the return type hint be the same as
flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
?
But the latter errors out during
pylint
and also in the editor with parsing error
Couldn’t find a lot of info on
__class_getitem__.<locals>._SpecificFormatClass
- would you mind giving me a bit of context as to what this is?
I tried creating a function with return type hint of
flytekit.types.file.PythonPickledFile
and then running
typing.get_type_hints
returns
flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
s

Samhita Alla

11/21/2022, 5:36 PM
Here’s the relevant code to understand that: https://github.com/flyteorg/flytekit/blob/f2dd4aa146fa2886b469c31a6eda0c392567fb56/flytekit/types/file/file.py#L151. So
PythonPickledFile
can be thought of as
FlyteFile[Annotated[str, "pickle"]]
and hence you get that type.
__class_getitem__
is to capture the item in
[…]
in FlyteFile.
PythonPickledFile
shouldn’t be used externally; it’s automatically used by Flytekit when the type isn’t recognizable. So you needn’t worry about it.
s

seunggs

11/21/2022, 5:39 PM
What is
<locals>
here?
Is that a valid python syntax?
e

Eduardo Apolinario (eapolinario)

11/21/2022, 9:38 PM
@seunggs,this is just to indicate that we're returning an object of a class defined inside
_class_getitem__
( https://github.com/flyteorg/flytekit/blob/f2dd4aa146fa2886b469c31a6eda0c392567fb56/flytekit/types/file/file.py#L163-L169 )
can you share the code you're seeing this? How are you running it?
s

seunggs

11/22/2022, 12:01 AM
Yeah so I’m just trying to understand this better: I’m running:
def test_fn() -> flytekit.types.file.FlyteFile["csv"]:
    pass
typing.get_type_hints(test_fn)
The result is
{'return': <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>}
And regardless of what string value I put into the FlyteFile type, I get the same return type hint
Wondering why that is
s

Samhita Alla

11/22/2022, 4:10 AM
If you put
FlyteFile
as the return type, then you wouldn’t be getting that.
s

seunggs

11/22/2022, 4:58 AM
Right
FlyteFile
works fine -
FlyteFile["csv"]
does not
s

Samhita Alla

11/22/2022, 4:59 AM
No, it does work fine. It’s that it calls that inner class and returns that as a type. Are you facing any issue with that?
s

seunggs

11/22/2022, 4:59 AM
Oh yeah I didn’t mean it doesn’t work - it’s just strange what
typing.get_type_hint
returns on a function that uses
FlyteFile["csv"]
Maybe
typing.get_type_hint
doesn’t know how to deal with
_class_getitem__
s

Samhita Alla

11/22/2022, 5:57 AM
I don’t think so.
_SpecificFormatClass
subclasses
FlyteFile
, so it’s returning the right type.
s

seunggs

11/22/2022, 2:15 PM
OK I guess I’m not understanding this correctly - so
FlyteFile["csv"]
and
FlyteFile["png"]
are the same type but with different metadata using
typing_extensions.Annotated
? So there’s no way to differentiate between the two if I were to use
typing.get_type_hint
to get types? For example, if I were to run
typing.get_type_hint
for
FlyteFile["csv"]
and
FlyteFile["png"]
, both would return the same
_SpecificFormatClass
- any way to find out which is which?
e

Eduardo Apolinario (eapolinario)

11/22/2022, 6:46 PM
@seunggs, under the covers (i.e. when we convert the types to the Flyte type system)
FlyteFile["csv"]
does not match
FlyteFile["png"]
(similarly
FlyteFile[Annotated[str, FileExt("abc")]]
is not going to match
FlyteFile[Annotated[str, FileExt("def")]]
). If I understand your concern correctly, you want better typehints, in that case I agree, flytekit could use some work to provide better type hints to static type checkers like mypy or pyright. Can you open a github issue to track this? The support for types is way more mature now than when
FlyteFile
was initially written, so possibly we'll be able to improve the ux there.
s

seunggs

11/22/2022, 6:47 PM
Oh I see - got it. Will open an issue in github
Thanks!!
e

Eduardo Apolinario (eapolinario)

11/22/2022, 6:47 PM
thank you!