https://flyte.org
Join Slack
Hi, tyring out `flytekit.extras` types, but it doesn’t seem to exist in `flytekit` (v1.2.3) - do I n...
# flyte-support
s
Hi, tyring out 
flytekit.extras
types, but it doesn’t seem to exist in 
flytekit
(v1.2.3) - do I need to install a separate module for this?
g
What kind of type you want to use in extras
s
flytekit.extras.pytorch.PyTorchCheckpoint
g
b
do you have 
torch
installed? The 
flytekit.extras.pytorch
module transformers are only registered if the 
torch
package is installed: https://github.com/flyteorg/flytekit/blob/dd3fcaea3604f606c98b05ce05e3a8fc7d520906/flytekit/extras/pytorch/__init__.py#L25-L31
s
Yes I do have torch installed and imported
Still getting this error:
Copy code
AttributeError: module 'flytekit.extras' has no attribute 'pytorch'
t
Copy code
from flytekit.extras.pytorch import PyTorchCheckpoint
This import should work. What’s your flytekit version?
s
1.2.3
t
I have the same version and it works for me.
Can you re-install flytekit and verify?
s
I can’t try it now but I will in the next couple of days and report back
Thanks for looking into it
t
Sure, no problem.
s
@tall-lock-23197 this is just out of curiosity since I’m still trying to get my head around Python, but looking at the docs,
Copy code
flytekit.types.file.PythonPickledFile = <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>
So shouldn’t specifying 
flytekit.types.file.PythonPickledFile
as the return type hint be the same as 
flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
?
But the latter errors out during 
pylint
and also in the editor with parsing error
Couldn’t find a lot of info on 
__class_getitem__.<locals>._SpecificFormatClass
- would you mind giving me a bit of context as to what this is?
I tried creating a function with return type hint of 
flytekit.types.file.PythonPickledFile
and then running 
typing.get_type_hints
returns 
flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass
t
Here’s the relevant code to understand that: https://github.com/flyteorg/flytekit/blob/f2dd4aa146fa2886b469c31a6eda0c392567fb56/flytekit/types/file/file.py#L151. So 
PythonPickledFile
can be thought of as 
FlyteFile[Annotated[str, "pickle"]]
and hence you get that type. 
__class_getitem__
is to capture the item in 
[…]
in FlyteFile. 
PythonPickledFile
shouldn’t be used externally; it’s automatically used by Flytekit when the type isn’t recognizable. So you needn’t worry about it.
s
What is 
<locals>
here?
Is that a valid python syntax?
h
@sticky-angle-28419,this is just to indicate that we're returning an object of a class defined inside 
_class_getitem__
( https://github.com/flyteorg/flytekit/blob/f2dd4aa146fa2886b469c31a6eda0c392567fb56/flytekit/types/file/file.py#L163-L169 )
can you share the code you're seeing this? How are you running it?
s
Yeah so I’m just trying to understand this better: I’m running:
Copy code
def test_fn() -> flytekit.types.file.FlyteFile["csv"]:
    pass
Copy code
typing.get_type_hints(test_fn)
The result is
Copy code
{'return': <class 'flytekit.types.file.file.FlyteFile.__class_getitem__.<locals>._SpecificFormatClass'>}
And regardless of what string value I put into the FlyteFile type, I get the same return type hint
Wondering why that is
t
If you put 
FlyteFile
as the return type, then you wouldn’t be getting that.
s
Right 
FlyteFile
works fine - 
FlyteFile["csv"]
does not
t
No, it does work fine. It’s that it calls that inner class and returns that as a type. Are you facing any issue with that?
s
Oh yeah I didn’t mean it doesn’t work - it’s just strange what 
typing.get_type_hint
returns on a function that uses 
FlyteFile["csv"]
Maybe 
typing.get_type_hint
doesn’t know how to deal with 
_class_getitem__
t
I don’t think so. 
_SpecificFormatClass
subclasses 
FlyteFile
, so it’s returning the right type.
s
OK I guess I’m not understanding this correctly - so 
FlyteFile["csv"]
and 
FlyteFile["png"]
are the same type but with different metadata using 
typing_extensions.Annotated
? So there’s no way to differentiate between the two if I were to use 
typing.get_type_hint
to get types? For example, if I were to run 
typing.get_type_hint
for 
FlyteFile["csv"]
and 
FlyteFile["png"]
, both would return the same 
_SpecificFormatClass
- any way to find out which is which?
h
@sticky-angle-28419, under the covers (i.e. when we convert the types to the Flyte type system) 
FlyteFile["csv"]
does not match 
FlyteFile["png"]
(similarly 
FlyteFile[Annotated[str, FileExt("abc")]]
is not going to match 
FlyteFile[Annotated[str, FileExt("def")]]
). If I understand your concern correctly, you want better typehints, in that case I agree, flytekit could use some work to provide better type hints to static type checkers like mypy or pyright. Can you open a github issue to track this? The support for types is way more mature now than when 
FlyteFile
was initially written, so possibly we'll be able to improve the ux there.
s
Oh I see - got it. Will open an issue in github
Thanks!!
h
thank you!
👍 1
151 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by