Pryce
05/28/2023, 11:39 PMType of Generic List type is not supported, Transformer for type <class 'tuple'> is restricted currently
So now I'm back to using FlyteDirectory but it's.. awkward since there's no metadata about the files and they're given arbitrary names on the backend unless I explicitly name them somehow.
I'm wondering how folks might go about dealing with this? Do I handle all the metadata via the FlyteFile names? Pass some sort of metadata object along with every FlyteDirectory? Maybe there's a way around the custom dataclass limitation, as that would be the most elegant solution. Thanks for reading! I appreciate any insight.Ketan (kumare3)
Pryce
05/29/2023, 5:13 PM@task
def get_file_contents(infile: FlyteFile) -> str:
local = Path(infile.path)
print(local.exists())
content = ''
with open(infile, 'r') as in_:
content = in_.read()
return content
This task runs without issue in the current form. However, that print statement will say False
, and if I try to open local
it will fail with a File not found
error.open
is called. My issue was trying to access FlyteFile.path
directly and passing it to another function, which would fail saying the file wasn't there. I think I falsely assumed this had to do with my custom dataclass.path
attribute specifies (e.g. /tmp/flytetsfn_0lu/local_flytekit/899733d34a9e428d2773b2c5ffd2914d/hello.txt
) where a FlyteFile will be downloaded to if the download
method is called, not where it actually exists?Samhita Alla
path
is populated after the file is downloaded. Try setting local
to infile.download()
.Pryce
05/30/2023, 5:50 PM