Hello there! 👋 Hope this is the right place to ask
I just started testing out Flyte. Following this tutorial, I keep getting some FileNotFound errors when I execute the workflow remotely on the sandbox cluster (
pyflyte run --remote filebased_wf.py wf --indirpath input-data
), despite creating the files in minio from browser. On the contrary, I can execute workflows which write files (from tmp). Is there any restriction on reading from minio in sandbox? How should I do it?
brash-autumn-99506
05/27/2024, 4:10 PM
For reference, the updated tutorials on FlyteFiles I found in the doc don't read files from s3, rather download them from url - that's also where the question stems from
f
freezing-airport-6809
05/27/2024, 4:12 PM
Pyflyte should automatically upload files if you point to local files
freezing-airport-6809
05/27/2024, 4:12 PM
Don’t need to upload manually to minio
b
brash-autumn-99506
05/27/2024, 4:16 PM
Thanks Ketan! My point was to try out a flow in which my workflow is reading data from S3 as later I'll need to read from GCS. I see that's not needed if I have local files, but what if I don't? Is there anything particular I should do to read from storage?
t
tall-lock-23197
05/27/2024, 4:57 PM
as long as flyte has access to the blob storage, it should work. did you provide s3 uri as input to the workflow?
b
brash-autumn-99506
05/28/2024, 6:40 AM
Thanks Samhita! I was providing with the bucket folder but not the URI, that solved accessing the files! Thank you