sparse-window-1536
03/31/2022, 10:40 PMFlyteFile
, the path is not really obvious/human-friendly.magnificent-teacher-86590
03/31/2022, 10:50 PMreturn FlyteFile(path=local_path, remote_path=somewhere_remote)
it should put your data there and you can configre paths and names to be human readablesparse-window-1536
03/31/2022, 11:07 PMreturn FlyteFile(path='test.pickle', remote_path='<s3://my-s3-bucket/flyte-example/say_hello.pickle>')
and then call another function to retrieve the file on the bucket?sparse-window-1536
03/31/2022, 11:08 PM@task
def retrieve_from_s3(uri: PythonPickledFile) -> str:
uri.download()
with open(uri.path, 'r') as handle:
x = pickle.load(handle)
return x
and it doesn't even run; in fact, I'm receiving a 500 error from the server.magnificent-teacher-86590
03/31/2022, 11:09 PMmagnificent-teacher-86590
03/31/2022, 11:09 PMwith open(uri, 'r') as handle:
FLyte should be able to automically download it for youmagnificent-teacher-86590
03/31/2022, 11:11 PM2022-03-10 08:25:02,782 [INFO] Entering timed context: Copying (<gs://remote_file> -> /tmp/flytequjfq95e/local_flytekit/local_file)
something like abovesparse-window-1536
03/31/2022, 11:13 PMMessage:
Failed to get data from <s3://my-s3-bucket/flyte-example/say_hello.pkl> to /tmp/flytek5pbstuv/local_flytekit/5186be74bee2a9b806fa2c15e358371b/say_hello.pkl (recursive=False).
Original exception: Called process exited with error code: 1. Stderr dump:
b'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\n'
User error.
sparse-window-1536
03/31/2022, 11:14 PMmagnificent-teacher-86590
03/31/2022, 11:14 PMmagnificent-teacher-86590
03/31/2022, 11:14 PMsparse-window-1536
03/31/2022, 11:17 PMsparse-window-1536
03/31/2022, 11:17 PMmagnificent-teacher-86590
03/31/2022, 11:28 PMsparse-window-1536
03/31/2022, 11:53 PMtall-lock-23197
sparse-window-1536
04/01/2022, 1:42 PMsparse-window-1536
04/01/2022, 1:42 PM