Ed Fincham
02/15/2023, 3:42 PMpyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'
I get a 403 error. There's a signed url, but this is rejected by the metadata bucket. The cluster itself has a flyte service account with read/write access to the bucket, but the above is all happening locally. Any ideas how I can debug this as I'm currently a bit stumped!
Thanks a lot 🙂Niels Bantilan
02/15/2023, 4:18 PMjeev
Ed Fincham
02/15/2023, 6:25 PMpyflyte run -p testflyte --remote example.py training_workflow --hyperparameters '{"C": 0.1}'
The traceback:
Traceback (most recent call last):
File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 472, in put_data
DataPersistencePlugins.find_plugin(remote_path)(data_config=self.data_config).put(
File "/home/ed/venv/lib/python3.10/site-packages/flytekit/extras/persistence/http.py", line 72, in put
raise user.FlyteValueException(
flytekit.exceptions.user.FlyteValueException: Value error! Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL>
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/ed/venv/bin/pyflyte", line 8, in <module>
sys.exit(main())
File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1130, in __call__
return self.main(*args, **kwargs)
File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1055, in main
rv = self.invoke(ctx)
File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1657, in invoke
return _process_result(sub_ctx.command.invoke(sub_ctx))
File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 1404, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/home/ed/venv/lib/python3.10/site-packages/click/core.py", line 760, in invoke
return __callback(*args, **kwargs)
File "/home/ed/venv/lib/python3.10/site-packages/flytekit/clis/sdk_in_container/run.py", line 552, in _run
remote_entity = remote.register_script(
File "/home/ed/venv/lib/python3.10/site-packages/flytekit/remote/remote.py", line 800, in register_script
upload_location, md5_bytes = fast_register_single_script(
File "/home/ed/venv/lib/python3.10/site-packages/flytekit/tools/script_mode.py", line 112, in fast_register_single_script
flyte_ctx.file_access.put_data(archive_fname, upload_location.signed_url)
File "/home/ed/venv/lib/python3.10/site-packages/flytekit/core/data_persistence.py", line 476, in put_data
raise FlyteAssertion(
flytekit.exceptions.user.FlyteAssertion: Failed to put data from /tmp/tmppay0olxb/script_mode.tar.gz to <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> (recursive=False)
Original exception: Value error! Received: 403. Request to send data <https://s3.eu-north-1.amazonaws.com/><BUCKET>/testflyte/development/OOXUOJ6GHCUQB2IUIQUAJ3257M%3D%3D%3D%3D%3D%3D/scriptmode.tar.gz?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=<CREDENTIAL> failed.
Thanks for getting back to me 🙂Niels Bantilan
02/15/2023, 8:07 PMjeev
Ed Fincham
02/15/2023, 8:31 PMjeev
Ed Fincham
02/15/2023, 8:33 PMjeev
Ed Fincham
02/15/2023, 8:35 PMserviceAccount.create
to truejeev
Ed Fincham
02/15/2023, 8:38 PM