acoustic-carpenter-78188
09/13/2023, 12:37 AMpyflyte-fast-execute in that case, the problem is that by default we add the argument --no-sign-request when using AWS CLI for S3 and that makes it impossible to download the content from private S3 buckets.
Expected behavior
Not being anonymous by default, we shouldn't add the argument --no-sign-request by default
Additional context to reproduce
1. Run `pyflyte-fast-execute`with the needed arguments
❯ pyflyte-fast-execute --additional-distribution s3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz --dest-dir /src
{"asctime": "2022-08-16 14:21:55,941", "name": "flytekit", "levelname": "ERROR", "message": "Error from command '['aws', '--no-sign-request', 's3', 'cp', 's3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz', '/src']':\nb'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\\n'\n"}
{"asctime": "2022-08-16 14:21:55,941", "name": "flytekit", "levelname": "ERROR", "message": "Exception when trying to execute ['aws', 's3', 'cp', '3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz', '/src'], reason: Called process exited with error code: 1. Stderr dump:\n\nb'fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden\\n'"}
2. It can also be reproduced by using the AWS CLI commands directly
When using --no-sign-request
❯ aws --no-sign-request s3 cp s3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz .
fatal error: An error occurred (403) when calling the HeadObject operation: Forbidden
When not using --no-sign-request
❯ aws s3 cp s3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz .
download: s3://<private-bucket-name>/ea/<project-name>/<domain>/path_to.tar.gz to ./fast14de81dc9a6b0f459a1ab49e1d871e01.tar.gz
Screenshots
No response
Are you sure this issue hasn't been raised already?
☑︎ Yes
Have you read the Code of Conduct?
☑︎ Yes
flyteorg/flyteacoustic-carpenter-78188
09/13/2023, 12:37 AM