acceptable-salesclerk-36214
11/15/2023, 4:29 AMflaky-daybreak-42119
11/15/2023, 4:32 AMexport AWS_ACCESS_KEY_ID=XXX
export AWS_SECRET_ACCESS_KEY=XXX
steep-summer-36262
11/15/2023, 4:56 AMfull-evening-87657
11/15/2023, 9:08 AMyou can fetch creds to 1 dict variable to function or use
@Task(
environment={
}
},
secret_requests=[]
)
acceptable-salesclerk-36214
11/16/2023, 5:50 AM@task(
container_image="ghcr.io/flyteorg/flytekit:py3.10-1.10.1b0",
environment={
"AWS_ACCESS_KEY_ID": "some-value"
"AWS_SECRET_ACCESS_KEY": "some-value",
"AWS_DEFAULT_REGION": "some-value"
}
)
def consume() -> None:
f = FlyteFile("<s3://some-s3/file>")
f.download() # fails here
# in the `user_script.sh` I want to do some manipulations with
# the file downloaded to the local filesystem
subprocess.run(["./user_script.sh"])
I'm getting the following error when calling `f.download()`:
packages/flytekit/types/file/file.py\", line 209, in __fspath__\n self._downloader()\n File \"/usr/local/lib/python3.10/site-packages/flytekit/types/file/file.py\", line 434, in _downloader\n return ctx.file_access.get_data(uri, local_path, is_multipart=False)\n File \"/usr/local/lib/python3.10/site-packages/flytekit/core/data_persistence.py\", line 467, in get_data\n raise FlyteAssertion(\n\nMessage:\n\n Failed to get data from <s3://some-s3/file> to /tmp/flyteslc48j5z/local_flytekit/0d47388e2e9630d969bb337a5dfb83ac/file (recursive=False).\n\nOriginal exception: Access Denied.\n\nUser error."}
{"asctime": "2023-11-16 05:33:31,327", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}
I'm able to download the same file using same credentials with aws cli