Hi All. I'm new to Flyte and I'm trying to run a s...
# ask-the-communityg
Grigoriy Sterin
11/15/2023, 4:29 AMHi All. I'm new to Flyte and I'm trying to run a simple workflow on a local demo cluster which would download files from S3 bucket in my AWS account. But tasks fail on attempt to get files from S3. What is the proper way to provide S3 credentials to Flyte?
y
Yi Chiu
11/15/2023, 4:32 AMyou can use environment variables, for example
Copy code
export AWS_ACCESS_KEY_ID=XXX
export AWS_SECRET_ACCESS_KEY=XXXy
Yun Sun
11/15/2023, 4:56 AMyou can use a profile to store the AWS credentials.
r
Ryuu
11/15/2023, 9:08 AM Copy code
you can fetch creds to 1 dict variable to function or use
@Task(
environment={
}
},
secret_requests=[]
)g
Grigoriy Sterin
11/16/2023, 5:50 AMThanks you all. Unfortunately that didn't seem to help. Here's the code of the Flyte task that fails:
Copy code
@task(
container_image="ghcr.io/flyteorg/flytekit:py3.10-1.10.1b0",
environment={
"AWS_ACCESS_KEY_ID": "some-value"
"AWS_SECRET_ACCESS_KEY": "some-value",
"AWS_DEFAULT_REGION": "some-value"
}
)
def consume() -> None:
f = FlyteFile("<s3://some-s3/file>")
f.download() # fails here
# in the `user_script.sh` I want to do some manipulations with
# the file downloaded to the local filesystem
subprocess.run(["./user_script.sh"]) Copy code
packages/flytekit/types/file/file.py\", line 209, in __fspath__\n self._downloader()\n File \"/usr/local/lib/python3.10/site-packages/flytekit/types/file/file.py\", line 434, in _downloader\n return ctx.file_access.get_data(uri, local_path, is_multipart=False)\n File \"/usr/local/lib/python3.10/site-packages/flytekit/core/data_persistence.py\", line 467, in get_data\n raise FlyteAssertion(\n\nMessage:\n\n Failed to get data from <s3://some-s3/file> to /tmp/flyteslc48j5z/local_flytekit/0d47388e2e9630d969bb337a5dfb83ac/file (recursive=False).\n\nOriginal exception: Access Denied.\n\nUser error."}
{"asctime": "2023-11-16 05:33:31,327", "name": "flytekit.entrypoint", "levelname": "ERROR", "message": "!! End Error Captured by Flyte !!"}