mammoth-mouse-1111
08/18/2025, 5:15 PMContainerTask
to work on my flyte-core
deployment. Specifically, the init-container flyte-copilot-downloader
is having a lot of trouble connecting to my on-prem s3 storage.
Inspecting the pod for the tutorial task I'm running:
• the image is <http://cr.flyte.org/flyteorg/flytecopilot-release:v1.16.0-b2|cr.flyte.org/flyteorg/flytecopilot-release:v1.16.0-b2>
• The command seems to be passing in the secret key, access key, and custom endpoint correctly.
command:
- /bin/flyte-copilot
- --storage.limits.maxDownloadMBs=0
- --storage.container=BUCKET
- --storage.type=s3
- --storage.connection.secret-key=SECRET
- --storage.connection.access-key=SECRET
- --storage.connection.auth-type=accesskey
- --storage.connection.region=us-east-1
- --storage.connection.endpoint=ENDPOINT
Despite this, I get the following error:
{"json":{},"level":"error","msg":"failed to get AWS credentials: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","ts":"2025-08-18T17:28:56Z"} │
• I've tried to supply environment variables via the configmap as suggested in previous threads but this did not propagate them to the download init container.
• I tried injecting the environment variables into the init container using a podtemplate. This did result in the right envvars on the running container, but a different error message presented itself:
failed to get AWS credentials: invalid data in credential fetch
If anyone here has faced a similar issue, I would really appreciate the help. Thanks 🙏mammoth-mouse-1111
08/18/2025, 8:19 PMflyte-copilot-downloader
init container and the flyte-copilot-uploader
sidecar with a podtemplate.
In our infra setup, we're using a custom on-prem s3 endpoint (ceph) and don't need AWS_ACCESS_TOKEN
to authenticate. However, the flyte source code explicitly checks for a token, which we were not providing. causing the "invalid data" error message. We worked around this by supplying a dummy value for this environment variable.