Hi all, hope you are doing well! I'm having some ...
# flyte-support
m
Hi all, hope you are doing well! I'm having some trouble getting
ContainerTask
to work on my
flyte-core
deployment. Specifically, the init-container
flyte-copilot-downloader
is having a lot of trouble connecting to my on-prem s3 storage. Inspecting the pod for the tutorial task I'm running: • the image is
<http://cr.flyte.org/flyteorg/flytecopilot-release:v1.16.0-b2|cr.flyte.org/flyteorg/flytecopilot-release:v1.16.0-b2>
• The command seems to be passing in the secret key, access key, and custom endpoint correctly.
Copy code
command:
    - /bin/flyte-copilot
    - --storage.limits.maxDownloadMBs=0
    - --storage.container=BUCKET
    - --storage.type=s3
    - --storage.connection.secret-key=SECRET
    - --storage.connection.access-key=SECRET
    - --storage.connection.auth-type=accesskey
    - --storage.connection.region=us-east-1
    - --storage.connection.endpoint=ENDPOINT
Despite this, I get the following error:
Copy code
{"json":{},"level":"error","msg":"failed to get AWS credentials: NoCredentialProviders: no valid providers in chain. Deprecated.\n\tFor verbose messaging see aws.Config.CredentialsChainVerboseErrors","ts":"2025-08-18T17:28:56Z"}                                                                                       │
• I've tried to supply environment variables via the configmap as suggested in previous threads but this did not propagate them to the download init container. • I tried injecting the environment variables into the init container using a podtemplate. This did result in the right envvars on the running container, but a different error message presented itself:
Copy code
failed to get AWS credentials: invalid data in credential fetch
If anyone here has faced a similar issue, I would really appreciate the help. Thanks 🙏
I've found a workaround, though its rather messy. I've injected the AWS environment variables to the
flyte-copilot-downloader
init container and the
flyte-copilot-uploader
sidecar with a podtemplate. In our infra setup, we're using a custom on-prem s3 endpoint (ceph) and don't need
AWS_ACCESS_TOKEN
to authenticate. However, the flyte source code explicitly checks for a token, which we were not providing. causing the "invalid data" error message. We worked around this by supplying a dummy value for this environment variable.