Rupsha Chaudhuri
03/25/2024, 4:27 PM[4/4] currentAttempt done. Last Error: USER::[1/1] currentAttempt done. Last Error: USER::[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[flyte-copilot-downloader] terminated with exit code (1). Reason [Error]. Message:
Type to use [iam, accesskey]. (default "iam")
--storage.connection.disable-ssl Disables SSL connection. Should only be used for development.
--storage.connection.endpoint string URL for storage client to connect to.
--storage.connection.region string Region to connect to. (default "us-east-1")
--storage.connection.secret-key string Secret to use when accesskey is set.
--storage.container string Initial container (in s3 a bucket) to create -if it doesn't exist-.'
--storage.defaultHttpClient.timeout string Sets time out on the http client. (default "0s")
--storage.enable-multicontainer If this is true, then the container argument is overlooked and redundant. This config will automatically open new connections to new containers/buckets as they are encountered
--storage.limits.maxDownloadMBs int Maximum allowed download size (in MBs) per call. (default 2)
--storage.stow.config stringToString Configuration for stow backend. Refer to github/graymeta/stow (default [])
--storage.stow.kind string Kind of Stow backend to use. Refer to github/graymeta/stow
--storage.type string Sets the type of storage to configure [s3/minio/local/mem/stow]. (default "s3")
--tls-server-name string If provided, this name will be used to validate server certificate. If this is not provided, hostname used to contact the server is used.
--token string Bearer token for authentication to the API server
--user string The name of the kubeconfig user to use
--username string Username for basic authentication to the API server
-v, --v Level number for the log level verbosity
--vmodule moduleSpec comma-separated list of pattern=N settings for file-filtered logging
Ketan (kumare3)
Ketan (kumare3)
Rupsha Chaudhuri
03/25/2024, 4:43 PMRupsha Chaudhuri
03/25/2024, 4:44 PMRupsha Chaudhuri
03/25/2024, 4:47 PMChris Grass
03/25/2024, 5:07 PMContainerTask
run, or only a seemingly-random subset of them?Rupsha Chaudhuri
03/25/2024, 5:08 PMRupsha Chaudhuri
03/25/2024, 5:11 PMChris Grass
03/25/2024, 5:11 PMflyte-copilot-downloader
container that was spawned from our ContainerTask
required credentials to the metadata bucket (to fetch an input FlyteFile). those credentials were not being propagated from the PodTemplate containerspec, so we had to manage them independently.Chris Grass
03/25/2024, 5:12 PMRupsha Chaudhuri
03/25/2024, 5:12 PMChris Grass
03/25/2024, 5:13 PMChris Grass
03/25/2024, 5:16 PMsecret_requests
because it added a prefix that was not configurable. we ended up having to manage the credentials outside of the ContainerTask
definitionKevin Su
03/25/2024, 5:28 PMKetan (kumare3)
L godlike
03/26/2024, 12:52 AMdid you see this error before?
Rupsha Chaudhuri
03/26/2024, 1:01 AML godlike
03/26/2024, 1:03 AMErich Shan
03/27/2024, 12:16 AM{"json":{},"level":"error","msg":"Failed to Get credentials.","ts":"2024-03-26T18:14:27Z"}
{"json":{},"level":"error","msg":"Failed to Get credentials.","ts":"2024-03-26T18:14:32Z"}
{"json":{},"level":"error","msg":"Failed to Get credentials.","ts":"2024-03-26T18:14:37Z"}
i guess what i don't understand so far is what is printing this error since we do have access to our s3 bucket from within this container (i reproduced this issue and can ls the contents of the relevant bucket)Erich Shan
03/27/2024, 12:17 AMErich Shan
03/27/2024, 5:25 AML godlike
03/27/2024, 6:57 AML godlike
03/27/2024, 6:58 AM