glamorous-father-92994
04/01/2022, 4:32 PM[1/1] currentAttempt done. Last Error: USER::[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[flyte-copilotdownloader] terminated with exit code (1). Reason [Error]. Message:
--storage.cache.max_size_mbs int Maximum size of the cache where the Blob store data is cached in-memory. If not specified or set to 0, cache is not used
--storage.cache.target_gc_percent int Sets the garbage collection target percentage.
I took a look at the config in the docs and looks like we need to add these. Is there anything else I’m missing and will these default values work for our deployment?
co-pilot:
cpu: 500m
default-input-path: /var/flyte/inputs
default-output-path: /var/flyte/outputs
image: <http://cr.flyte.org/flyteorg/flytecopilot:v0.0.15|cr.flyte.org/flyteorg/flytecopilot:v0.0.15>
input-vol-name: flyte-inputs
memory: 128Mi
name: flyte-copilot-
output-vol-name: flyte-outputs
start-timeout: 1m40s
storage: ""
freezing-airport-6809
great-school-54368
04/01/2022, 5:18 PMglamorous-father-92994
04/01/2022, 5:21 PMInit Containers:
flyte-copilotdownloader:
Container ID: <docker://3adf25af273b66204973e5e05bc2d95f546de53809e9b4e2ade381a91f3a300>5
Image: library.pdx.l5.woven-planet.tech/application/flyteplugins/flytecopilot:v0.0.1
Image ID: <docker-pullable://library.pdx.l5.woven-planet.tech/application/flyteplugins/flytecopilot@sha256:5f74032b747e38079cee222a345e8a8d553702fa3702c7c66370024ef2e4288a>
Port: <none>
Host Port: <none>
Command:
/bin/flyte-copilot
--storage.limits.maxDownloadMBs=0
--storage.type=s3
--storage.enable-multicontainer=false
--storage.container=lyft-av-prod-pdx-flyte
--storage.connection.secret-key=
--storage.connection.access-key=
--storage.connection.auth-type=iam
--storage.connection.region=us-west-2
--storage.connection.endpoint=
Args:
glamorous-father-92994
04/01/2022, 5:22 PMglamorous-father-92994
04/01/2022, 5:23 PMgreat-school-54368
04/01/2022, 5:24 PMglamorous-father-92994
04/01/2022, 5:28 PMglamorous-father-92994
04/01/2022, 5:29 PMgreat-school-54368
04/01/2022, 5:29 PMgreat-school-54368
04/01/2022, 5:30 PMgreat-school-54368
04/01/2022, 5:31 PMglamorous-father-92994
04/01/2022, 5:42 PMglamorous-father-92994
04/01/2022, 5:43 PMgreat-school-54368
04/03/2022, 9:10 AMfreezing-airport-6809
great-school-54368
04/03/2022, 2:15 PMglamorous-father-92994
04/06/2022, 12:47 PM[avcloud-prod-pdx] ➜ flytepropeller git:(master) kubectl -n prod logs -f l34ofjb147-fh3qf1ma-0 -c flyte-copilotdownloader
time="2022-04-06T01:33:39Z" level=info msg="[0] Couldn't find a config file []. Relying on env vars and pflags."
{"json":{},"level":"error","msg":"Failed to Get credentials.","ts":"2022-04-06T01:33:44Z"}
{"json":{},"level":"error","msg":"Failed to Get credentials.","ts":"2022-04-06T01:33:49Z"}
{"json":{},"level":"error","msg":"Failed to Get credentials.","ts":"2022-04-06T01:33:54Z"}
{"json":{},"level":"error","msg":"Failed to Get credentials.","ts":"2022-04-06T01:33:59Z"}
{"json":{},"level":"error","msg":"Failed to Get credentials.","ts":"2022-04-06T01:34:04Z"}
Any ideas or pointers?
cc: @astonishing-lizard-78628great-school-54368
04/06/2022, 12:48 PMglamorous-father-92994
04/06/2022, 7:31 PMk8s:
co-pilot:
name: "flyte-copilot"
image: ${ecr_repository}:flytecopilot-${flytecopilot_version}
start-timeout: "30s"
scheduler-name: flyte-scheduler
inject-finalizer: true
The yaml for the pod itself is
- args:
- sidecar
- --start-timeout
- 30s
- --to-raw-output
- <s3://lyft-av-prod-pdx-flyte/49/sh9aeoqd2s-fiaqf4qy-0>
- --to-output-prefix
- <s3://lyft-av-prod-pdx-flyte/metadata/propeller/production/avperceptionworkflows-prod-sh9aeoqd2s/n1/data/0/dynamic-run-metrics-workers-n5/0>
- --from-local-dir
- /var/flyte/outputs
- --interface
- CgASAA==
command:
- /bin/flyte-copilot
- --storage.limits.maxDownloadMBs=0
- --storage.type=s3
- --storage.enable-multicontainer=false
- --storage.container=lyft-av-prod-pdx-flyte
- --storage.connection.secret-key=
- --storage.connection.access-key=
- --storage.connection.auth-type=iam
- --storage.connection.region=us-west-2
- --storage.connection.endpoint=
env:
- name: L5_DATACENTER
value: pdx
- name: L5_BASE_DOMAIN
value: l5.woven-planet.tech
- name: L5_ENVIRONMENT
value: pdx
- name: RUNTIME_POD_NAME
valueFrom:
fieldRef:
average-school-71780
03/06/2024, 5:32 AM