wonderful-piano-17919
02/24/2023, 2:05 PMflyte-binary
helm chart with eks-starter.yaml
?wonderful-piano-17919
02/24/2023, 2:05 PM"Requested MEMORY limit [2Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration"}"
freezing-airport-6809
wonderful-piano-17919
02/24/2023, 2:48 PMwonderful-piano-17919
02/24/2023, 2:51 PMwonderful-piano-17919
02/24/2023, 2:51 PMflytectl get task-resource-attribute -p flytesnacks -d development
{"project":"flytesnacks","domain":"development","defaults":{"cpu":"2","memory":"2Gi"},"limits":{"cpu":"4","memory":"8Gi"}}
wonderful-piano-17919
02/24/2023, 2:51 PMpyflyte run --remote ...
wonderful-piano-17919
02/24/2023, 2:51 PMdebug_error_string = "UNKNOWN:Error received from peer ipv6:%5B::1%5D:8089 {grpc_message:"Requested MEMORY limit [2Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration", grpc_status:3, created_time:"2023-02-24T15:51:00.19866+01:00"}"
freezing-airport-6809
wonderful-piano-17919
02/24/2023, 2:53 PMflytectl get task-resource-attribute -p flytesnacks -d development
{"project":"flytesnacks","domain":"development","limits":{"cpu":"4","memory":"8Gi"}}
wonderful-piano-17919
02/24/2023, 2:53 PMwonderful-piano-17919
02/24/2023, 2:53 PMfreezing-airport-6809
wonderful-piano-17919
02/24/2023, 2:55 PMwonderful-piano-17919
02/24/2023, 2:57 PMwonderful-piano-17919
02/24/2023, 3:02 PMfreezing-airport-6809
wonderful-piano-17919
02/24/2023, 3:07 PMkubectl -n flyte scale deployment flyte-backend-flyte-binary --replicas=0
# Wait 1 minute
kubectl -n flyte scale deployment flyte-backend-flyte-binary --replicas=1
wonderful-piano-17919
02/24/2023, 3:07 PMflytectl
changes anything in the helm chart itself...freezing-airport-6809
wonderful-piano-17919
02/24/2023, 3:08 PMfreezing-airport-6809
freezing-airport-6809
wonderful-piano-17919
02/24/2023, 3:12 PMfreezing-airport-6809
wonderful-piano-17919
02/24/2023, 3:13 PMfreezing-airport-6809
freezing-airport-6809
wonderful-piano-17919
02/24/2023, 3:14 PMfreezing-airport-6809
freezing-airport-6809
wonderful-piano-17919
02/24/2023, 3:15 PMwonderful-piano-17919
02/24/2023, 3:15 PMconfiguration:
task_resource_defaults:
task_resources:
defaults:
cpu: 1
memory: 2Gi
limits:
cpu: 4
memory: 8Gi
wonderful-piano-17919
02/24/2023, 3:15 PMfreezing-airport-6809
wonderful-piano-17919
02/24/2023, 3:16 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
wonderful-piano-17919
02/24/2023, 3:18 PMfreezing-airport-6809
freezing-boots-56761
freezing-boots-56761
wonderful-piano-17919
02/24/2023, 4:21 PMfreezing-boots-56761
configuration:
inline:
task_resource_defaults:
task_resources:
defaults:
cpu: 1
memory: 2Gi
limits:
cpu: 4
memory: 8Gi
wonderful-piano-17919
02/24/2023, 4:38 PMhelm upgrade flyte-backend ...
but the changes weren't reflected, even though the deployment restartedfreezing-boots-56761
freezing-boots-56761
wonderful-piano-17919
02/24/2023, 4:39 PMwonderful-piano-17919
02/24/2023, 4:39 PM2023/02/24 16:39:13 /go/pkg/mod/gorm.io/driver/postgres@v1.2.3/migrator.go:106
[0.495ms] [rows:1] SELECT count(*) FROM pg_indexes WHERE tablename = 'artifacts' AND indexname = 'artifacts_dataset_uuid_idx' AND schemaname = CURRENT_SCHEMA()
{"metrics-prefix":"flyte:","certDir":"/var/run/flyte/certs","localCert":true,"listenPort":9443,"serviceName":"flyte-backend-flyte-binary-webhook","servicePort":443,"secretName":"flyte-backend-flyte-binary-webhook-secret","secretManagerType":"K8s","awsSecretManager":{"sidecarImage":"<http://docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4|docker.io/amazon/aws-secrets-manager-secret-sidecar:v0.1.4>","resources":{"limits":{"cpu":"200m","memory":"500Mi"},"requests":{"cpu":"200m","memory":"500Mi"}}},"vaultSecretManager":{"role":"flyte","kvVersion":"2"}}
freezing-boots-56761
configuration:
inline:
task_resources:
defaults:
cpu: 1
memory: 2Gi
limits:
cpu: 4
memory: 8Gi
freezing-boots-56761
wonderful-piano-17919
02/24/2023, 4:40 PMfreezing-boots-56761
wonderful-piano-17919
02/24/2023, 4:42 PMconfiguration:
...
inline:
task_resources:
defaults:
cpu: 1
memory: 2Gi
limits:
cpu: 4
memory: 8Gi
wonderful-piano-17919
02/24/2023, 4:42 PMwonderful-piano-17919
02/24/2023, 4:43 PMfreezing-boots-56761
wonderful-piano-17919
02/24/2023, 4:46 PMwonderful-piano-17919
02/24/2023, 4:46 PMLimits:
cpu: 1
memory: 2Gi
Requests:
cpu: 1
memory: 2Gi
wonderful-piano-17919
02/24/2023, 4:46 PMfreezing-boots-56761
wonderful-piano-17919
02/24/2023, 4:49 PMflyte-binary
as per the doc, and ran into OOMKilled
, then tried to increase the resources with the @task
decorator, and then ran into this resource limitwonderful-piano-17919
02/24/2023, 4:49 PMfreezing-boots-56761
wonderful-piano-17919
02/27/2023, 1:16 PMfreezing-airport-6809
Yes this was indeed the idea - cc @thankful-minister-83577 / @high-accountant-32689 we should do this.
thankful-minister-83577
freezing-airport-6809
thankful-minister-83577
thankful-minister-83577
thankful-minister-83577
resources: {}
thankful-minister-83577
{}
millions-table-8574
03/10/2023, 9:22 AMError from server (NotFound): configmaps "sandbox-flyte-binary-config" not found
and when checking:
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get cm -n flyte
NAME DATA AGE
flyte-sandbox-cluster-resource-templates 1 16m
flyte-sandbox-config 5 16m
flyte-sandbox-docker-registry-config 1 16m
flyte-sandbox-extra-cluster-resource-templates 0 16m
flyte-sandbox-extra-config 0 16m
flyte-sandbox-proxy-config 1 16m
kubernetes-dashboard-settings 0 16m
kube-root-ca.crt 1 16m
(flyte) briant@Ashriels-MacBook-Pro flyte-test % k get deployment -n flyte
NAME READY UP-TO-DATE AVAILABLE AGE
flyte-sandbox-proxy 1/1 1 1 17m
flyte-sandbox-kubernetes-dashboard 1/1 1 1 17m
flyte-sandbox-docker-registry 1/1 1 1 17m
flyte-sandbox-minio 1/1 1 1 17m
flyte-sandbox 1/1 1 1 17m
millions-table-8574
03/10/2023, 9:25 AMfreezing-boots-56761