https://flyte.org
Join Slack
My understanding of the cluster resources <docs> is that anything applied from `cra.yaml` will be ap...
# flyte-deployment
h
My understanding of the cluster resources docs is that anything applied from 
cra.yaml
will be applied in k8s via the 
resourcequota
on the project-domain namespace. Is that not the case? After updating 
cluster-resources-attribute
, I'm seeing following mismatch:
Copy code
flytectl get cluster-resource-attribute -p flytesnacks -d development
returns
Copy code
{
  "project": "flytesnacks",
  "domain": "development",
  "attributes": {
    "projectQuotaCpu": "10",
    "projectQuotaMemory": "14G"
  }
}
but
Copy code
kubectl -n flytesnacks-development get resourcequotas
returns
Copy code
NAME            AGE    REQUEST   LIMIT
project-quota   142m             limits.cpu: 0/4, limits.memory: 0/3000Mi
a
hey @high-garage-86992 are you running the cluster resource controller sync command as a container? what are the template files passed to the cluster resource controller?
h
I'm not sure. I'm using the sandbox deployment, so I would think not. I do not know how to check flyteadmin config. I'm attempting to follow this page I wrote the following `cra.yaml`:
Copy code
attributes:
    projectQuotaCpu: "10"
    projectQuotaMemory: "14G"
domain: development
project: flytesnacks
And then applying with 
flytectl update cluster-resource-attribute --attrFile cra.yaml
a
ah one more question, are you running 
flytectl sandbox start
or 
flytectl demo start
?
h
flytectl sandbox start
a
great, i believe that should bring up the sync container, can i ask you to double check that the deployment/pod for it is happy?
h
The 
syncresouces
pod is status ready. It's log shows an error, but I think it's from when the cluster started not from when I updated the cra.
Copy code
time="2022-10-06T17:42:59Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml]"

2022/10/06 17:42:59 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/project_repo.go:80 ERROR: relation "projects" does not exist (SQLSTATE 42P01)
[0.455ms] [rows:0] SELECT * FROM "projects" WHERE state <> 1 ORDER BY created_at desc 
{"json":{},"level":"warning","msg":"Failed cluster resource creation loop with: cannot query with specified table attributes: relation \"projects\" does not exist","ts":"2022-10-06T17:42:59Z"}
a
oh! this is an issue - are your flyteadmin and postgres pods running? it looks like your database is not migrated/misconfigured
h
no errors for flyteadmin. postgres pod has these errors:
Copy code
2022-10-06 17:42:24.344 UTC [1] LOG:  database system is ready to accept connections
2022-10-06 17:42:52.020 UTC [74] FATAL:  database "datacatalog" does not exist
2022-10-06 17:42:59.099 UTC [78] ERROR:  relation "projects" does not exist at character 15
2022-10-06 17:42:59.099 UTC [78] STATEMENT:  SELECT * FROM "projects" WHERE state <> $1 ORDER BY created_at desc
a
do you mind restarting the admin pod? the init containers should run the migrations to create the databases and tables
oh and restart the datacatalog one as well
h
Will do. I have meetings for an hour, but will let you know what happens after.
🙏 1
so after restarting both (with 
rollout restart
), the 
flytesnacks-development project-quota
is synced correctly. If i update the cra again, then those changes don't make it to the 
project-quota
. Restarting flytedmin again then causes them to sync.
175 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by