My understanding of the cluster resources <docs> i...
# flyte-deployment
z
My understanding of the cluster resources docs is that anything applied from
cra.yaml
will be applied in k8s via the
resourcequota
on the project-domain namespace. Is that not the case? After updating
cluster-resources-attribute
, I'm seeing following mismatch:
Copy code
flytectl get cluster-resource-attribute -p flytesnacks -d development
returns
Copy code
{
  "project": "flytesnacks",
  "domain": "development",
  "attributes": {
    "projectQuotaCpu": "10",
    "projectQuotaMemory": "14G"
  }
}
but
Copy code
kubectl -n flytesnacks-development get resourcequotas
returns
Copy code
NAME            AGE    REQUEST   LIMIT
project-quota   142m             limits.cpu: 0/4, limits.memory: 0/3000Mi
k
hey @Zachary Kimble are you running the cluster resource controller sync command as a container? what are the template files passed to the cluster resource controller?
z
I'm not sure. I'm using the sandbox deployment, so I would think not. I do not know how to check flyteadmin config. I'm attempting to follow this page I wrote the following `cra.yaml`:
Copy code
attributes:
    projectQuotaCpu: "10"
    projectQuotaMemory: "14G"
domain: development
project: flytesnacks
And then applying with
flytectl update cluster-resource-attribute --attrFile cra.yaml
k
ah one more question, are you running
flytectl sandbox start
or
flytectl demo start
?
z
flytectl sandbox start
k
great, i believe that should bring up the sync container, can i ask you to double check that the deployment/pod for it is happy?
z
The
syncresouces
pod is status ready. It's log shows an error, but I think it's from when the cluster started not from when I updated the cra.
Copy code
time="2022-10-06T17:42:59Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/db.yaml /etc/flyte/config/domain.yaml]"

2022/10/06 17:42:59 /go/src/github.com/flyteorg/flyteadmin/pkg/repositories/gormimpl/project_repo.go:80 ERROR: relation "projects" does not exist (SQLSTATE 42P01)
[0.455ms] [rows:0] SELECT * FROM "projects" WHERE state <> 1 ORDER BY created_at desc 
{"json":{},"level":"warning","msg":"Failed cluster resource creation loop with: cannot query with specified table attributes: relation \"projects\" does not exist","ts":"2022-10-06T17:42:59Z"}
k
oh! this is an issue - are your flyteadmin and postgres pods running? it looks like your database is not migrated/misconfigured
z
no errors for flyteadmin. postgres pod has these errors:
Copy code
2022-10-06 17:42:24.344 UTC [1] LOG:  database system is ready to accept connections
2022-10-06 17:42:52.020 UTC [74] FATAL:  database "datacatalog" does not exist
2022-10-06 17:42:59.099 UTC [78] ERROR:  relation "projects" does not exist at character 15
2022-10-06 17:42:59.099 UTC [78] STATEMENT:  SELECT * FROM "projects" WHERE state <> $1 ORDER BY created_at desc
k
do you mind restarting the admin pod? the init containers should run the migrations to create the databases and tables
oh and restart the datacatalog one as well
z
Will do. I have meetings for an hour, but will let you know what happens after.
so after restarting both (with
rollout restart
), the
flytesnacks-development project-quota
is synced correctly. If i update the cra again, then those changes don't make it to the
project-quota
. Restarting flytedmin again then causes them to sync.
113 Views