https://flyte.org logo
#ask-the-community
Title
# ask-the-community
f

Frank Shen

01/19/2024, 12:58 AM
Hello, I have a run Dev flyte cluster with domain id: development, name: development. I tried to deploy via helm chart another flyte cluster with domain id: integration, name: integration. Flyteadmin pods failed for : sync-cluster-resources {“json”{},“level”“warning”,“msg”:“Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flyteexamples-integration] with err: namespaces \“flyteexamples-integration\” not found”,“ts”“2024 01 18T1939:43Z”} sync-cluster-resources {“json”{},“level”“warning”,“msg”:“Failed to create cluster resources for namespace [flyteexamples-integration] with err: Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flyteexamples-integration] with err: namespaces \“flyteexamples-integration\” not found”,“ts”“2024 01 18T1939:43Z”}Now in the Dev cluster, namespace flyteexamples-development was created successfully and being used by flyte transient pods when I use flyteexamples as project. I couldn’t figure out the root cause after exploring the templates. Could you suggest anything? Thanks!
d

David Espejo (he/him)

01/19/2024, 2:55 PM
@Frank Shen typically what you set here https://github.com/flyteorg/flyte/blob/2866d5ac032df1175c93094f3004bb1e6a00dd14/charts/flyte-core/values.yaml#L47 and if
cluster_resource_manager.enabled
is set to true, the config will be eventually reconciled and the namespaces will be created
f

Frank Shen

01/19/2024, 4:12 PM
Hi @David Espejo (he/him) I did all of those.
@David Espejo (he/him), @Kevin Su, @Samhita Alla How are the namespaces created? Could you point me to the flyte templates’ code? Thanks
d

David Espejo (he/him)

01/19/2024, 8:14 PM
This controller is what reads from the flyteadmin db and reconciles into K8s resources (namespaces in this case) https://github.com/flyteorg/flyte/blob/master/flyteadmin/pkg/clusterresource/controller.go
f

Frank Shen

01/19/2024, 8:21 PM
@David Espejo (he/him), thanks. So maybe it’s not connected to the db? datacatalog and syncresources pods are running fine. They both need to connect to the same db.
d

David Espejo (he/him)

01/19/2024, 8:24 PM
is this a multi cluster setup?
f

Frank Shen

01/19/2024, 8:24 PM
yes
@David Espejo (he/him), The flyte console is returning 503, and showing a blank page. https://flyte.int.api.discomax.com/console/
d

David Espejo (he/him)

01/19/2024, 8:29 PM
anything interesting in the logs?
f

Frank Shen

01/19/2024, 8:30 PM
only this:
Copy code
Express server ready!                                                                                                                                                                                                                      │
│ App started with the following config: {                                                                                                                                                                                                   │
│   "ADMIN_API_USE_SSL": "http",                                                                                                                                                                                                             │
│   "BASE_URL": "/console",                                                                                                                                                                                                                  │
│   "NODE_ENV": "production",                                                                                                                                                                                                                │
│   "ENABLE_GA": "false",                                                                                                                                                                                                                    │
│   "GA_TRACKING_ID": "",                                                                                                                                                                                                                    │
│   "ASSETS_PATH": "/console/assets/",                                                                                                                                                                                                       │
│   "CERTIFICATE_PATH": "../script/certificate",                                                                                                                                                                                             │
│   "HORIZONTAL_LAYOUT": "false",                                                                                                                                                                                                            │
│   "processEnv": {                                                                                                                                                                                                                          │
│     "BASE_URL": "/console",                                                                                                                                                                                                                │
│     "ENABLE_GA": "false",                                                                                                                                                                                                                  │
│     "GA_TRACKING_ID": "",                                                                                                                                                                                                                  │
│     "NODE_ENV": "production",                                                                                                                                                                                                              │
│     "FLYTE_NAVIGATION": "",                                                                                                                                                                                                                │
│     "HORIZONTAL_LAYOUT": "false"                                                                                                                                                                                                           │
│   }                                                                                                                                                                                                                                        │
│ }                                                                                                                                                                                                                                          │
│ Please open <http://undefined:8080/console>
Both flyteadmin and flytescheduler pods are in Init:CrashLoopBackOff and not running.
@David Espejo (he/him)
d

David Espejo (he/him)

01/19/2024, 8:32 PM
please share
kubectl logs <flyteadmin-pod-name> -n flyte
f

Frank Shen

01/19/2024, 8:34 PM
Copy code
sync-cluster-resources panic: Failed to sync cluster resources : Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flyteexamples-integration] with err: namespaces "flyteexamples-integration" not found, Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flytetester-integration] with err: namespaces "flytetester-integration" not found, Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flytesnacks-integration] with err: namespaces "flytesnacks-integration" not found
sync-cluster-resources 
sync-cluster-resources goroutine 1 [running]:
sync-cluster-resources main.main()
sync-cluster-resources     /go/src/github.com/flyteorg/flyteadmin/cmd/main.go:14 +0x9f
@David Espejo (he/him)
which flyte pod or service is writing to the db for the domains (e.g. flyteexamples)? I want to check its log to see if there is error preventing it from writing it.
@David Espejo (he/him) '
Hi @David Espejo (he/him), What is the proper way to re-deploy or re-start Flyte? I suspect Flyte needs to be re-started to insert the namespaces to DB.
d

David Espejo (he/him)

01/22/2024, 6:49 PM
which flyte pod or service is writing to the db for the domains (e.g. flyteexamples)?
Hi Frank, it is flyteadmin
If you absolutely need to uninstall Flyte, a `helm uninstall <release name> -n flyte`should be enough. It's very likely that there will a CRD that's not deleted by Helm (this is a Helm limitation) If, when you're installing again, you find an error about a
flyteworkflow
CRD that already exists, please delete it manually like
kubectl delete crd <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com>
and then proceeed with
helm install ...
as usual
f

Frank Shen

01/22/2024, 6:59 PM
Hi @David Espejo (he/him) That a very good lead. I see flyteworkflows.flyte.lyft.com that has been 340 days old. I cannot uninstall it without going through hoops. Can I just delete the CRD and then try to re-deploy flyteadmin?
d

David Espejo (he/him)

01/22/2024, 8:18 PM
that CRD can only be deleted when Flyte is uninstalled
you can restart Flyteadmin by doing a
kubectl rollout restart deployment/flyteadmin -n flyte
f

Frank Shen

01/22/2024, 8:28 PM
Thank you @David Espejo (he/him), I did rollout restart, but it didn’t fix the issue.
I also have issues with the flytescheduler and flyteconsole. I asked in the community channel. Could you help with those?