Hello,
I have a run Dev flyte cluster with domain ...
# ask-the-communityf
Frank Shen
01/19/2024, 12:58 AMHello,
I have a run Dev flyte cluster with domain id: development, name: development.
I tried to deploy via helm chart another flyte cluster with domain id: integration, name: integration.
Flyteadmin pods failed for :
sync-cluster-resources {“json”{},“level”“warning”,“msg”:“Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flyteexamples-integration] with err: namespaces \“flyteexamples-integration\” not found”,“ts”“2024 01 18T1939:43Z”}
sync-cluster-resources {“json”{},“level”“warning”,“msg”:“Failed to create cluster resources for namespace [flyteexamples-integration] with err: Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flyteexamples-integration] with err: namespaces \“flyteexamples-integration\” not found”,“ts”“2024 01 18T1939:43Z”}Now in the Dev cluster, namespace flyteexamples-development was created successfully and being used by flyte transient pods when I use flyteexamples as project.
I couldn’t figure out the root cause after exploring the templates.
Could you suggest anything?
Thanks!
d
David Espejo (he/him)
01/19/2024, 2:55 PM@Frank Shen typically what you set here
https://github.com/flyteorg/flyte/blob/2866d5ac032df1175c93094f3004bb1e6a00dd14/charts/flyte-core/values.yaml#L47
and if
cluster_resource_manager.enabledf
Frank Shen
01/19/2024, 4:12 PMHi @David Espejo (he/him)
I did all of those.
Frank Shen
01/19/2024, 4:40 PM@David Espejo (he/him), @Kevin Su, @Samhita Alla
How are the namespaces created?
Could you point me to the flyte templates’ code?
Thanks
d
David Espejo (he/him)
01/19/2024, 8:14 PMThis controller is what reads from the flyteadmin db and reconciles into K8s resources (namespaces in this case)
https://github.com/flyteorg/flyte/blob/master/flyteadmin/pkg/clusterresource/controller.go
f
Frank Shen
01/19/2024, 8:21 PM@David Espejo (he/him), thanks.
So maybe it’s not connected to the db?
datacatalog and syncresources pods are running fine.
They both need to connect to the same db.
d
David Espejo (he/him)
01/19/2024, 8:24 PMis this a multi cluster setup?
f
Frank Shen
01/19/2024, 8:24 PMyes
Frank Shen
01/19/2024, 8:28 PM@David Espejo (he/him),
The flyte console is returning 503, and showing a blank page.
https://flyte.int.api.discomax.com/console/
d
David Espejo (he/him)
01/19/2024, 8:29 PManything interesting in the logs?
f
Frank Shen
01/19/2024, 8:30 PMonly this:
Copy code
Express server ready! │
│ App started with the following config: { │
│ "ADMIN_API_USE_SSL": "http", │
│ "BASE_URL": "/console", │
│ "NODE_ENV": "production", │
│ "ENABLE_GA": "false", │
│ "GA_TRACKING_ID": "", │
│ "ASSETS_PATH": "/console/assets/", │
│ "CERTIFICATE_PATH": "../script/certificate", │
│ "HORIZONTAL_LAYOUT": "false", │
│ "processEnv": { │
│ "BASE_URL": "/console", │
│ "ENABLE_GA": "false", │
│ "GA_TRACKING_ID": "", │
│ "NODE_ENV": "production", │
│ "FLYTE_NAVIGATION": "", │
│ "HORIZONTAL_LAYOUT": "false" │
│ } │
│ } │
│ Please open <http://undefined:8080/console>Frank Shen
01/19/2024, 8:31 PMBoth flyteadmin and flytescheduler pods are in Init:CrashLoopBackOff and not running.
Frank Shen
01/19/2024, 8:31 PM@David Espejo (he/him)
d
David Espejo (he/him)
01/19/2024, 8:32 PMplease share
kubectl logs <flyteadmin-pod-name> -n flytef
Frank Shen
01/19/2024, 8:34 PM Copy code
sync-cluster-resources panic: Failed to sync cluster resources : Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flyteexamples-integration] with err: namespaces "flyteexamples-integration" not found, Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flytetester-integration] with err: namespaces "flytetester-integration" not found, Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flytesnacks-integration] with err: namespaces "flytesnacks-integration" not found
sync-cluster-resources
sync-cluster-resources goroutine 1 [running]:
sync-cluster-resources main.main()
sync-cluster-resources /go/src/github.com/flyteorg/flyteadmin/cmd/main.go:14 +0x9fFrank Shen
01/19/2024, 8:36 PM@David Espejo (he/him)
Frank Shen
01/19/2024, 8:43 PMwhich flyte pod or service is writing to the db for the domains (e.g. flyteexamples)?
I want to check its log to see if there is error preventing it from writing it.
Frank Shen
01/19/2024, 8:43 PM@David Espejo (he/him) '
Frank Shen
01/22/2024, 5:23 PMHi @David Espejo (he/him),
What is the proper way to re-deploy or re-start Flyte?
I suspect Flyte needs to be re-started to insert the namespaces to DB.
d
David Espejo (he/him)
01/22/2024, 6:49 PMwhich flyte pod or service is writing to the db for the domains (e.g. flyteexamples)?Hi Frank, it is flyteadmin
David Espejo (he/him)
01/22/2024, 6:51 PMIf you absolutely need to uninstall Flyte, a `helm uninstall <release name> -n flyte`should be enough.
It's very likely that there will a CRD that's not deleted by Helm (this is a Helm limitation)
If, when you're installing again, you find an error about a
flyteworkflowkubectl delete crd <http://flyteworkflows.flyte.lyft.com|flyteworkflows.flyte.lyft.com>helm install ...f
Frank Shen
01/22/2024, 6:59 PMHi @David Espejo (he/him)
That a very good lead.
I see flyteworkflows.flyte.lyft.com that has been 340 days old.
I cannot uninstall it without going through hoops.
Can I just delete the CRD and then try to re-deploy flyteadmin?
d
David Espejo (he/him)
01/22/2024, 8:18 PMthat CRD can only be deleted when Flyte is uninstalled
David Espejo (he/him)
01/22/2024, 8:19 PMyou can restart Flyteadmin by doing a
kubectl rollout restart deployment/flyteadmin -n flytef
Frank Shen
01/22/2024, 8:28 PMThank you @David Espejo (he/him),
I did rollout restart, but it didn’t fix the issue.
Frank Shen
01/22/2024, 8:29 PMI also have issues with the flytescheduler and flyteconsole.
I asked in the community channel.
Could you help with those?
3 Views