https://flyte.org logo
Join Slack
Powered by
Hi there! While deploying the flyte service in an ...
# flyte-support
d
Hi there! While deploying the flyte service in an EKS cluster for testing I changed the flyte projects to
Copy code
flyteadmin:
 initialProjects:
    - flytePOC
After that I started to see the following error in the 
syncresources
Copy code
{"json":{},"level":"error","msg":"Failed cluster resource creation loop with: Failed to create kubernetes object from config template [aa_namespace.yaml] for namespace [flytePOC-development] with err: Namespace \"flytePOC-development\" is invalid: metadata.name: Invalid value: \"flytePOC-development\": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character ....
so to undo this I just uninstalled the deployment
Copy code
helm uninstall
and removed the initialProjects setting from the original value.yaml file. The problem is that even after removing this, a fresh 
helm install...
still produces this error. What is the resource that needs to be deleted manually to remove the error? since I can't figure out from the error message (btw, I think that this is the issue, but it might be also something else)
a
what namespace are you using in the 
helm install
operation? Typically, the only Flyte resource that is left after a 
helm uninstall
operation is a 
MutatingWebhookConfiguration
kubectl get mutatingwebhookconfiguration
d
namespace is just flyte, as in the guide
Copy code
helm install -n flyte -f values-eks.yaml --create-namespace flyte flyteorg/flyte-core
a
not entirely sure that it's the cause, but please delete the 
MutatingWebhookConfiguration
that refers to Flyte and try again
d
sure, let me try
btw, also all these namespaces get created automatically
Copy code
flyteexamples-development   Active   23m
flyteexamples-production    Active   23m
flyteexamples-staging       Active   23m
flytesnacks-development     Active   23m
flytesnacks-production      Active   23m
flytesnacks-staging         Active   23m
flytetester-development     Active   23m
flytetester-production      Active   23m
flytetester-staging         Active   23m
+ the flyte one. I'll also delete those
a
what chart are you using?
d
Copy code
helm repo add flyteorg <https://flyteorg.github.io/flyte>
just the original one
👍🏽 1
I guess the uninstall should also delete those?
a
not necessarily. Namespaces are the substrate where Helm charts are installed/upgraded etc, so when a chart is uninstalled, it'd be unable to remove the namespace where it used to live
I se you\re using the 
flyte-core
chart. Those namespaces got created because this default setting:
Copy code
initialProjects:
    - flytesnacks
    - flytetester
    - flyteexamples
Typically, in order to run the examples in the docs, you'll only need 
flytesnacks
d
👍 any other resource that comes to your mind that might cause the issue before I rerun the install? 🙂
still same error
a
can you share the output of 
kubectl get all --all-namespaces
d
Removed all the sensitive information but for what flyte relates:
Copy code
NAMESPACE          NAME                                                         READY   STATUS                  RESTARTS          AGE
flyte              pod/datacatalog-59dcf9f974-l69k7                             1/1     Running                 0                 4m48s
flyte              pod/flyte-pod-webhook-568769565f-774jr                       1/1     Running                 0                 4m48s
flyte              pod/flyteadmin-6cb7759fcb-xd5sj                              0/1     Init:CrashLoopBackOff   5 (80s ago)       4m48s
flyte              pod/flyteconsole-784d7f5969-wc9lj                            1/1     Running                 0                 4m48s
flyte              pod/flytepropeller-76f8bd454d-79xcp                          1/1     Running                 0                 4m48s
flyte              pod/flytescheduler-689c576d79-j6nrf                          0/1     Init:CrashLoopBackOff   5 (109s ago)      4m48s
flyte              pod/syncresources-855bcbc6c9-m7r2j                           1/1     Running                 0                 4m48s


NAMESPACE          NAME                                                         TYPE           CLUSTER-IP       EXTERNAL-IP                                                                     PORT(S)                                    AGE
flyte              service/datacatalog                                          NodePort       xxxx    <none>                                                                          8089:32501/TCP,88:32339/TCP,89:32465/TCP   4m49s
flyte              service/flyte-pod-webhook                                    ClusterIP      xxxx   <none>                                                                          443/TCP                                    4m49s
flyte              service/flyteadmin                                           ClusterIP      xxxx    <none>                                                                          80/TCP,81/TCP,87/TCP,10254/TCP             4m49s
flyte              service/flyteconsole                                         ClusterIP      xxxx    <none>                                                                          80/TCP                                     4m49s
                                                                      
NAMESPACE          NAME                                                   READY   UP-TO-DATE   AVAILABLE   AGE
flyte              deployment.apps/datacatalog                            1/1     1            1           4m50s
flyte              deployment.apps/flyte-pod-webhook                      1/1     1            1           4m50s
flyte              deployment.apps/flyteadmin                             0/1     1            0           4m50s
flyte              deployment.apps/flyteconsole                           1/1     1            1           4m50s
flyte              deployment.apps/flytepropeller                         1/1     1            1           4m50s
flyte              deployment.apps/flytescheduler                         0/1     1            0           4m50s
flyte              deployment.apps/syncresources                          1/1     1            1           4m50s


NAMESPACE          NAME                                                              DESIRED   CURRENT   READY   AGE

flyte              replicaset.apps/datacatalog-59dcf9f974                            1         1         1       4m50s
flyte              replicaset.apps/flyte-pod-webhook-568769565f                      1         1         1       4m50s
flyte              replicaset.apps/flyteadmin-6cb7759fcb                             1         1         0       4m50s
flyte              replicaset.apps/flyteconsole-784d7f5969                           1         1         1       4m50s
flyte              replicaset.apps/flytepropeller-76f8bd454d                         1         1         1       4m50s
flyte              replicaset.apps/flytescheduler-689c576d79                         1         1         0       4m50s
flyte              replicaset.apps/syncresources-855bcbc6c9                          1         1         1       4m50s
flyte admin and schedule are crashlooping and I think that the reason is the one I posted
flyteadmin has error
Copy code
Back-off restarting failed container sync-cluster-resources in pod flyteadmin-
so it seems related to the 
syncresources
service issue
a
thanks what about 
kubectl logs syncresources-855bcbc6c9-m7r2j -n flyte
d
Copy code
{
  "json": {},
  "level": "error",
  "msg": "Failed cluster resource creation loop with: Failed to create kubernetes object from config template [aa_namespace.yaml] for namespace [flytePOC-development] with err: Namespace \"flytePOC-development\" is invalid: metadata.name: Invalid value: \"flytePOC-development\": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), Failed to create kubernetes object from config template [aab_default_service_account.yaml] for namespace [flytePOC-development] with err: namespaces \"flytePOC-development\" not found, Failed to create kubernetes object from config template [ab_project_resource_quota.yaml] for namespace [flytePOC-development] with err: namespaces \"flytePOC-development\" not found, Failed to create kubernetes object from config template [aa_namespace.yaml] for namespace [flytePOC-staging] with err: Namespace \"flytePOC-staging\" is invalid: metadata.name: Invalid value: \"flytePOC-staging\": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), Failed to create kubernetes object from config template [aab_default_service_account.yaml] for namespace [flytePOC-staging] with err: namespaces \"flytePOC-staging\" not found, Failed to create kubernetes object from config template [ab_project_resource_quota.yaml] for namespace [flytePOC-staging] with err: namespaces \"flytePOC-staging\" not found, Failed to create kubernetes object from config template [aa_namespace.yaml] for namespace [flytePOC-production] with err: Namespace \"flytePOC-production\" is invalid: metadata.name: Invalid value: \"flytePOC-production\": a lowercase RFC 1123 label must consist of lower case alphanumeric characters or '-', and must start and end with an alphanumeric character (e.g. 'my-name',  or '123-abc', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?'), Failed to create kubernetes object from config template [aab_default_service_account.yaml] for namespace [flytePOC-production] with err: namespaces \"flytePOC-production\" not found, Failed to create kubernetes object from config template [ab_project_resource_quota.yaml] for namespace [flytePOC-production] with err: namespaces \"flytePOC-production\" not found",
  "ts": "2023-08-07T14:56:19Z"
}
it gets angry about the creation of namespaces with capital letters, which were the ones I had defined at some point in values-eks.yaml. To ensure that those are gone, I re-downloaded the original file and I am applying that one.
As mentioned, I had set this at some point
Copy code
flyteadmin:
 initialProjects:
    - flytePOC
but I have removed it again and tried to uninstall and reinstall already a few times
I tried to render the local helm chart and it looks good, so there might be some resource in the eks cluster that is not being cleaned up. I'' try to reinstall in a new namspace
ok even a different namespace didn't do the trick. Same error on the 
syncresources
service
a
that's strange. Especially after a helm uninstall @dazzling-advantage-95256 may I suggest checking out this resource, already "battle tested" with the community to deploy Flyte on EKS? https://github.com/davidmirror-ops/flyte-the-hard-way
132 Views
Open in Slack
PreviousNext
Powered by