Hello, Could anyone point me to how the extra namespaces (e.g. flytesnacks-development) that the pods for tasks are running under are managed (created, etc.) by Flyte? I think it’s in the GO code. I need to understand it to trouble-shoot why these namespaces were not found by flyteadmin at containers startup.

sync-cluster-resources panic: Failed to sync cluster resources : Failed to create kubernetes object from config template [ad_spark_service_account.yaml] for namespace [flyteexamples-integration] with err: namespaces "flyteexamples-integration" not found,

Thanks.

what namespaces do you currently have?

Hi @tall-lock-23197. namespace: flyte

@salmon-refrigerator-32115 The Cluster Resource Manager is the component that connects to flyteadmin, get the projects and reconcille the info it fins there into K8s resources. In the case of Spark, it's not only namespaces but Roles, SAs, etc This is the part of the code where the CRM interfaces with admin: https://github.com/flyteorg/flyte/blob/fb9ffd56e81e7f7e4657cd668e53b2f1557e9178/flyteadmin/pkg/clusterresource/interfaces/admin.go The CRM is enabled by default but it's worth to double check that this is set: https://github.com/flyteorg/flyte/blob/fb9ffd56e81e7f7e4657cd668e53b2f1557e9178/charts/flyte-core/values.yaml#L838

Hi @average-finland-92144, @tall-lock-23197, I was able to pinpoint the root cause of the namespace not found issue after digging : I added the single template to the values override file because I need the annotations:

cluster_resource_manager:
            templates:
              - key: ad_spark_service_account
                value: |
                  apiVersion: v1
                  kind: ServiceAccount
                  metadata:
                    name: spark
                    namespace: {{ namespace }}
                    annotations:
                      <http://eks.amazonaws.com/role-arn|eks.amazonaws.com/role-arn>: '{{ defaultIamRole }}'

And since the templates must be executed by the GO code in sequence to create the namespace first, the above change somehow caused that template to be executed first, but since no namespaces has been created, hans the error. I’ve added the flyte template annotation to the main flyte charts and remove the template from the values override file., And that resolved the namespace not found issue. Thank you very much @average-finland-92144 for the help!

Oh I get it, thanks for sharing @salmon-refrigerator-32115!