Hello, im currently deploying flyte-binary on a quite standard GKE cluster. When i wanted to execute the hello_world workflow I noticed that the default sa in the flytesnacks namespace does not have any annotation for the gcp sa attached to it and flyte-binary logs:

{"json":{"src":"controller.go:602"},"level":"warning","msg":"Failed to create cluster resources for namespace [flytesnacks-production] with err: Failed to create kubernetes object from config template [002_serviceaccount.yaml] for namespace [flytesnacks-production] with err: serviceaccounts is forbidden: User \"sys
tem:serviceaccount:flyte:flyte-backend-flyte-binary\" cannot create resource \"serviceaccounts\" in API group \"\" in the namespace \"flytesnacks-production\"","ts":"2024-03-05T14:11:44Z"}

my current helm release looks like:

inline:
        tasks:
          task-plugins:
            enabled-plugins:
              - container
              - sidecar
              - K8S-ARRAY
            default-for-task-types:
              - container: container
              - container_array: k8s-array
        cluster_resources:
          customData:
          - production:
            - gsa:
                value: <mailto:cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com|cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com>
          - staging:
            - gsa:
                value: <mailto:cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com|cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com>
          - development:
            - gsa:
                value: <mailto:cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com|cool-gcp-sa@gke-foo-bar.iam.gserviceaccount.com>

    clusterResourceTemplates:
      inline:
        001_namespace.yaml: |
          apiVersion: v1
          kind: Namespace
          metadata:
            name: '{{ namespace }}'
        002_serviceaccount.yaml: |
          apiVersion: v1
          kind: ServiceAccount
          metadata:
            name: default
            namespace: '{{ namespace }}'
            annotations:
              <http://iam.gke.io/gcp-service-account|iam.gke.io/gcp-service-account>: '{{ gsa }}'

Help would be much appreciated, thanks in advance.

Welcome @Jonas G The only thing I see off so far in your config is the indentation under

customData

Considering what's in the reference implementation The modules there use Workload Identity so there are bindings at multiple levels. Let me know if some of this is helpful

Hey, thanks for the quick response. Sadly neither the indentation nor a more similar implementation to the reference project fixed the problem. The only think i could validate is that the iap binding works in theory when editing the serviceaccount manually.