#3267 [BUG] Datacatalog deployment is stuck with permission issues on GCP Issue created by afridigithub Describe the bug I am trying to install flyte on a GCP, I see below error with datacatalog deployment. All other deployments are up and running with same GSA (with permissions superset of all required permissions).

{"json":{},"level":"error","msg":"Container [<gcp-bucket-name>] lookup failed. Error Get \"<https://storage.googleapis.com/storage/v1/b/<gcp-bucket-name>?alt=json\u0026prettyPrint=false\u0026projection=full\>": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\t<https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`%22,%22ts%22:%222023-01-23T12:56:22Z%22}|https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`","ts":"2023-01-23T12:56:22Z"}>
{"json":{"app_name":"datacatalog"},"level":"error","msg":"Failed to create DataStore \u0026{stow {{{      false   }} iam   us-east-1 false} {google map[json: project_id:prj-moj-p-ds-training scopes:<https://www.googleapis.com/auth/devstorage.read_write>]} <gcp-bucket-name> false {0 0} {10} {map[] 0s} {map[]}}, err Get \"<https://storage.googleapis.com/storage/v1/b/<gcp-bucket-name>e?alt=json\u0026prettyPrint=false\u0026projection=full\>": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\t<https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`%22,%22ts%22:%222023-01-23T12:56:22Z%22}|https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n`","ts":"2023-01-23T12:56:22Z"}>
{"json":{},"level":"fatal","msg":"caught panic: Get \"<https://storage.googleapis.com/storage/v1/b/<gcp-bucket-name>?alt=json\u0026prettyPrint=false\u0026projection=full\>": compute: Received 403 `Unable to generate access token; IAM returned 403 Forbidden: Permission 'iam.serviceAccounts.getAccessToken' denied on resource (or it may not exist).\nThis error could be caused by a missing IAM policy binding on the target IAM service account.\nFor more information, refer to the Workload Identity documentation:\n\t<https://cloud.google.com/kubernetes-engine/docs/how-to/workload-identity#authenticating_to\n\n>` [goroutine 1 [running]:\nruntime/debug.Stack()\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x65\<http://ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:86|ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService.func1()\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:86> +0x9c\npanic({0x1222fc0, 0xc000a148a0})\n\t/usr/local/go/src/runtime/panic.go:838 +0x207\<http://ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:94|ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.NewDataCatalogService()\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:94> +0x906\<http://ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc00093fca8|ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.newGRPCServer({0xc00093fca8>?, 0x474ddc?}, 0xc000143fb0)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:143 +0x33\<http://ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x16bba38|ngithub.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice.ServeInsecure({0x16bba38>, 0xc000046018}, 0xc000143fb0)\n\t/go/src/github.com/flyteorg/datacatalog/pkg/rpc/datacatalogservice/service.go:129 +0x45\<http://ngithub.com/flyteorg/datacatalog/cmd/entrypoints.glob..func3(0x1f7f6e0|ngithub.com/flyteorg/datacatalog/cmd/entrypoints.glob..func3(0x1f7f6e0>?, {0x1382675?, 0x2?, 0x2?})\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/serve.go:33 +0x115\<http://ngithub.com/spf13/cobra.(*Command).execute(0x1f7f6e0|ngithub.com/spf13/cobra.(*Command).execute(0x1f7f6e0>, {0xc0002b4540, 0x2, 0x2})\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:856 +0x67c\<http://ngithub.com/spf13/cobra.(*Command).ExecuteC(0x1f7f460)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974|ngithub.com/spf13/cobra.(*Command).ExecuteC(0x1f7f460)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:974> +0x3b4\<http://ngithub.com/spf13/cobra.(*Command).Execute(...)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\ngithub.com/flyteorg/datacatalog/cmd/entrypoints.Execute()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/root.go:46|ngithub.com/spf13/cobra.(*Command).Execute(...)\n\t/go/pkg/mod/github.com/spf13/cobra@v1.4.0/command.go:902\ngithub.com/flyteorg/datacatalog/cmd/entrypoints.Execute()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/entrypoints/root.go:46> +0x25\nmain.main()\n\t/go/src/github.com/flyteorg/datacatalog/cmd/main.go:10 +0x6b\n]","ts":"2023-01-23T12:56:22Z"}

Expected behavior The deployment should be up and running as other deployments with similar requirement are up with same account. I ensured that datacatalog Serviceaccount has binding to corresponding GSA similar to other deployments. Observation: I see for other deployments corresponding cluster-role and cluster-role-bindings are created, but its missing for datacatalog, isn't it required for datacatalog as well ?

flyte-flyte-pod-webhook    flyte-pod-webhook
flyte-flyteadmin-binding   flyteadmin
flyte-flytepropeller       flytepropeller

Additional context to reproduce Following GCP manual Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte