#3688 [BUG] Wrong kubernetes client is used in plugin manager Issue created by sonjaer Describe the bug We are working on enabling the ray plugin in Propeller running in our cluster A with the

remoteClusterConfig

connecting to a remote Ray cluster B. During Propeller startup, we hit a problem:

ray: [PluginInitializationFailed] Error getting informer for %!s(\u003cnil\u003e), caused by: no matches for kind \"RayJob\" in version \"<http://ray.io/v1alpha1\|ray.io/v1alpha1\>""

Installing the ray crd in cluster A solved this problem. However, we would expect it to be enough for it to just be installed in cluster B. After fixing that, we also hit another issue during startup where a permission is needed for Propeller service account to list rayjobs in cluster A, where this permission should only be needed in cluster B.

W0516 13:51:19.779009       1 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: failed to list *v1alpha1.RayJob: <http://rayjobs.ray.io|rayjobs.ray.io> is forbidden: User "<service-account>" cannot list resource "rayjobs" in API group "<http://ray.io|ray.io>" at the cluster scope
E0516 13:51:19.779088       1 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1alpha1.RayJob: failed to list *v1alpha1.RayJob: <http://rayjobs.ray.io|rayjobs.ray.io> is forbidden: User "<service-account>" cannot list resource "rayjobs" in API group "<http://ray.io|ray.io>" at the cluster scope

The problem seem to be that the plugin manager doesn't use the correct k8 client (e.g., the k8 client in the plugin). Expected behavior The plugin manager should use the custom kubernetes client from the plugin if it exists. However at many places it doesn't, e.g.,: 1.https://github.com/flyteorg/flytepropeller/blob/9a4ea000af6bb7b959daa00f26abea7c2e3262e7/pkg/controller/nodes/task/k8s/plugin_manager.go#L654 2.https://github.com/flyteorg/flytepropeller/blob/9a4ea000af6bb7b959daa00f26abea7c2e3262e7/pkg/controller/nodes/task/k8s/plugin_manager.go#L546 Additional context to reproduce No response Screenshots No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte