Hi, I am trying to enable the dask plugin in my flyte deployment following the documentation and get the following error (in the flyte pod logs):

W1222 10:08:49.441894       7 reflector.go:324] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: failed to list *v1.DaskJob: daskjobs.kubernetes.dask.org is forbidden: User "system:serviceaccount:flyte:flyte-backend-flyte-binary" cannot list resource "daskjobs" in API group "kubernetes.dask.org" at the cluster scope                                                                                                   E1222 10:08:49.441919       7 reflector.go:138] pkg/mod/k8s.io/client-go@v0.24.1/tools/cache/reflector.go:167: Failed to watch *v1.DaskJob: failed to list *v1.DaskJob: daskjobs.kubernetes.dask.org is forbidden: User "system:serviceaccount:flyte:flyte-backend-flyte-binary" cannot list resource "daskjobs" in API group "kubernetes.dask.org" at the cluster scope

Steps: Install dask operator with the helm chart (chart:dask-kubernetes-operator-2023.10.0, app-version:2022.4.1), update values, deploy flyte-binary helm chart (chartflyte binary v1.9.0,app version1.16.0) Does anyone have an idea what could be the problem? Thanks

I do ✋ Seems like your flyte service account does not have the permissions to look at dask’s custom resources. You’d probably wanna have a

ClusterRole

which you can then bind your used service account. The Cluster role should provide access to those resource, so something along the lines of:

apiVersion: <http://rbac.authorization.k8s.io/v1|rbac.authorization.k8s.io/v1>
kind: ClusterRole
metadata:
  name: dask-dask-kubernetes-operator-role-cluster
  labels:
    <http://app.kubernetes.io/managed-by|app.kubernetes.io/managed-by>: Helm
  annotations:
    <http://meta.helm.sh/release-name|meta.helm.sh/release-name>: dask
    <http://meta.helm.sh/release-namespace|meta.helm.sh/release-namespace>: dask
rules:
  - verbs:
      - list
      - watch
    apiGroups:
      - <http://apiextensions.k8s.io|apiextensions.k8s.io>
    resources:
      - customresourcedefinitions
  - verbs:
      - get
      - list
      - watch
      - patch
      - create
      - delete
    apiGroups:
      - <http://kubernetes.dask.org|kubernetes.dask.org>
    resources:
      - daskclusters
      - daskworkergroups
      - daskjobs
      - daskjobs/status
      - daskautoscalers
      - daskworkergroups/scale

YMMV with which permissions you need

Works now, thanks a lot! 🙂