Having some difficulty using the Spark plugin for Flyte; in particular the guide specifies to add templates to the cluster default templates, however the SA used by flyte-binary does not have the necessary privileges to create those resources (ClusterRole, Service Account, CRB). What is the best way to deal with this without giving the flyte SA privileges it shouldn’t have?

Looks like the spark plugin setup instructions for flyte binary helm chart are missing. We'll work on updating the docs. Here's an example of how you need to specify the relevant permissions:

clusterResourceTemplates:
  inline:
    001_namespace.yaml: |
      apiVersion: v1
      kind: Namespace
      metadata:
        name: '{{ namespace }}'
    002_serviceaccount.yaml: |
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: default
        namespace: '{{ namespace }}'
        annotations:
          eks.amazonaws.com/role-arn: '{{ defaultIamRole }}'
    010_spark_role.yaml: |
      apiVersion: rbac.authorization.k8s.io/v1
      kind: Role
      metadata:
        name: spark-role
        namespace: '{{ namespace }}'
      rules:
      - apiGroups:
        - ""
        resources:
        - pods
        - services
        - configmaps
        verbs:
        - '*'
    011_spark_service_account.yaml: |
      apiVersion: v1
      kind: ServiceAccount
      metadata:
        name: spark
        namespace: '{{ namespace }}'
        annotations:
          eks.amazonaws.com/role-arn: '{{ defaultIamRole }}'
    012_spark_role_binding.yaml: |
      apiVersion: rbac.authorization.k8s.io/v1
      kind: RoleBinding
      metadata:
        name: spark-role-binding
        namespace: '{{ namespace }}'
      roleRef:
        apiGroup: rbac.authorization.k8s.io
        kind: Role
        name: spark-role
      subjects:
      - kind: ServiceAccount
        name: spark
        namespace: '{{ namespace }}'

You also need to add spark to

task-plugins

and under the

plugins

section as well:

spark:
        spark-config-default:
        - spark.driver.cores: "1"
        - spark.executorEnv.HTTP2_DISABLE: "true"
        - spark.hadoop.fs.AbstractFileSystem.s3.impl: "org.apache.hadoop.fs.s3a.S3A"
        - spark.hadoop.fs.AbstractFileSystem.s3a.impl: "org.apache.hadoop.fs.s3a.S3A"
        - spark.hadoop.fs.AbstractFileSystem.s3n.impl: "org.apache.hadoop.fs.s3a.S3A"
        - spark.hadoop.fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
        - spark.hadoop.fs.s3a.acl.default: "BucketOwnerFullControl"
        - spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
        - spark.hadoop.fs.s3n.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
        - spark.kubernetes.allocation.batch.size: "50"
        - spark.kubernetes.driverEnv.HTTP2_DISABLE: "true"
        - spark.network.timeout: 600s
        - spark.executorEnv.KUBERNETES_REQUEST_TIMEOUT: 100000
        - spark.executor.heartbeatInterval: 60s
        - spark.hadoop.fs.s3a.aws.credentials.provider: com.amazonaws.auth.WebIdentityTokenCredentialsProvider

Let me know if anything's still unclear to you.

@tall-lock-23197 those templates are resolved when a new “cluster”/project/namespace is created right? My issue is that the Flyte SA does not have the privileges necessary to create the role/binding resources for the new namespace. Should the Flyte SA be able to grant privileges to other SAs? I’ve temporarily gotten around this by deploying those resources listed above (Spark SA, Role, and RoleBinding) to the relevant namespace manually; however I still receive this error:

ERROR Client: Please check "kubectl auth can-i create pod" first. It should be yes.

The Pod created by Flyte to run that workflow is definitely running with the

spark

SA

> those templates are resolved when a new “cluster”/project/namespace is created right? The templates should actually be applied to the namespace where you use the spark service account (execution namespace to be precise). So if you're running the spark workflow in the flytesnacks-development namespace, the templates should be deployed in there, and that has to happen automatically.

Hm. I’ll have to give this a try again another time. For now I have just resolved to running spark locally on a really big node – i.e. without

SparkTask

– which is sufficient for my purposes for the moment.