salmon-refrigerator-32115
11/07/2023, 5:21 PMfreezing-airport-6809
salmon-refrigerator-32115
11/07/2023, 5:50 PMfreezing-airport-6809
freezing-airport-6809
freezing-airport-6809
glamorous-carpet-83516
11/07/2023, 8:59 PMglamorous-carpet-83516
11/07/2023, 9:40 PMsalmon-refrigerator-32115
11/07/2023, 9:48 PMdamp-lion-88352
11/08/2023, 2:51 AMdamp-lion-88352
11/08/2023, 2:51 AMdamp-lion-88352
11/08/2023, 2:52 AMdamp-lion-88352
11/08/2023, 2:52 AMsalmon-refrigerator-32115
11/08/2023, 2:54 AMdamp-lion-88352
11/08/2023, 2:55 AMdamp-lion-88352
11/08/2023, 2:55 AMdamp-lion-88352
11/08/2023, 2:55 AMdamp-lion-88352
11/08/2023, 2:56 AMsalmon-refrigerator-32115
11/08/2023, 8:04 PMcluster_resource_manager: <- Is this needed?
enabled: true
config:
cluster_resources:
refreshInterval: 5m
templatePath: "/etc/flyte/clusterresource/templates"
customData:
- production:
- projectQuotaCpu:
value: "5"
- projectQuotaMemory:
value: "4000Mi"
- staging:
- projectQuotaCpu:
value: "2"
- projectQuotaMemory:
value: "3000Mi"
- development:
- projectQuotaCpu:
value: "4"
- projectQuotaMemory:
value: "3000Mi"
refresh: 5m
# -- Resource templates that should be applied
templates:
# -- Template for namespaces resources
- key: aa_namespace
value: |
apiVersion: v1
kind: Namespace
metadata:
name: {{ namespace }}
spec:
finalizers:
- kubernetes
- key: ab_project_resource_quota
value: |
apiVersion: v1
kind: ResourceQuota
metadata:
name: project-quota
namespace: {{ namespace }}
spec:
hard:
limits.cpu: {{ projectQuotaCpu }}
limits.memory: {{ projectQuotaMemory }}
- key: ac_spark_role <- Is this needed?
value: |
apiVersion: <http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>
kind: Role
metadata:
name: spark-role
namespace: {{ namespace }}
rules:
- apiGroups: ["*"]
resources:
- pods
verbs:
- '*'
- apiGroups: ["*"]
resources:
- services
verbs:
- '*'
- apiGroups: ["*"]
resources:
- configmaps
verbs:
- '*'
- key: ad_spark_service_account <- Is this needed?
value: |
apiVersion: v1
kind: ServiceAccount
metadata:
name: spark
namespace: {{ namespace }}
- key: ae_spark_role_binding <- Is this needed?
value: |
apiVersion: <http://rbac.authorization.k8s.io/v1beta1|rbac.authorization.k8s.io/v1beta1>
kind: RoleBinding
metadata:
name: spark-role-binding
namespace: {{ namespace }}
roleRef:
apiGroup: <http://rbac.authorization.k8s.io|rbac.authorization.k8s.io>
kind: Role
name: spark-role
subjects:
- kind: ServiceAccount
name: spark
namespace: {{ namespace }}
sparkoperator: <- Is this needed?
enabled: true
plugin_config:
plugins:
spark:
# Edit the Spark configuration as you see fit
spark-config-default:
- spark.driver.cores: "1"
- spark.hadoop.fs.s3a.aws.credentials.provider: "com.amazonaws.auth.DefaultAWSCredentialsProviderChain"
- spark.kubernetes.allocation.batch.size: "50"
- spark.hadoop.fs.s3a.acl.default: "BucketOwnerFullControl"
- spark.hadoop.fs.s3n.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
- spark.hadoop.fs.AbstractFileSystem.s3n.impl: "org.apache.hadoop.fs.s3a.S3A"
- spark.hadoop.fs.s3.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
- spark.hadoop.fs.AbstractFileSystem.s3.impl: "org.apache.hadoop.fs.s3a.S3A"
- spark.hadoop.fs.s3a.impl: "org.apache.hadoop.fs.s3a.S3AFileSystem"
- spark.hadoop.fs.AbstractFileSystem.s3a.impl: "org.apache.hadoop.fs.s3a.S3A"
- spark.network.timeout: 600s
- spark.executorEnv.KUBERNETES_REQUEST_TIMEOUT: 100000
- spark.executor.heartbeatInterval: 60s
Also what is the equivalent databricks plugin configuration like (in the helm chart)? Could you point me to an example?
3) How is the databricks spark job logging integrated with Flyte UI?
4) When running a flyte spark task on the server, is the --service-account spark option still required?salmon-refrigerator-32115
12/11/2023, 7:13 PMglamorous-carpet-83516
12/11/2023, 7:14 PMglamorous-carpet-83516
12/11/2023, 7:14 PMglamorous-carpet-83516
12/11/2023, 7:14 PMsalmon-refrigerator-32115
12/11/2023, 7:15 PMglamorous-carpet-83516
12/11/2023, 7:17 PMglamorous-carpet-83516
12/11/2023, 7:17 PMsalmon-refrigerator-32115
12/11/2023, 7:20 PMglamorous-carpet-83516
12/11/2023, 7:21 PMsalmon-refrigerator-32115
12/11/2023, 7:21 PMsalmon-refrigerator-32115
12/11/2023, 7:21 PMglamorous-carpet-83516
12/11/2023, 7:22 PMsalmon-refrigerator-32115
12/11/2023, 7:50 PMglamorous-carpet-83516
12/11/2023, 7:53 PM