Sathish kumar Venkatesan
09/14/2022, 11:32 AM- spark.hadoop.fs.AbstractFileSystem.s3a.impl: "org.apache.hadoop.fs.s3a.S3A"
- spark.hadoop.fs.s3a.multipart.threshold: "536870912"
- spark.blacklist.enabled: "true"
- spark.blacklist.timeout: "5m"
- spark.task.maxfailures: "8"
logs:
mixed:
kubernetes-enabled: true
kubernetes-template-uri: '<https://app.datadoghq.com/logs?query=kube_namespace:{{> .namespace }} pod_name:{{ .podName }}&cols=&index=&messageDisplay=inline&stream_sort=time,desc&viz=stream&from_ts=1663061891694&to_ts=1663148291694&live=true'
system:
kubernetes-enabled: true
kubernetes-template-uri: '<https://app.datadoghq.com/logs?query=kube_namespace:{{> .namespace }} pod_name:{{ .podName }}&cols=&index=&messageDisplay=inline&stream_sort=time,desc&viz=stream&from_ts=1663061891694&to_ts=1663148291694&live=true'
all-user:
kubernetes-enabled: true
kubernetes-template-uri: '<https://app.datadoghq.com/logs?query=kube_namespace:{{> .namespace }} pod_name:{{ .podName }}&cols=&index=&messageDisplay=inline&stream_sort=time,desc&viz=stream&from_ts=1663061891694&to_ts=1663148291694&live=true'
however flytepropeller pod is not getting created and throwing below error.
time="2022-09-14T112612Z" level=error msg="\nWhile parsing config: yaml: line 6: mapping values are not allowed in this context"Yee
Dan Rammer (hamersaw)
09/15/2022, 2:58 PMSathish kumar Venkatesan
09/22/2022, 6:29 AMDan Rammer (hamersaw)
09/23/2022, 11:31 AMSathish kumar Venkatesan
09/29/2022, 9:24 AMspark.yaml: |
plugins:
spark:
logs:
all-user:
cloudwatch-enabled: true
cloudwatch-template-uri: <https://app.datadoghq.com/logs?query=kube_namespace>: pod_name:
mixed:
cloudwatch-enabled: true
cloudwatch-template-uri: <https://app.datadoghq.com/logs?query=kube_namespace>: pod_name:
system:
cloudwatch-enabled: true
cloudwatch-template-uri: <https://app.datadoghq.com/logs?query=kube_namespace>: pod_name:
spark-config-default:
- spark.hadoop.fs.s3a.aws.credentials.provider: com.amazonaws.auth.DefaultAWSCredentialsProviderChain
- spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: "2"
- spark.kubernetes.allocation.batch.size: "50"
Dan Rammer (hamersaw)
09/29/2022, 3:18 PM.namespace
, .podName
replacements your seeing. @Smriti Satyan did we get this documented?
(2) currently you have a space in the URI between the kube_namespace:
and pod_name:
in yaml this is manifesting as a map rather than a value. i'm not sure if this can be quoted or if the space needs to be completely removed.Smriti Satyan
09/29/2022, 3:21 PMDan Rammer (hamersaw)
09/29/2022, 3:22 PMSmriti Satyan
09/29/2022, 3:22 PMSathish kumar Venkatesan
09/29/2022, 3:28 PM