Team, i am trying to update the spark logging link with below template uri.

- spark.hadoop.fs.AbstractFileSystem.s3a.impl: "org.apache.hadoop.fs.s3a.S3A"
          - spark.hadoop.fs.s3a.multipart.threshold: "536870912"
          - spark.blacklist.enabled: "true"
          - spark.blacklist.timeout: "5m"
          - spark.task.maxfailures: "8"
        logs:
          mixed:
            kubernetes-enabled: true
            kubernetes-template-uri: '<https://app.datadoghq.com/logs?query=kube_namespace:{{> .namespace }} pod_name:{{ .podName }}&cols=&index=&messageDisplay=inline&stream_sort=time,desc&viz=stream&from_ts=1663061891694&to_ts=1663148291694&live=true'
          system:
            kubernetes-enabled: true
            kubernetes-template-uri: '<https://app.datadoghq.com/logs?query=kube_namespace:{{> .namespace }} pod_name:{{ .podName }}&cols=&index=&messageDisplay=inline&stream_sort=time,desc&viz=stream&from_ts=1663061891694&to_ts=1663148291694&live=true'
          all-user:
            kubernetes-enabled: true
            kubernetes-template-uri: '<https://app.datadoghq.com/logs?query=kube_namespace:{{> .namespace }} pod_name:{{ .podName }}&cols=&index=&messageDisplay=inline&stream_sort=time,desc&viz=stream&from_ts=1663061891694&to_ts=1663148291694&live=true'

however flytepropeller pod is not getting created and throwing below error. time="2022-09-14T112612Z" level=error msg="\nWhile parsing config: yaml: line 6: mapping values are not allowed in this context"

is there any more to the error message btw?

@mammoth-lion-71167 I tested locally with your configuration and this seems to work fine. The 'mapping values are not allowed in this context' means that the configuration is looking for a value rather than a map in the yaml. Could you provide the entire configuration? That way we can figure out exactly what the issue is.

@hallowed-mouse-14616 sorry for the delay in my response. please find the helm template for the flyte deployment we are using which has link to datadog.

cc @great-school-54368

Were you able to verify our yaml file for datadog links?

@mammoth-lion-71167 what is the generated configmap for flytepropeller?

@hallowed-mouse-14616 i see {{ .namespace }} pod_name:{{ .podName }} is getting empty value

spark.yaml: |
    plugins:
      spark:
        logs:
          all-user:
            cloudwatch-enabled: true
            cloudwatch-template-uri: <https://app.datadoghq.com/logs?query=kube_namespace>: pod_name:
          mixed:
            cloudwatch-enabled: true
            cloudwatch-template-uri: <https://app.datadoghq.com/logs?query=kube_namespace>: pod_name:
          system:
            cloudwatch-enabled: true
            cloudwatch-template-uri: <https://app.datadoghq.com/logs?query=kube_namespace>: pod_name:
        spark-config-default:
        - spark.hadoop.fs.s3a.aws.credentials.provider: com.amazonaws.auth.DefaultAWSCredentialsProviderChain
        - spark.hadoop.mapreduce.fileoutputcommitter.algorithm.version: "2"
        - spark.kubernetes.allocation.batch.size: "50"

@mammoth-lion-71167 ok, we have seen this before, basically the helm chart uses the same templating for args (ie. {{ and }}) so when helm compiles the chart it replaces the Flyte log link templates as well. we should be able to fix this in two steps: (1) use escaped templating for flyte logs in the helm charts. this will correct the

.namespace

,

.podName

replacements your seeing. @billowy-sundown-31926 did we get this documented? (2) currently you have a space in the URI between the

kube_namespace:

and

pod_name:

in yaml this is manifesting as a map rather than a value. i'm not sure if this can be quoted or if the space needs to be completely removed.

@hallowed-mouse-14616 this will be documented in a while! Sorry for the delay

no problem, can you link to the conversation, or just put a brief solution, regarding the fix here?

https://flyte-org.slack.com/archives/CP2HDHKE1/p1661450232630339 - here is the conversation

thank you all. i will try.