Trying to debug some odd behavior. I have two separate environments with the same propeller configmap deployed, but the Console UI is showing different logging links. One is showing what I configured in the config map, but the other is showing the default settings. I pulled the configmaps using

kubectl

but they're the same. Is this correct configuration for updating log links in propeller? Do I need to restart anything other than propeller to update the log links?

task_logs.yaml: | 
    plugins:
      logs:
        cloudwatch-enabled: false
        kubernetes-enabled: false
        templates:
          - displayName: "Data Dog"
            templateUris:
              - "<https://app.datadoghq.com/logs?query=pod_name%3A{{> .podName }}%20kube_namespace%3A{{ .namespace }}%20&cols=&index=&messageDisplay=inline&stream_sort=desc&viz=stream&from_ts={{ .podUnixStartTime }}000&to_ts={{ .podUnixFinishTime }}000&live=false"

Are you looking back at the log links for tasks in old executions or starting a new workflow execution to see the results of this config? I think the log link template is evaluated at some point during a tasks execution and the resulting string is stored, so you won't see the results of the current config until a new task executes.

I'm starting new ones to test

In that case, I don't know. I thought restarting propeller would be sufficient.

Does anything else impact these log llinks, like log level? Although I haven't changed those between the two environments from the default

@limited-dog-47035 this is very odd. You are correct, updating the configmap and the restarting FlytePropeller - and you shouldn't actually need to restart propeller anyways it should automatically detect the updates and apply them.

Ah, disregard my mention of the log level; the log level config I was looking at is for admin/datacatalog so it shouldn't impact that

So the spark plugin seems to have it's own separate log configuration.

I see. I was reading the documentation

Spark Plugin uses the same default log configuration as explained in Configuring Logging Links in UI.

to mean that it used the task_logs.yaml config

🤦 we need to fix that. i would have thought the same thing.

Yeah I read that section mentioning separating the user logs and system logs as optional; but that if not configured it would use the default

oh sure, looking through the code it looks like you get the logs that are configured. by default these are mixed k8s logs. so you should be able to modify that configuration and get the logs how you want them. does that seem reasonable?

Yeah, okay, good. I think the documentation is fine then