• Brian Tang

    Brian Tang

    2 months ago
    Cross posting from https://flyte-org.slack.com/archives/C01P3B761A6/p1654537014728059 to see if anybody here might know πŸ™‚ . I tried restarting the
    flytepropeller
    deployment and it’s hard to tell if the system is picking up the changes:
    $ kubectl logs -f deployment/flytepropeller -nflyte | grep -i logs
    Found 6 pods, using pod/flytepropeller-785bcc6f6d-sr8bl
    {"json":{"src":"viper.go:398"},"level":"debug","msg":"Config section [plugins.logs] updated. No update handler registered.","ts":"2022-06-08T16:32:15Z"}
    {"json":{"src":"viper.go:396"},"level":"debug","msg":"Config section [plugins.logs] hasn't changed.","ts":"2022-06-10T22:36:33Z"}
    From looking at the plugins β€” it seems like adding that config should just automatically work
  • Robin Kahlow

    Robin Kahlow

    2 months ago
    I see you took the yaml from https://docs.flyte.org/projects/cookbook/en/stable/auto/deployment/configure_logging_links.html, I haven't seen this before, mine looks like this
    task_logs.yaml: |
        plugins:
          logs:
            cloudwatch-enabled: false
            kubernetes-enabled: true
            kubernetes-template-uri: <https://my.papertrailapp.com/systems/{{> .podName }}/events
  • Brian Tang

    Brian Tang

    2 months ago
    ah β€” so you just stuck a custom template uri in the kubernetes property. lemme try that
  • 😞 still no joy:
    $ kubectl get configmaps -n flyte flyte-propeller-config -oyaml
    apiVersion: v1
    data:
    ...
      task_logs.yaml: |
        plugins:
          logs:
            cloudwatch-enabled: false
            kubernetes-enabled: true
            kubernetes-template-uri: "<https://splunk.corp.stripe.com/en-US/app/search/search?q=search%20source=>"/pay/log/pods/{{.namespace }}_{{.podName }}*"%20%7C%20reverse&display.events.type=raw&display.prefs.events.count=50&earliest=-7d&latest=now"
    ...
  • Robin Kahlow

    Robin Kahlow

    2 months ago
    Maybe because eg. the second double-quote isnt escaped? Dunno how that works in yaml
  • Brian Tang

    Brian Tang

    2 months ago
    :lolcry: ah β€” I realized that for some reason the
    messageFormat: "json"
    line was causing the pods to crash loop. Removing that and fixing the quotes seems to fix it. Thanks for the look over!