hi all - i am trying to configure the logging link in my newly deployed EKS single cluster setup. We...

millions-table-8574

04/20/2023, 8:15 AM

hi all - i am trying to configure the logging link in my newly deployed EKS single cluster setup. We already have fluentbit running in EKS that’s sending application logs to CloudWatch whose stream name contains the

nodeName

the pod is currently running on. based on the docs here and the code here, i try to access

{{.hostname}}

in the logging configuration, but it’s not populating the template correctly: it’s missing the node name still

millions-table-8574

04/20/2023, 8:17 AM

here is the config:

Copy code

logging:
    level: 1
    plugins:
      cloudwatch:
        enabled: true
        templateUri: |-
          <https://console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#logEventViewer:group=/aws/containerinsights/ncai-dev/application;stream={{.hostname}}-application.var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log>

Note

{{.hostname}}

right before

-application.var.log.containers.xxxx

millions-table-8574

04/20/2023, 8:18 AM

here’s the redirected CW link. it’s missing the node name

millions-table-8574

04/20/2023, 8:19 AM

am i missing something?

millions-table-8574

04/20/2023, 9:20 AM

https://github.com/flyteorg/flyteplugins/blob/b0684d97a1cf240f1a44f310f4a79cc21844caa9/go/tasks/pluginmachinery/tasklog/template.go#L72

getTaskLog

doesn’t take in a

hostname

, and even the tests don’t test for

hostname

freezing-airport-6809

04/20/2023, 1:54 PM

Hmm this works, let me ask some expert. Cc @many-wire-75890 have you seen this

millions-table-8574

04/21/2023, 4:01 PM

hi @many-wire-75890 - any ideas? i suspect it’s this, but not too sure how the plugin is being used

millions-table-8574

04/27/2023, 4:19 AM

found this issue has been pending since last year: https://github.com/flyteorg/flyteplugins/pull/293

freezing-airport-6809

04/27/2023, 4:26 AM

sorry James has been out

white-painting-22485

04/27/2023, 4:57 AM

Hey @millions-table-8574 👋. Yes as you stumbled on it looks like hostname isn't plumbed through to the template logic today. That said, do you have a hard dependency on hostname in the log stream name? Looks like the default uses

...stream=var.log.containers.{{ .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log

, which is why I suspect most other users haven't encountered this

millions-table-8574

04/27/2023, 8:26 AM

hi @white-painting-22485 thanks for getting in touch. we don’t necessarily have a hard dependency, our fluentbit logs to a CW stream using the default log name, which has the node name in it. yet, it seems like a suboptimal way of going about addressing this by changing the fluentbit log config, rather than fixing/exposing the nodename in the flyte plugin

millions-table-8574

04/27/2023, 8:28 AM

based on the comments in the PR, i’m also unsure if

hostname

is the correct template param, or if the new change requires

pod.Spec.NodeName

that @EngHabu was referncing

white-painting-22485

04/27/2023, 1:32 PM

Definitely agree on finding a way to solve this in Flyte - just speculating on why this hasn't bubbled up to be a more common problem Have you tried using

logStreamNameFilter

which can match a known subset of the stream (and ignore hostname)? While some installs may have hostname prefix by default, I recently used this chart which has a

fluentbit-

prefix. Using

logStreamNameFilter

may be the safest bet for Flyte's default plugin config.

high-park-82026

04/27/2023, 6:38 PM

cc @freezing-boots-56761

freezing-boots-56761

04/27/2023, 7:39 PM

i’m not sure that this is a flyte thing. this sounds more like an implementation detail of the infrastructure. namely the fluentbit config. there are options for dealing with log stream names with node names. you won’t be able to link directly to the log stream, but you can use a template uri that presents a filtered list of log streams that match the pod name - what @white-painting-22485 said

freezing-boots-56761

04/27/2023, 7:43 PM

@millions-table-8574 i might be able to help craft a template uri that works if that’ll help

freezing-boots-56761

04/27/2023, 8:00 PM

can you try:

Copy code

logging:
    level: 1
    plugins:
      cloudwatch:
        enabled: true
        templateUri: |-
          <https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#logsV2:log-groups/log-group/$252Faws$252Fcontainerinsights$252Fncai-dev$252Fapplication$3FlogStreamNameFilter$3Dapplication.var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log>

👍 1

millions-table-8574

05/10/2023, 10:14 AM

thanks @freezing-boots-56761 @white-painting-22485 - we went with the template uri you proposed and it’s good enough for us. much appreciated 🙏

👍 3

334 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.