https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
b

Brian Tang

04/20/2023, 8:15 AM
hi all - i am trying to configure the logging link in my newly deployed EKS single cluster setup. We already have fluentbit running in EKS that’s sending application logs to CloudWatch whose stream name contains the 
nodeName
the pod is currently running on. based on the docs here and the code here, i try to access 
{{.hostname}}
in the logging configuration, but it’s not populating the template correctly: it’s missing the node name still
here is the config:
Copy code
logging:
    level: 1
    plugins:
      cloudwatch:
        enabled: true
        templateUri: |-
          <https://console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#logEventViewer:group=/aws/containerinsights/ncai-dev/application;stream={{.hostname}}-application.var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log>
Note 
{{.hostname}}
right before 
-application.var.log.containers.xxxx
here’s the redirected CW link. it’s missing the node name
am i missing something?
k

Ketan (kumare3)

04/20/2023, 1:54 PM
Hmm this works, let me ask some expert. Cc @James Sutton have you seen this
b

Brian Tang

04/21/2023, 4:01 PM
hi @James Sutton - any ideas? i suspect it’s this, but not too sure how the plugin is being used
found this issue has been pending since last year: https://github.com/flyteorg/flyteplugins/pull/293
k

Ketan (kumare3)

04/27/2023, 4:26 AM
sorry James has been out
a

Andrew Dye

04/27/2023, 4:57 AM
Hey @Brian Tang 👋. Yes as you stumbled on it looks like hostname isn't plumbed through to the template logic today. That said, do you have a hard dependency on hostname in the log stream name? Looks like the default uses 
...stream=var.log.containers.{{ .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log
, which is why I suspect most other users haven't encountered this
b

Brian Tang

04/27/2023, 8:26 AM
hi @Andrew Dye thanks for getting in touch. we don’t necessarily have a hard dependency, our fluentbit logs to a CW stream using the default log name, which has the node name in it. yet, it seems like a suboptimal way of going about addressing this by changing the fluentbit log config, rather than fixing/exposing the nodename in the flyte plugin
based on the comments in the PR, i’m also unsure if 
hostname
is the correct template param, or if the new change requires 
pod.Spec.NodeName
that @EngHabu was referncing
a

Andrew Dye

04/27/2023, 1:32 PM
Definitely agree on finding a way to solve this in Flyte - just speculating on why this hasn't bubbled up to be a more common problem Have you tried using 
logStreamNameFilter
which can match a known subset of the stream (and ignore hostname)? While some installs may have hostname prefix by default, I recently used this chart which has a 
fluentbit-
prefix. Using 
logStreamNameFilter
may be the safest bet for Flyte's default plugin config.
h

Haytham Abuelfutuh

04/27/2023, 6:38 PM
cc @jeev
j

jeev

04/27/2023, 7:39 PM
i’m not sure that this is a flyte thing. this sounds more like an implementation detail of the infrastructure. namely the fluentbit config. there are options for dealing with log stream names with node names. you won’t be able to link directly to the log stream, but you can use a template uri that presents a filtered list of log streams that match the pod name - what @Andrew Dye said
@Brian Tang i might be able to help craft a template uri that works if that’ll help
can you try:
Copy code
logging:
    level: 1
    plugins:
      cloudwatch:
        enabled: true
        templateUri: |-
          <https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home?region=ap-southeast-1#logsV2:log-groups/log-group/$252Faws$252Fcontainerinsights$252Fncai-dev$252Fapplication$3FlogStreamNameFilter$3Dapplication.var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log>
b

Brian Tang

05/10/2023, 10:14 AM
thanks @jeev @Andrew Dye - we went with the template uri you proposed and it’s good enough for us. much appreciated 🙏
19 Views
Powered by