Hey all, I think this is a deployment related issu...
# flyte-deployment
e
Hey all, I think this is a deployment related issue I've been facing. I don't have the ability to view the logs of any nodes in a workflow. the aws cloudwatch group shows this and the config map shows the
log
plugin is defined as well
Copy code
task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-enabled: true
        cloudwatch-log-group: 'bv-ml-pipelines'
        cloudwatch-region: 'us-east-1'
        kubernetes-enabled: false
y
hey - sorry for the delay. we will look into this tomorrow
h
Hey @Edgar Trujillo apologies for the delay! How are you pushing logs to cloudwatch? FluentD Bit? You will need to install and configure that following AWS docs. E.g. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html And then you will need to modify the fluentd-bit to format the log stream name… By default, flyte’s cloudwatch log plugin assumes the log stream to be named
Copy code
<https://console.aws.amazon.com/cloudwatch/home?region=%s#logEventViewer:group=%s;stream=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log
You can, alternatively, use
cloudwatch-template-uri
to specify any other log stream url you configured…
Copy code
task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
        kubernetes-enabled: false
I just filed this issue to change the default template to work with FluentD OOB. If you also want array tasks to use the same config, you will, at least as of now, configure them separately like this:
Copy code
task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
        kubernetes-enabled: false
      k8s-array:
        logs:
          config:
            cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
            kubernetes-enabled: false
cc @Yuvraj @Brian Connolly
y
@Marc Paquette i don’t think what haytham wrote is in our documentation - mind finding a home for it?
👍 1
👀 1
e
Yes, using FluentD setup through Container-Insights-setup-EKS-quickstart.html I see logs in
/aws/containerinsights/ml-pipelines/*
but with the updated config map matching
Copy code
task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
        kubernetes-enabled: false
      k8s-array:
        logs:
          config:
            cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
            kubernetes-enabled: false
FlyteConsole is showing no logs... can this be a permissions issue? Either flyte-user/system role not having the permissions?
I also updated
fluent-bit-config
to:
log_stream_prefix   var.
as the default was
log_stream_prefix   ${HOST_NAME}-
y
at the same level as
cloudwatch-template-uri
, could you set
cloudwatch-enabled
to
true
as well?
that is the only bit I’m seeing that might prevent that from showing up.
h
Yes that ^ I missed that… thanks Yee
113 Views