Hey all, I think this is a deployment related issue I've been facing. I don't have the ability to view the logs of any nodes in a workflow. the aws cloudwatch group shows this and the config map shows the

log

plugin is defined as well

task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-enabled: true
        cloudwatch-log-group: 'bv-ml-pipelines'
        cloudwatch-region: 'us-east-1'
        kubernetes-enabled: false

hey - sorry for the delay. we will look into this tomorrow

Hey @bumpy-morning-40916 apologies for the delay! How are you pushing logs to cloudwatch? FluentD Bit? You will need to install and configure that following AWS docs. E.g. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html And then you will need to modify the fluentd-bit to format the log stream name… By default, flyte’s cloudwatch log plugin assumes the log stream to be named

<https://console.aws.amazon.com/cloudwatch/home?region=%s#logEventViewer:group=%s;stream=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log

You can, alternatively, use

cloudwatch-template-uri

to specify any other log stream url you configured…

task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
        kubernetes-enabled: false

I just filed this issue to change the default template to work with FluentD OOB. If you also want array tasks to use the same config, you will, at least as of now, configure them separately like this:

task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
        kubernetes-enabled: false
      k8s-array:
        logs:
          config:
            cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
            kubernetes-enabled: false

@echoing-queen-96254 i don’t think what haytham wrote is in our documentation - mind finding a home for it?

Yes, using FluentD setup through Container-Insights-setup-EKS-quickstart.html I see logs in

/aws/containerinsights/ml-pipelines/*

but with the updated config map matching

task_logs.yaml: |
    plugins:
      logs:
        cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
        kubernetes-enabled: false
      k8s-array:
        logs:
          config:
            cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
            kubernetes-enabled: false

FlyteConsole is showing no logs... can this be a permissions issue? Either flyte-user/system role not having the permissions?

at the same level as

cloudwatch-template-uri

, could you set

cloudwatch-enabled

to

true

as well?

Yes that ^ I missed that… thanks Yee