Thread
#flyte-deployment
    e

    Edgar Trujillo

    3 months ago
    Hey all, I think this is a deployment related issue I've been facing. I don't have the ability to view the logs of any nodes in a workflow. the aws cloudwatch group shows this and the config map shows the
    log
    plugin is defined as well
    task_logs.yaml: |
        plugins:
          logs:
            cloudwatch-enabled: true
            cloudwatch-log-group: 'bv-ml-pipelines'
            cloudwatch-region: 'us-east-1'
            kubernetes-enabled: false
    Yee

    Yee

    3 months ago
    hey - sorry for the delay. we will look into this tomorrow
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    2 months ago
    Hey @Edgar Trujillo apologies for the delay! How are you pushing logs to cloudwatch? FluentD Bit? You will need to install and configure that following AWS docs. E.g. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-setup-logs-FluentBit.html And then you will need to modify the fluentd-bit to format the log stream name… By default, flyte’s cloudwatch log plugin assumes the log stream to be named
    <https://console.aws.amazon.com/cloudwatch/home?region=%s#logEventViewer:group=%s;stream=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log
    You can, alternatively, use
    cloudwatch-template-uri
    to specify any other log stream url you configured…
    task_logs.yaml: |
        plugins:
          logs:
            cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
            kubernetes-enabled: false
    I just filed this issue to change the default template to work with FluentD OOB. If you also want array tasks to use the same config, you will, at least as of now, configure them separately like this:
    task_logs.yaml: |
        plugins:
          logs:
            cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
            kubernetes-enabled: false
          k8s-array:
            logs:
              config:
                cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
                kubernetes-enabled: false
    cc @Yuvraj @Brian Connolly
    Yee

    Yee

    2 months ago
    @Marc Paquette i don’t think what haytham wrote is in our documentation - mind finding a home for it?
    e

    Edgar Trujillo

    2 months ago
    Yes, using FluentD setup through Container-Insights-setup-EKS-quickstart.html I see logs in
    /aws/containerinsights/ml-pipelines/*
    but with the updated config map matching
    task_logs.yaml: |
        plugins:
          logs:
            cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
            kubernetes-enabled: false
          k8s-array:
            logs:
              config:
                cloudwatch-template-uri: '<https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logsV2:log-groups/log-group/bv-ml-pipelines$3FlogStreamNameFilter=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}'
                kubernetes-enabled: false
    FlyteConsole is showing no logs... can this be a permissions issue? Either flyte-user/system role not having the permissions?
    I also updated
    fluent-bit-config
    to:
    log_stream_prefix   var.
    as the default was
    log_stream_prefix   ${HOST_NAME}-
    Yee

    Yee

    2 months ago
    at the same level as
    cloudwatch-template-uri
    , could you set
    cloudwatch-enabled
    to
    true
    as well?
    that is the only bit I’m seeing that might prevent that from showing up.
    Haytham Abuelfutuh

    Haytham Abuelfutuh

    2 months ago
    Yes that ^ I missed that… thanks Yee