Hi all I m hoping to add logging to our `flyte binary` deplo Flyte #flyte-deployment

Hi all, I'm hoping to add logging to our `flyte-bi...

Cody Scandore

06/22/2023, 8:30 PM

Hi all, I'm hoping to add logging to our

flyte-binary

deployment on AWS EKS. Is there any change in AWS role permissions to allow Cloudwatch writing? If so, it looks like I have 4 active roles in IAM related to flyte (nodeInstance, cluster service, load balancer, and

flyte-system-role

Cody Scandore

06/22/2023, 8:59 PM

While I'm able to link out to Cloudwatch, when I click through, I get "log stream does not exist" in the AWS console.

Cody Scandore

06/22/2023, 8:59 PM

Screenshot 2023-06-22 at 1.59.52 PM.png

Samhita Alla

06/23/2023, 5:21 AM

How does your

task_logs

configuration look like? https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/configure_logging_links.html

Cody Scandore

06/23/2023, 6:50 PM

Copy code

logging:
    level: 1
    plugins:
      cloudwatch:
        enabled: true
        templateUri: |-
          <https://console.aws.amazon.com/cloudwatch/home?region=us-west-2#logEventViewer:group=/eks/opta-development/cluster;stream=var.log.containers.{{> .podName }}_{{ .namespace }}_{{ .containerName }}-{{ .containerId }}.log

jeev

06/26/2023, 1:27 PM

Logs are assumed to be sent to cloudwatch via fluentd or fluentbit or equivalent. Flyte task containers simply log to stdout/err. The k8s cluster needs to be configured to correctly handle the container logs. https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html

jeev

06/26/2023, 1:29 PM

Assuming that is already set up correctly, you might want to view the log groups available on Cloudwatch and see if you can navigate to the container logs in question. From there work backwards to the correct logs link template. Are you sure “/eks/opta-development” is right?

153 Views

Open in Slack

Previous Next