Team, my cloudwatch log push is still not working. i dont see any logs after our spark workflow gets complete. this is my yaml configuraion. task_logs: plugins: logs: kubernetes-enabled: true # -- One option is to enable cloudwatch logging for EKS, update the region and log group accordingly # You can even disable this cloudwatch-enabled: true # -- region where logs are hosted cloudwatch-region: "us-east-1" # -- cloudwatch log-group cloudwatch-log-group: "/aws/vpcflowLogs/dev-max-ml-flyte" templates: - displayName: logs_test templateUris: - "https://console.aws.amazon.com/cloudwatch/home?region=us-east-1#logEventViewer:group=/aws/vpcflowLogs/dev-max-ml-flyte;stream=var.log.containers.{{.podName}}_{{.namespace}}_{{.containerName}}-{{.containerId}}.log" #messageFormat: "JSON" # "unknown" | "csv" | "json" ======================================================= i also see this exception after clicking the cloudwatch log link from the flyte console. however log groups are configured properly in aws

@icy-agent-73298, could you help?

@mammoth-lion-71167 are you able to see other non spark task logs in cloudwatch. Also for spark can you check this which requires to segregate user, system logs https://docs.flyte.org/projects/cookbook/en/latest/auto/integrations/kubernetes/k8s_spark/index.html#step-3-optionally-setup-visibility Would be good to see first if normal tasks logs are flowing to cloud watch correctly using your initial configuration . And also when you click the log link you need to be in the same aws account where logs are being published to.

@icy-agent-73298 currently i am testing only spark. and aws account in same between flyte config and published account. is there any command like kubectle logs to check the clouwatch agent services logs at flyte side.

Can you check this thread https://flyte-org.slack.com/archives/C01P3B761A6/p1656366983976689 You might be missing configuring the mechanism for pushing the logs and we seem to missing docs on it. cc @tall-lock-23197

is fluent-bit mandatory for pushing logs from flyte to cloudwatch?

Amazon recommends using that https://aws.amazon.com/premiumsupport/knowledge-center/cloudwatch-stream-container-logs-eks/ for streaming logs from eks containers https://docs.aws.amazon.com/AmazonCloudWatch/latest/monitoring/Container-Insights-EKS-logs.html

thanks for that. since fluent-bit nowhere mentioned in flyte document, my assumption was by default flyte should able to steam logs to cloudwatch.

fair assumption with missing docs. Would you help with contributing this bit of docs once you have this setup in your environment. would really benefit the community.

@icy-agent-73298 sure. once i make it work from my side

@mammoth-lion-71167 Flyte cannot talk about how you push logs. There are multiple options. I do not think it is fair for Flyte to document how to push the logs- this is completely independent