Hi, wondering what people are using for viewing/archiving container logs on self managed on prem k8s... did anyone find a nice setup yet, since flyte can't collect and archive task logs for you?

A common thing to consider for on-prem logging infra is the "ELK" stack (ElasticSearch, LogStash, & Kibana), or the alternative "EFK" stack (which replaces LogStash w/ fluentd). The drawback here is needing to manage an on-prem elasticsearch deployment, which can be a bit onerous. https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes If you're comfortable with using a hosted service for this, https://logz.io/ is a pretty good solution and can integrate w/ on prem environments.

Thanks! The ELK stack is too heavy weight for me. Right now I'm using parts of the Grafana LGTM stack (so Loki for logs), but was wondering if there is an even simpler solution e.g. via a sidecar that uploads the logs to S3 (e.g. Argo workflows already has something like that integrated).

I rigged up a thing with fluentd that writes JSON logs to S3 that uses https://docs.fluentd.org/output/s3. We wanted a more generic solution that worked for both Argo & Flyte incidentally. Notably this achieves this w/ a daemonset rather than a sidecar

cool, thanks! I'll go with grafana agent operator (also creates daemonsets to scrape logs and metrics) and see how it goes...