Hey, is anyone sending logs from Flyte to Datadog using Datadog agent? We want to do it using annotations but DataDog log scraping discovery relies on this annotation:

<http://ad.datadoghq.com/|ad.datadoghq.com/><CONTAINER_IDENTIFIER>.logs: '[<LOG_CONFIG>]'

->

<CONTAINER_IDENTIFIER>

here needs to be identical to the container name from which to scrape the logs. When running Flyte tasks the container name is generated from the execution ID which cannot be known in advance. It is possible to configure default annotations for all Flyte workflow pods centrally. It is even possible to inject things like project name and domain dynamically into these default annotations but it is not possible to dynamically inject the container name(s).

I guess you could/should? scrape all logs in the flyte project namespaces and then just create links to the logs for the respective task where you can specify the container id: https://docs.flyte.org/projects/cookbook/en/stable/auto/deployment/configure_logging_links.html

Yeah but I don’t wanna do that manually as Data Scientists are creating projects on their own and I don’t want to have to edit anything for new projects which means new namespaces.