https://flyte.org logo
#ask-the-community
Title
# ask-the-community
m

Miki ke

02/13/2024, 9:27 AM
Hi everyone, I’m currently attempting to set up Flyte monitoring with Datadog, but I’m only seeing a few infrastructure-related metrics. I’ve been searching for documentation and examples, but haven’t found much. Does anyone have experience with this and could offer assistance? Thank you!
here is what I tried so far added the following to flyteadmin section
Copy code
podAnnotations:
  ad.datadoghq.com/flyteadmin.checks: |
    {
      "openmetrics": {
        "init_config": {},
        "instances": [
          {
            "openmetrics_endpoint": "http://%%host%%:10254/metrics",
            "namespace": "ml-platform-flyte",
            "metrics": [
              ".*"
            ],
            "tags": [
              "service:ml-platform-flyte",
              "servicetier:tier3",
              "application:ml-platform-flyte",
        ],
        "send_monotonic_counter": false
      }
    ]
  }
}
and also tried to add the following to global
Copy code
datadog:
  openmetrics:
    - port: 10254
      endpoint: /management/prometheus
      # Have the namespace prepended to the metric name e.g. my_namespace.example_metric.one instead of example_metric.one N.B. existing dashboards/alerts using old format will break and needs to be updated
      namespacedMetrics: false
      # Please note that custom metrics have a $$ cost. If you need to use these metrics, please use the following property to filter the ones that you need.
      # Do not use "*".
      metricsToFetch:
        - "*"

  metadata:
    # The Datadog admission controller will inject DD_AGENT_HOST into container env variables.
    # For more info see <https://docs.datadoghq.com/agent/cluster_agent/admission_controller/>
    inject: true
    enableProfiler: false
  # Dynamically rename services via configuration. Useful for making databases have distinct names across different services.
  # Example: "mysql:my-mysql-service-name-db, postgres:my-postgres-service-name-db"
  serviceMapping: ""

  # Multi-line event log breaking. All events must start with the regex pattern, which must be a date\timestamp
  # for reference see <https://docs.datadoghq.com/logs/guide/commonly-used-log-processing-rules/>
  logging: {}
managed to make it work
s

Samhita Alla

02/21/2024, 5:34 AM
@Miki ke glad that you made it work! do you think we can add this to our docs to make the setup process easier?