@ancient-wolf-19325 I have installed Flyte with the flyte-core Helm chart. What do I need to add to the chart values to export traces to an OTLP gRPC collector?

To export traces to an OTLP gRPC collector using Flyte, you need to configure the OpenTelemetry (OTEL) settings in your Flyte deployment. According to the Flyte documentation, you should set the

otel

section in your configuration to use

otlpgrpc

as the type. You will also need to specify the endpoint for the OTLP gRPC collector. Here's a basic example of what you might add to your Helm chart values:

otel:
  type: otlpgrpc
  otlpgrpc:
    endpoint: "<http://your-otlp-grpc-collector-endpoint:4317>"
  sampler:
    parentSampler: always

Would you like more detail on configuring other aspects of the OTEL settings? Sources: - Flyte documentation 👍 1 button 👎 0 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.

@ancient-wolf-19325 in which section (or sections) of the chart values should I add that configuration for the multiple services (admin, propeller, scheduler, datacatalog)?

It seems likely that you would need to add the OpenTelemetry configuration under each service's specific configuration section in the Helm chart values. Each Flyte component (admin, propeller, scheduler, datacatalog) may have its own configuration block where you can specify environment variables or other settings. For example, you might add the OTEL configuration under the

env

section for each service like this:

flyteadmin:
  env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "<http://your-otlp-grpc-collector-endpoint:4317>"
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: "grpc"

flytepropeller:
  env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "<http://your-otlp-grpc-collector-endpoint:4317>"
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: "grpc"

flytescheduler:
  env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "<http://your-otlp-grpc-collector-endpoint:4317>"
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: "grpc"

datacatalog:
  env:
    - name: OTEL_EXPORTER_OTLP_ENDPOINT
      value: "<http://your-otlp-grpc-collector-endpoint:4317>"
    - name: OTEL_EXPORTER_OTLP_PROTOCOL
      value: "grpc"

Would you like more detail on how to apply these settings or verify them? Sources: - https://github.com/flyteorg/flyte/blob/master/charts/flyte/values.yaml 👍 1 button 👎 0 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.

@ancient-wolf-19325 there's no alternative? other configurations are added to the configmaps

Yes, you can use ConfigMaps to manage configuration for Flyte services. According to the Flyte GitHub pull request, you can specify additional configurations via an existing ConfigMap. This approach allows you to manage configurations centrally and inject them into the Flyte services. To use a ConfigMap for OTEL configuration, you would create a ConfigMap with the necessary settings and then reference it in your Helm chart values. Here's a general approach: 1. Create a ConfigMap: Define a ConfigMap with your OTEL configuration.

apiVersion: v1
kind: ConfigMap
metadata:
  name: flyte-otel-config
data:
  otel-config.yaml: |
    otel:
      type: otlpgrpc
      otlpgrpc:
        endpoint: "<http://your-otlp-grpc-collector-endpoint:4317>"
      sampler:
        parentSampler: always

2. Reference the ConfigMap in Helm Values: Update your Helm chart values to use this ConfigMap.

flyteadmin:
  configMap: flyte-otel-config

flytepropeller:
  configMap: flyte-otel-config

flytescheduler:
  configMap: flyte-otel-config

datacatalog:
  configMap: flyte-otel-config

Would you like more detail on creating and applying ConfigMaps in Kubernetes? Sources: - Flyte GitHub pull request 👍 0 button 👎 1 button 💡 Hint: Mention @ancient-wolf-19325 in the thread for followups.

@ripe-intern-70240 you'd need to set it under

configmap.core.propeller.otel

thanks @average-finland-92144. I'll try that.

yeah, it takes some "reverse engineering" to go from the charts to the config reference. There's some heavy templating in the charts that accounts for part of this, but the rest is just waiting to be simplified

@average-finland-92144 I got this in the propeller core.yaml config.

propeller:
  otel:
    otlpgrpc:
      endpoint: '<http://otelcollector.observability.svc.cluster.local:4317>'
    type: otlpgrpc

can you please confirm this is correct? I don't see any traces and nothing shows up in the propeller pod logs, like an error for not being able to connect to the collector. btw, what calls are traced?

sorry, what about setting it under

configmap.otel

directly? There's a

propeller

section under the propeller config which may be causing that this config is not having any implication

@average-finland-92144 I moved it one section up -

configmap.core.otel

and it worked 😉

oh right this is a specially confusing corner of the config unf

this will only get traces from the propeller to other services, right? I see that

RegisterTracerProviderWithContext

is also called in flyteadmin and datacatalog, besides flytepropeller. or is

cmd/single

the entrypoint for all services which enables all providers?