https://flyte.org
Join Slack
Hi I would like to set an environment variable for every Pod that gets scheduled to k8s. What is the...
# announcements
r
Hi I would like to set an environment variable for every Pod that gets scheduled to k8s. What is the preferred way to do this? Specifically I’m trying to set the following for datadog metrics
Copy code
env:
    - name: DD_AGENT_HOST
      valueFrom:
          fieldRef:
              fieldPath: status.hostIP
I have tried creating a Pod task and specifying the 
env
using 
V1EnvVar
, but Flyte doesn’t seem to translate to this field to the Pod. The request/limits are set on the Pod as I expect.
Copy code
@task(
    task_config=Pod(
        pod_spec=V1PodSpec(
            containers=[
                V1Container(
                    name="primary",
                    resources=V1ResourceRequirements(
                        requests={"cpu": ".25", "memory": "250Mi"},
                        limits={"cpu": ".5", "memory": "500Mi"},
                    ),
                    env=[V1EnvVar(name="MY_VAL", value="TEST")],
                )
            ],
        ),
        primary_container_name="primary",
    )
)
am I missing something?
t
you can also do
Copy code
kind: Service
apiVersion: v1
metadata:
  name: flytestatsd
  namespace: datadog
spec:
  selector:
    app: "your-datadog"
  ports:
  - protocol: UDP
    port: 8125
    targetPort: 8125
  topologyKeys:
    - "<http://kubernetes.io/hostname|kubernetes.io/hostname>"
    - "*"
and
Copy code
k8s:
  plugins:
    k8s:
      default-env-vars:
        - FLYTE_STATSD_HOST: "flytestatsd.datadog.svc.cluster.local"
        - FLYTE_STATSD_DISABLE_TAGS: "True"
but that latter block is how you do it
r
Hi Yee thanks! this seems like exactly what I’m looking for
what is the purpose of 
FLYTE_STATSD_DISABLE_TAGS
?
t
i think we did it specifically for datadog
other systems might be better at handling the cardinality
r
good to know thanks. Will try this shortly
@thankful-minister-83577 I’m seeing my custom metrics show up in DataDog now thank you. Should I also be seeing flyte custom metrics?
t
do you want them? we do it for propeller and admin
let’s see
Copy code
flytepropeller:
  podAnnotations:
    <http://ad.datadoghq.com/flytepropeller.check_names|ad.datadoghq.com/flytepropeller.check_names>: |
      ["openmetrics"]
    <http://ad.datadoghq.com/flytepropeller.init_configs|ad.datadoghq.com/flytepropeller.init_configs>: |
      [{}]
    <http://ad.datadoghq.com/flytepropeller.instances|ad.datadoghq.com/flytepropeller.instances>: |
      [
        {
          "prometheus_url": "<http://%%host%>%:10254/metrics",
          "namespace": "propeller",
          "metrics": [ "*" ],
          "max_returned_metrics": 10000
        }
      ]
something like that will do it.
we have this also
Copy code
plugins:
      logs:
        cloudwatch-enabled: true
        cloudwatch-log-group: '/eks/opta-development/cluster'
        cloudwatch-region: us-east-2
        kubernetes-enabled: false
        templates:
        - displayName: Datadog
          templateUris:
          - |
            <https://app.datadoghq.com/logs?event&from_ts={{> .podUnixStartTime }}000&live=false&query=pod_name%3A{{ .podName }}&to_ts={{ .podUnixFinishTime }}000
it’s pretty hacky but you can get logs assuming it’s scraping logs.
198 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by