I’ve just realised all the environment variable defined within the inline config are becoming lowercase in the actual task pods. This snippet is from the

flyte-backend-flyte-binary-config

ConfigMap and everything looks correct

plugins:
  k8s:
    default-env-vars:
      AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
      AWS_METADATA_SERVICE_TIMEOUT: 5
      MLFLOW_TRACKING_UI: <http://mlflow.mlflow.svc.cluster.local>

However, looking at the manifest of a task using the Flyte MLFlow plugin, the env vars look like

- name: aws_metadata_service_timeout
  value: '5'
- name: mlflow_tracking_ui
  value: <http://mlflow.mlflow.svc.cluster.local>
- name: aws_metadata_service_num_attempts
  value: '20'

This is causing the MLFlow plugin to track locally as it can’t find MLFLOW_TRACKING_UI Is this a bug or am I doing something wrong?

@glamorous-carpet-83516, any idea why this might be happening?

it’s a bug. which version of flytepropeller are you using

I’m not sure exactly how to find the flytepropeller version? I’m currently on 1.4.0 of Flyte if that helps?

it helps. going to test it today. it should not be converted to lower case.

I just updated to 1.4.2, and the same happens still btw

ok, thanks for the information, it’s a high priority. fixing it now.

@glamorous-carpet-83516 @hallowed-mouse-14616 - Sorry to pester, but did anything come of this? Not been able to find anything on my side

@gray-ocean-62145 I'm also not able to reproduce this. Cay you say more about your environment? For example what what k8s provider and version, etc?

Hey, yeah its EKS on k8s v1.22. Weirdly all the environment variables seem to be provided twice… 😆 I’ll do a rewrite of the helm values at some point - its in unwieldy terraform at the moment - but everything looks right in the ConfigMap, which makes me think its not a deployment issue?

Yeah this is very strange 😅, because our demo cluster is EKS with k8s 1.22 as well and there are no issues.

Nope, with everything - it was happening with those AWS vars the whole time, I only noticed when trying out the mlflow plugin because it wasn’t picking the address up so I went digging