https://flyte.org logo
#ask-the-community
Title
# ask-the-community
m

Michael Tinsley

03/17/2023, 4:08 PM
I’ve just realised all the environment variable defined within the inline config are becoming lowercase in the actual task pods. This snippet is from the
flyte-backend-flyte-binary-config
ConfigMap and everything looks correct
Copy code
plugins:
  k8s:
    default-env-vars:
      AWS_METADATA_SERVICE_NUM_ATTEMPTS: 20
      AWS_METADATA_SERVICE_TIMEOUT: 5
      MLFLOW_TRACKING_UI: <http://mlflow.mlflow.svc.cluster.local>
However, looking at the manifest of a task using the Flyte MLFlow plugin, the env vars look like
Copy code
- name: aws_metadata_service_timeout
  value: '5'
- name: mlflow_tracking_ui
  value: <http://mlflow.mlflow.svc.cluster.local>
- name: aws_metadata_service_num_attempts
  value: '20'
This is causing the MLFlow plugin to track locally as it can’t find MLFLOW_TRACKING_UI Is this a bug or am I doing something wrong?
s

Samhita Alla

03/20/2023, 6:48 AM
@Kevin Su, any idea why this might be happening?
k

Kevin Su

03/20/2023, 7:03 AM
it’s a bug. which version of flytepropeller are you using
m

Michael Tinsley

03/20/2023, 8:14 AM
I’m not sure exactly how to find the flytepropeller version? I’m currently on 1.4.0 of Flyte if that helps?
k

Kevin Su

03/20/2023, 4:02 PM
it helps. going to test it today. it should not be converted to lower case.
m

Michael Tinsley

03/20/2023, 4:04 PM
I just updated to 1.4.2, and the same happens still btw
k

Kevin Su

03/20/2023, 4:34 PM
ok, thanks for the information, it’s a high priority. fixing it now.
I just tried 1.4.2. my envs are still uppercase. cc @Dan Rammer (hamersaw) any idea why this might be happening?
m

Michael Tinsley

03/30/2023, 8:14 AM
@Kevin Su @Dan Rammer (hamersaw) - Sorry to pester, but did anything come of this? Not been able to find anything on my side
d

Dan Rammer (hamersaw)

03/30/2023, 3:05 PM
@Michael Tinsley I'm also not able to reproduce this. Cay you say more about your environment? For example what what k8s provider and version, etc?
m

Michael Tinsley

03/30/2023, 3:21 PM
Hey, yeah its EKS on k8s v1.22. Weirdly all the environment variables seem to be provided twice… 😆 I’ll do a rewrite of the helm values at some point - its in unwieldy terraform at the moment - but everything looks right in the ConfigMap, which makes me think its not a deployment issue?
d

Dan Rammer (hamersaw)

03/30/2023, 3:27 PM
Yeah this is very strange 😅, because our demo cluster is EKS with k8s 1.22 as well and there are no issues.
Do you know if this only occurs with the Flyte MLFlow plugin?
m

Michael Tinsley

03/30/2023, 3:31 PM
Nope, with everything - it was happening with those AWS vars the whole time, I only noticed when trying out the mlflow plugin because it wasn’t picking the address up so I went digging
73 Views