https://flyte.org
Join Slack
I have a weird db authentication problem. Using Cloud SQL on GCP with private IP, I created a db and...
# flyte-deployment
b
I have a weird db authentication problem. Using Cloud SQL on GCP with private IP, I created a db and user 
my-flyteadmin
. I also changed the name in values.yaml accordingly:
Copy code
userSettings:
  dbHost: xx.xx.xx.xx
  dbPassword: secret

  datacatalog:
    database:
      port: 5432
      username: "my-flyteadmin"
      host: "{{ .Values.userSettings.dbHost }}"
      dbname: "my-flyteadmin"
      passwordPath: /etc/db/pass.txt

  admin:
    database:
      port: 5432
      username: "my-flyteadmin"
      host: "{{ .Values.userSettings.dbHost }}"
      dbname: "my-flyteadmin"
      passwordPath: /etc/db/pass.txt
After installing flyte, 
flyteadmin
fails to connect to the db:
Copy code
run-migrations [error] failed to initialize database, got error failed to connect to `host=xx.xx.xx.xx user=my-flyteadmin database=my-flyteadmin`: server error (FATAL: password authentication failed for user "my-flyteadmin" (SQLSTATE 28P01))
At the same time, 
datacatalog
with the same settings works fine. Any idea what could cause such a behavior? In another installation where db and username is still the default 
flyteadmin
this setup works fine.
This is a fresh installation using the chart from current master.
flyteadmin v0.6.104
g
b
@great-school-54368 yes I tried that and it's working fine. 
datacatalog
can connect as well. I wonder why 
flyteadmin
behaves differently here.
g
cc: @icy-agent-73298
b
Going back to v0.6.99 fixed the issue for me. Since it appears with v0.6.100, my suspicion is that it could be related to the db changes in https://github.com/flyteorg/flyteadmin/pull/345 @acceptable-policeman-57188
i
strange that it works with default flyteadmin user and db though .
b
Perhaps I was still using an older version of the chart (and therefore a working flyteadmin version) when I tried with the default flyteadmin user.
Yes I can trigger the issue even with the default flyteadmin user by upgrading to > v0.6.99
i
cool thanks Soren . Do you mind opening an issue for it.
b
i
Hi @boundless-pizza-95864 i am unable to reproduce this issue from the master branch . I am performing the following steps ā€¢ cd charts/flyte ā€¢ helm install -n flyte -f values.yaml --create-namespace flyte .
Copy code
kubectl get pods -n flyte                                      
NAME                                          READY   STATUS              RESTARTS   AGE
datacatalog-58cdbb6989-lw5mb                  1/1     Running             0          4m13s
flyte-contour-contour-7df75dc949-lnfvg        1/1     Running             0          4m13s
flyte-contour-envoy-7bg2h                     2/2     Running             0          4m13s
flyte-kubernetes-dashboard-7cb76cfdb5-799gl   1/1     Running             0          4m13s
flyte-pod-webhook-cc799b94-h87r9              1/1     Running             0          4m13s
flyte-spark-operator-6577758bff-khzdt         1/1     Running             0          4m13s
flyteadmin-59f5d5c5f8-bzsl4                   2/2     Running             0          4m13s
flyteconsole-6c46567545-2lp9x                 1/1     Running             0          4m13s
flytepropeller-86c9444478-sx9pg               1/1     Running             0          4m13s
flytescheduler-595d6dc786-n56jg               1/1     Running             0          4m13s
minio-68744577b6-wspfj                        1/1     Running             0          4m13s
postgres-84c58d5c66-knzpz                     1/1     Running             0          4m13s
Can you help me with what steps you are performing to get to this issue
b
I've only tested it using the GCP setup with cloud sql. Sandbox might behave differently.
i
Since this works on sandbox , it seems to be a config issue with GCP. charts. We will try this on an internal GCP setup.
šŸ‘ 1
b
Not sure if this is really GCP specific. I think it doesn't show in sandbox because the auth method in sandbox is 
trust
, which essentially means all auth checks are disabled: https://github.com/flyteorg/flyte/blob/44786a8869d1ec9521e3acfd9f7126241cd97652/charts/flyte/templates/postgres/deployment.yaml#L24-L25
g
@boundless-pizza-95864 I also tested latest flyteadmin on aws and it works fine, AWS have similar config to GCP
b
Hmm that's weird, so maybe it is specific to GCP/Cloud SQL. But I still guess it must be related to some recent changes because older versions work for me too.
i
This should now be resolved in v0.6.107 of flyteadmin. Thanks @acceptable-policeman-57188 for the fix
332 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by