#flyte-deployment - I have a weird db authenticati

userSettings:
  dbHost: xx.xx.xx.xx
  dbPassword: secret

  datacatalog:
    database:
      port: 5432
      username: "my-flyteadmin"
      host: "{{ .Values.userSettings.dbHost }}"
      dbname: "my-flyteadmin"
      passwordPath: /etc/db/pass.txt

  admin:
    database:
      port: 5432
      username: "my-flyteadmin"
      host: "{{ .Values.userSettings.dbHost }}"
      dbname: "my-flyteadmin"
      passwordPath: /etc/db/pass.txt

After installing flyte,

flyteadmin

fails to connect to the db:

run-migrations [error] failed to initialize database, got error failed to connect to `host=xx.xx.xx.xx user=my-flyteadmin database=my-flyteadmin`: server error (FATAL: password authentication failed for user "my-flyteadmin" (SQLSTATE 28P01))

At the same time,

datacatalog

with the same settings works fine. Any idea what could cause such a behavior? ~In another installation where db and username is still the default

flyteadmin

this setup works fine.~

This is a fresh installation using the chart from current master.
flyteadmin v0.6.104
y
Yuvraj
5 months ago
Can you test your db credential directly like this https://docs.flyte.org/en/latest/deployment/aws/manual.html#check-connectivity-to-the-rds-database-from-the-eks-cluster ?
s
Sören Brunk
5 months ago
@Yuvraj yes I tried that and it's working fine.
```
datacatalog
```
can connect as well. I wonder why
```
flyteadmin
```
behaves differently here.
y
Yuvraj
5 months ago
cc: @Prafulla Mahindrakar
s
Sören Brunk
5 months ago
Going back to v0.6.99 fixed the issue for me. Since it appears with v0.6.100, my suspicion is that it could be related to the db changes inhttps://github.com/flyteorg/flyteadmin/pull/345 @katrina
p
Prafulla Mahindrakar
5 months ago
strange that it works with default flyteadmin user and db though .
s
Sören Brunk
5 months ago
Perhaps I was still using an older version of the chart (and therefore a working flyteadmin version) when I tried with the default flyteadmin user.
Yes I can trigger the issue even with the default flyteadmin user by upgrading to > v0.6.99
p
Prafulla Mahindrakar
5 months ago
cool thanks Soren . Do you mind opening an issue for it.
s
Sören Brunk
5 months ago
Sure: https://github.com/flyteorg/flyte/issues/2211
🙏 1

Prafulla Mahindrakar

5 months ago

Hi @Sören Brunk i am unable to reproduce this issue from the master branch . I am performing the following steps • cd charts/flyte • helm install -n flyte -f values.yaml --create-namespace flyte .

kubectl get pods -n flyte                                      
NAME                                          READY   STATUS              RESTARTS   AGE
datacatalog-58cdbb6989-lw5mb                  1/1     Running             0          4m13s
flyte-contour-contour-7df75dc949-lnfvg        1/1     Running             0          4m13s
flyte-contour-envoy-7bg2h                     2/2     Running             0          4m13s
flyte-kubernetes-dashboard-7cb76cfdb5-799gl   1/1     Running             0          4m13s
flyte-pod-webhook-cc799b94-h87r9              1/1     Running             0          4m13s
flyte-spark-operator-6577758bff-khzdt         1/1     Running             0          4m13s
flyteadmin-59f5d5c5f8-bzsl4                   2/2     Running             0          4m13s
flyteconsole-6c46567545-2lp9x                 1/1     Running             0          4m13s
flytepropeller-86c9444478-sx9pg               1/1     Running             0          4m13s
flytescheduler-595d6dc786-n56jg               1/1     Running             0          4m13s
minio-68744577b6-wspfj                        1/1     Running             0          4m13s
postgres-84c58d5c66-knzpz                     1/1     Running             0          4m13s

Can you help me with what steps you are performing to get to this issue

s
Sören Brunk
5 months ago
I've only tested it using the GCP setup with cloud sql. Sandbox might behave differently.
p
Prafulla Mahindrakar
5 months ago
Since this works on sandbox , it seems to be a config issue with GCP. charts. We will try this on an internal GCP setup.
👍 1
s
Sören Brunk
5 months ago
Not sure if this is really GCP specific. I think it doesn't show in sandbox because the auth method in sandbox is
```
trust
```
, which essentially means all auth checks are disabled:https://github.com/flyteorg/flyte/blob/44786a8869d1ec9521e3acfd9f7126241cd97652/charts/flyte/templates/postgres/deployment.yaml#L24-L25
y
Yuvraj
5 months ago
@Sören Brunk I also tested latest flyteadmin on aws and it works fine, AWS have similar config to GCP
s
Sören Brunk
5 months ago
Hmm that's weird, so maybe it is specific to GCP/Cloud SQL. But I still guess it must be related to some recent changes because older versions work for me too.
p
Prafulla Mahindrakar
5 months ago
This should now be resolved in v0.6.107 of flyteadmin. Thanks @katrina for the fix

Join thread in Slack

View count: 19