• s

    Sören Brunk

    5 months ago
    I have a weird db authentication problem. Using Cloud SQL on GCP with private IP, I created a db and user
    my-flyteadmin
    . I also changed the name in values.yaml accordingly:
    userSettings:
      dbHost: xx.xx.xx.xx
      dbPassword: secret
    
      datacatalog:
        database:
          port: 5432
          username: "my-flyteadmin"
          host: "{{ .Values.userSettings.dbHost }}"
          dbname: "my-flyteadmin"
          passwordPath: /etc/db/pass.txt
    
      admin:
        database:
          port: 5432
          username: "my-flyteadmin"
          host: "{{ .Values.userSettings.dbHost }}"
          dbname: "my-flyteadmin"
          passwordPath: /etc/db/pass.txt
    After installing flyte,
    flyteadmin
    fails to connect to the db:
    run-migrations [error] failed to initialize database, got error failed to connect to `host=xx.xx.xx.xx user=my-flyteadmin database=my-flyteadmin`: server error (FATAL: password authentication failed for user "my-flyteadmin" (SQLSTATE 28P01))
    At the same time,
    datacatalog
    with the same settings works fine. Any idea what could cause such a behavior? ~In another installation where db and username is still the default
    flyteadmin
    this setup works fine.~
  • This is a fresh installation using the chart from current master.
  • flyteadmin v0.6.104
  • y

    Yuvraj

    5 months ago
  • s

    Sören Brunk

    5 months ago
    @Yuvraj yes I tried that and it's working fine.
    datacatalog
    can connect as well. I wonder why
    flyteadmin
    behaves differently here.
  • y

    Yuvraj

    5 months ago
    cc: @Prafulla Mahindrakar
  • s

    Sören Brunk

    5 months ago
    Going back to v0.6.99 fixed the issue for me. Since it appears with v0.6.100, my suspicion is that it could be related to the db changes inhttps://github.com/flyteorg/flyteadmin/pull/345 @katrina
  • p

    Prafulla Mahindrakar

    5 months ago
    strange that it works with default flyteadmin user and db though .
  • s

    Sören Brunk

    5 months ago
    Perhaps I was still using an older version of the chart (and therefore a working flyteadmin version) when I tried with the default flyteadmin user.
  • Yes I can trigger the issue even with the default flyteadmin user by upgrading to > v0.6.99
  • p

    Prafulla Mahindrakar

    5 months ago
    cool thanks Soren . Do you mind opening an issue for it.
  • s

    Sören Brunk

    5 months ago
  • p

    Prafulla Mahindrakar

    5 months ago
    Hi @Sören Brunk i am unable to reproduce this issue from the master branch . I am performing the following steps • cd charts/flyte • helm install -n flyte -f values.yaml --create-namespace flyte .
    kubectl get pods -n flyte                                      
    NAME                                          READY   STATUS              RESTARTS   AGE
    datacatalog-58cdbb6989-lw5mb                  1/1     Running             0          4m13s
    flyte-contour-contour-7df75dc949-lnfvg        1/1     Running             0          4m13s
    flyte-contour-envoy-7bg2h                     2/2     Running             0          4m13s
    flyte-kubernetes-dashboard-7cb76cfdb5-799gl   1/1     Running             0          4m13s
    flyte-pod-webhook-cc799b94-h87r9              1/1     Running             0          4m13s
    flyte-spark-operator-6577758bff-khzdt         1/1     Running             0          4m13s
    flyteadmin-59f5d5c5f8-bzsl4                   2/2     Running             0          4m13s
    flyteconsole-6c46567545-2lp9x                 1/1     Running             0          4m13s
    flytepropeller-86c9444478-sx9pg               1/1     Running             0          4m13s
    flytescheduler-595d6dc786-n56jg               1/1     Running             0          4m13s
    minio-68744577b6-wspfj                        1/1     Running             0          4m13s
    postgres-84c58d5c66-knzpz                     1/1     Running             0          4m13s
    Can you help me with what steps you are performing to get to this issue
  • s

    Sören Brunk

    5 months ago
    I've only tested it using the GCP setup with cloud sql. Sandbox might behave differently.
  • p

    Prafulla Mahindrakar

    5 months ago
    Since this works on sandbox , it seems to be a config issue with GCP. charts. We will try this on an internal GCP setup.
  • s

    Sören Brunk

    5 months ago
    Not sure if this is really GCP specific. I think it doesn't show in sandbox because the auth method in sandbox is
    trust
    , which essentially means all auth checks are disabled:https://github.com/flyteorg/flyte/blob/44786a8869d1ec9521e3acfd9f7126241cd97652/charts/flyte/templates/postgres/deployment.yaml#L24-L25
  • y

    Yuvraj

    5 months ago
    @Sören Brunk I also tested latest flyteadmin on aws and it works fine, AWS have similar config to GCP
  • s

    Sören Brunk

    5 months ago
    Hmm that's weird, so maybe it is specific to GCP/Cloud SQL. But I still guess it must be related to some recent changes because older versions work for me too.
  • p

    Prafulla Mahindrakar

    5 months ago
    This should now be resolved in v0.6.107 of flyteadmin. Thanks @katrina for the fix