https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
l

Lawson Nerenberg

03/19/2024, 11:12 PM
Hello flyte-squadron, I'm attempting to deploy
flyte-core v1.10.6
via terragrunt on gcp and the apply is failing due to the helm release for flyte timing out (context deadline exceeded). I've identified that the init run-migrations container for
flyteadmin
and
datacatalog
is failing to connect to the database: `failed to initialize database, got error failed to connect to `host=DB_IP user=DB_USER database=flyte`: dial error (timeout: dial tcp DB_IP5432 connect: connection timed out)` Do you all have any suggestions for triaging this failing database connection? I've copied the flyteadmin run-migrations container spec, and have modified it to run ubuntu to allow me to test with bash. With that manual spec applied (with all secrets mounted), I was able to reproduce connection timeout with
psql
but wasn't able to find any more clues towards network misconfigurations. *Now thinking it has to be a vpc firewall misconfiguration. So less of a flyte question but any net test tips greatly appreciated
k

Ketan (kumare3)

03/20/2024, 4:37 AM
you did the right thing, psql needs to work for flyte to work. usually on AWS this would mean security groups. on gcp a gcp expert here can point it
i know one thing that you need to deploy the sqlproxy for certain workloads. but not an expert
c

Cornelis Boon

03/20/2024, 8:51 AM
Either cloudsqlproxy (have tried this, but didn't get it to work) or have your cluster/VM and postgres DB in the same VPC (see https://github.com/unionai-oss/deploy-flyte/blob/main/environments/gcp/flyte-core/README.md for an example)
l

Lawson Nerenberg

03/20/2024, 3:36 PM
Thank you all very much for the references! To clarify, this is a GCP based deployment. Cloudsql proxy approach is preferred by us but after an initial attempt at that, I wanted to start with direct connection with everything on the same VPC. There is likely something misconfigured VPC/Firewall rule wise, Trying to get the gcloud network connectivity tests to shed some light on what wrong there.
c

Cornelis Boon

03/20/2024, 3:41 PM
If you make progress on the cloudsql proxy approach, would be glad to hear about it!