https://flyte.org logo
#flyte-deployment
Title
# flyte-deployment
d

Devin McCabe

02/23/2024, 4:12 PM
I'm trying to deploy Flyte to GCP using deploy-flyte and there's one resource creation that seems to be timing out:
Copy code
helm_release.flyte-core: Still creating... [5m10s elapsed]
╷
│ Warning: Helm release "flyte-core" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│
│   with helm_release.flyte-core,
│   on <http://flyte.tf|flyte.tf> line 6, in resource "helm_release" "flyte-core":
│    6: resource "helm_release" "flyte-core" {
│
╵
╷
│ Error: context deadline exceeded
I'm not sure how to "use the
helm
command to investigate". I do wonder whether this entry in Log Explorer is related:
Copy code
"panic: failed to connect to `host=10.99.0.3 user=flyteadmin database=flyteadmin`: failed SASL auth (FATAL: password authentication failed for user "flyteadmin" (SQLSTATE 28P01))

goroutine 1 [running]:
<http://github.com/flyteorg/flyte/datacatalog/pkg/repositories.Migrate({0x26dd960|github.com/flyteorg/flyte/datacatalog/pkg/repositories.Migrate({0x26dd960>, 0x38a2e00})
	/go/src/github.com/flyteorg/datacatalog/pkg/repositories/initialize.go:63 +0x786"
This is a fresh install, so I wonder if maybe flyte-deploy and flyte-core have gotten out of sync somehow?
f

Fabio Grätz

02/23/2024, 4:15 PM
What does
kubectl get pods -n flyte
say?
d

Devin McCabe

02/23/2024, 4:17 PM
Copy code
NAME                                 READY   STATUS                  RESTARTS        AGE
datacatalog-5ddfd6798-bc277          0/1     Init:CrashLoopBackOff   10 (106s ago)   28m
flyte-pod-webhook-8688dcf4bd-65cm5   1/1     Running                 0               28m
flyteadmin-854879bdd7-ffrr4          0/1     Init:CrashLoopBackOff   10 (117s ago)   28m
flyteconsole-b5f48f75b-l4kjm         1/1     Running                 0               28m
flytepropeller-b8f98cffb-sbncm       1/1     Running                 0               28m
Copy code
➜  flyte-core git:(main) ✗ kubectl logs flyteadmin-854879bdd7-ffrr4 -n flyte
Defaulted container "flyteadmin" out of: flyteadmin, run-migrations (init), seed-projects (init), generate-secrets (init)
Error from server (BadRequest): container "flyteadmin" in pod "flyteadmin-854879bdd7-ffrr4" is waiting to start: PodInitializing
➜  flyte-core git:(main) ✗ kubectl logs datacatalog-5ddfd6798-bc277 -n flyte
Defaulted container "datacatalog" out of: datacatalog, run-migrations (init)
Error from server (BadRequest): container "datacatalog" in pod "datacatalog-5ddfd6798-bc277" is waiting to start: PodInitializing
f

Fabio Grätz

02/23/2024, 4:19 PM
Can you pls check the init containers’ logs?
d

Devin McCabe

02/23/2024, 4:22 PM
Sorry, I'm new to k8s. Do you mean something other than the logs in my last message?
For both containers, I see these entries repeated many times
Ok, I think I have more things working now. Well, my containers have all started at least. I had to get the
db-pass
secret from kubectl and manually create a
flyteadmin
user on the DB. There seems to be an inconsistency between sql.tf and
flyte-db
in terms of what users need to exist, since
sql.tf
only creates a
flyte
user, not
flyteadmin
. At least that's my theory.
f

Fabio Grätz

02/23/2024, 5:19 PM
Yes I’m sure the user needs to be called
flyteadmin
!
d

Devin McCabe

02/23/2024, 5:19 PM
I believe the dbname should also be
flyteadmin
, not
flyte
.
f

Fabio Grätz

02/23/2024, 5:19 PM
Yes
I can confirm that our db has user and db flyteadmin, not flyte.
Would you be willing to file a PR in the union oss repo?
I can review
d

Devin McCabe

02/23/2024, 5:20 PM
Sure
f

Fabio Grätz

02/23/2024, 5:20 PM
@David Espejo (he/him) fyi
d

Devin McCabe

02/23/2024, 5:28 PM
PR created
looks like it was changed from
flyteadmin
to
flyte
recently
d

David Espejo (he/him)

02/27/2024, 7:04 PM
right @Alex Beach. By itself and as long as we use the same in values, it doesn't cause issues
6 Views