https://flyte.org
Join Slack
Menu
Home
Archive
I'm trying to deploy Flyte to GCP using <deploy-flyte> and there's one resource creation that seems ...
# flyte-deployment
r
I'm trying to deploy Flyte to GCP using deploy-flyte and there's one resource creation that seems to be timing out:
Copy code
helm_release.flyte-core: Still creating... [5m10s elapsed]
╷
│ Warning: Helm release "flyte-core" was created but has a failed status. Use the `helm` command to investigate the error, correct it, then run Terraform again.
│
│   with helm_release.flyte-core,
│   on <http://flyte.tf|flyte.tf> line 6, in resource "helm_release" "flyte-core":
│    6: resource "helm_release" "flyte-core" {
│
╵
╷
│ Error: context deadline exceeded
I'm not sure how to "use the 
helm
command to investigate". I do wonder whether this entry in Log Explorer is related:
Copy code
"panic: failed to connect to `host=10.99.0.3 user=flyteadmin database=flyteadmin`: failed SASL auth (FATAL: password authentication failed for user "flyteadmin" (SQLSTATE 28P01))

goroutine 1 [running]:
<http://github.com/flyteorg/flyte/datacatalog/pkg/repositories.Migrate({0x26dd960|github.com/flyteorg/flyte/datacatalog/pkg/repositories.Migrate({0x26dd960>, 0x38a2e00})
	/go/src/github.com/flyteorg/datacatalog/pkg/repositories/initialize.go:63 +0x786"
This is a fresh install, so I wonder if maybe flyte-deploy and flyte-core have gotten out of sync somehow?
c
What does 
kubectl get pods -n flyte
say?
r
Copy code
NAME                                 READY   STATUS                  RESTARTS        AGE
datacatalog-5ddfd6798-bc277          0/1     Init:CrashLoopBackOff   10 (106s ago)   28m
flyte-pod-webhook-8688dcf4bd-65cm5   1/1     Running                 0               28m
flyteadmin-854879bdd7-ffrr4          0/1     Init:CrashLoopBackOff   10 (117s ago)   28m
flyteconsole-b5f48f75b-l4kjm         1/1     Running                 0               28m
flytepropeller-b8f98cffb-sbncm       1/1     Running                 0               28m
Copy code
➜  flyte-core git:(main) ✗ kubectl logs flyteadmin-854879bdd7-ffrr4 -n flyte
Defaulted container "flyteadmin" out of: flyteadmin, run-migrations (init), seed-projects (init), generate-secrets (init)
Error from server (BadRequest): container "flyteadmin" in pod "flyteadmin-854879bdd7-ffrr4" is waiting to start: PodInitializing
➜  flyte-core git:(main) ✗ kubectl logs datacatalog-5ddfd6798-bc277 -n flyte
Defaulted container "datacatalog" out of: datacatalog, run-migrations (init)
Error from server (BadRequest): container "datacatalog" in pod "datacatalog-5ddfd6798-bc277" is waiting to start: PodInitializing
c
Can you pls check the init containers’ logs?
r
Sorry, I'm new to k8s. Do you mean something other than the logs in my last message?
For both containers, I see these entries repeated many times
Ok, I think I have more things working now. Well, my containers have all started at least. I had to get the 
db-pass
secret from kubectl and manually create a 
flyteadmin
user on the DB. There seems to be an inconsistency between sql.tf and 
flyte-db
in terms of what users need to exist, since 
sql.tf
only creates a 
flyte
user, not 
flyteadmin
. At least that's my theory.
c
Yes I’m sure the user needs to be called 
flyteadmin
!
r
I believe the dbname should also be 
flyteadmin
, not 
flyte
.
c
Yes
I can confirm that our db has user and db flyteadmin, not flyte.
Would you be willing to file a PR in the union oss repo?
I can review
r
Sure
c
@average-finland-92144 fyi
👀 1
r
PR created
h
looks like it was changed from 
flyteadmin
to 
flyte
recently
a
right @hallowed-dog-74273. By itself and as long as we use the same in values, it doesn't cause issues
13 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by