Hey guys, sorry if this is a dumb question but recently been deploying a series of clusters with Flyte but one of them is constantly running into this error:

2023/06/21 16:43:05 /go/pkg/mod/gorm.io/gorm@v1.24.1-0.20221019064659-5dd2bb482755/callbacks.go:134
[126.528ms] [rows:-] SELECT c.column_name, constraint_name, constraint_type FROM information_schema.table_constraints tc JOIN information_schema.constraint_column_usage AS ccu USING (constraint_schema, constraint_name) JOIN information_schema.columns AS c ON c.table_schema = tc.constraint_schema AND tc.table_name = c.table_name AND ccu.column_name = c.column_name WHERE constraint_type IN ('PRIMARY KEY', 'UNIQUE') AND c.table_catalog = 'app' AND c.table_schema = CURRENT_SCHEMA() AND c.table_name = 'datasets'
{"json":{"src":"start.go:174"},"level":"panic","msg":"Failed to start Admin, err: database migration failed: ERROR: relation \"description_entities\" does not exist (SQLSTATE 42P01)","ts":"2023-06-21T16:43:05Z"}
panic: (*logrus.Entry) 0xc000492230

goroutine 68 [running]:
<http://github.com/sirupsen/logrus.(*Entry).log(0xc0004921c0|github.com/sirupsen/logrus.(*Entry).log(0xc0004921c0>, 0x0, {0xc001218200, 0x7d})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:259 +0x45b
<http://github.com/sirupsen/logrus.(*Entry).Log(0xc0004921c0|github.com/sirupsen/logrus.(*Entry).Log(0xc0004921c0>, 0x0, {0xc001487e68?, 0x1?, 0x1?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:293 +0x4f
<http://github.com/sirupsen/logrus.(*Entry).Logf(0xc0004921c0|github.com/sirupsen/logrus.(*Entry).Logf(0xc0004921c0>, 0x0, {0x3071945?, 0x0?}, {0xc0009c3c50?, 0x0?, 0x0?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:338 +0x85
<http://github.com/sirupsen/logrus.(*Entry).Panicf(0x3f0ca00|github.com/sirupsen/logrus.(*Entry).Panicf(0x3f0ca00>?, {0x3071945?, 0x416667?}, {0xc0009c3c50?, 0x29b5840?, 0x1?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.8.1/entry.go:376 +0x34
<http://github.com/flyteorg/flytestdlib/logger.Panicf({0x3f0ca00|github.com/flyteorg/flytestdlib/logger.Panicf({0x3f0ca00>?, 0xc0009ead40?}, {0x3071945, 0x1e}, {0xc0009c3c50, 0x1, 0x1})
	/go/pkg/mod/github.com/flyteorg/flytestdlib@v1.0.17/logger/logger.go:188 +0x64
<http://github.com/flyteorg/flyte/cmd/single.glob..func4.1()|github.com/flyteorg/flyte/cmd/single.glob..func4.1()>
	/flyteorg/build/cmd/single/start.go:174 +0xcc
<http://golang.org/x/sync/errgroup.(*Group).Go.func1()|golang.org/x/sync/errgroup.(*Group).Go.func1()>
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
created by <http://golang.org/x/sync/errgroup.(*Group).Go|golang.org/x/sync/errgroup.(*Group).Go>
	/go/pkg/mod/golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5

Two other clusters were deployed just yesterday and never ran into this error. Hopefully it's something in terms of configuration that I've overlooked but was wondering if anybody had some insight. EDIT: Two other clusters were using Postgres 14.6 and the one erroring out is using 14.7; could that be the source of the error?

the version of postgres should not matter.

That's strange. I recreated the database in RDS with postgres engine 14.6 and the error disappeared. Nothing else was changed. Very strange but glad to have solved it. Not sure if it's worth investigating though on the Flyte dev team's end as I'm sure there are more pressing matters to attend to. Sorry for the bother!

and you’re starting from scratch each time? the error message is saying “database migration failed”. can you check to see if there’s any more logs from that?

To be honest, there are too many variables to say definitively that the engine version was the root cause of the error. While I do think it may be a potential issue, I don't want to push the issue further as it may not be productive if it turns out to be something else

I am having the same issue too. Flyte with postgres 14.6 is working but not with postgres 14.7. This is a big issue for us. Blocking us from going to PROD.

let us investigate this today. thank you for raising this!