Hi team! Hope to get some help on my recent upgrade to v1.14.0. We are at v1.11.0 now, when I tried to start the flyte admin in v1.14.0, I got the error message below:

{"json":{"src":"serve.go:32"},"level":"panic","msg":"Failed to Start profiling and Metrics server. Error, listen tcp :10254: bind: address already in use","ts":"2025-01-08T16:18:35Z"}

panic: (*logrus.Entry) 0xc000914e70
goroutine 12 [running]:
<http://github.com/sirupsen/logrus.(*Entry).log(0xc000914e00|github.com/sirupsen/logrus.(*Entry).log(0xc000914e00>, 0x0, {0xc00025c460, 0x64})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x491
<http://github.com/sirupsen/logrus.(*Entry).Log(0xc000914e00|github.com/sirupsen/logrus.(*Entry).Log(0xc000914e00>, 0x0, {0xc00141bed0?, 0x1?, 0x1?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x48
<http://github.com/sirupsen/logrus.(*Entry).Logf(0xc000914e00|github.com/sirupsen/logrus.(*Entry).Logf(0xc000914e00>, 0x0, {0x2ce9f43?, 0xc00141bde8?}, {0xc0014083a0?, 0x0?, 0x0?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x7c
<http://github.com/sirupsen/logrus.(*Entry).Panicf(0x3474b18|github.com/sirupsen/logrus.(*Entry).Panicf(0x3474b18>?, {0x2ce9f43?, 0x411f65?}, {0xc0014083a0?, 0x2650de0?, 0x2801?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:387 +0x2e
<http://github.com/flyteorg/flyte/flytestdlib/logger.Panicf({0x3474b18|github.com/flyteorg/flyte/flytestdlib/logger.Panicf({0x3474b18>?, 0x4d75f20?}, {0x2ce9f43, 0x37}, {0xc0014083a0, 0x1, 0x1})
	/go/src/github.com/flyteorg/flytestdlib/logger/logger.go:189 +0x5e
<http://github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.init.func7.1()|github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.init.func7.1()>
	/go/src/github.com/flyteorg/flyteadmin/cmd/entrypoints/serve.go:32 +0xd2
created by <http://github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.init.func7|github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.init.func7> in goroutine 1
	/go/src/github.com/flyteorg/flyteadmin/cmd/entrypoints/serve.go:28 +0x85
time="2025-01-08T16:15:39Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/database.yaml /etc/flyte/config/domains.yaml /etc/flyte/config/logger.yaml /etc/flyte/config/namespace_mapping.yaml /etc/flyte/config/registration.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resources.yaml]"

Could someone point me out the direction to debug?

Hey Gilbert. I suspect this should be temporary. Was it a

helm upgrade

operation? What's the status of the old flyteadmin pod?

I think we pull the cr.flyte.org/flyteorg/flyteadmin:v1.14.0 into our ecr, and use this container to spin up flyte admin. In term of the old flyteadmin pod, what status are you referring to? I can provide logs.

I mean, is it Running, or Deleting etc

It is running. We deploy it on K8s

oh, this is an "in place" update

Correct.. There is no issue in v1.11.0

right!

sure

{"json":{"src":"initialize.go:51"},"level":"info","msg":"Migration ran successfully","ts":"2025-01-08T17:16:50Z"}
Show syntax highlighted
time="2025-01-08T17:16:51Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/database.yaml /etc/flyte/config/domains.yaml /etc/flyte/config/logger.yaml /etc/flyte/config/namespace_mapping.yaml /etc/flyte/config/registration.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resources.yaml]"
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [server] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [otel] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [notifications] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [externalevents] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [cloudevents] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [task_resources] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [propeller] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [secrets] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [auth] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [remotedata] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [domains] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [namespace_mapping] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:391"},"level":"debug","msg":"Config section [admin] updated. Firing updated event.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [storage] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [clusters] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [clusterpools] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [queues] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [qualityofservice] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [plugins] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [plugins.k8s] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [plugins.catalogcache] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [database] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [flyteadmin] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [scheduler] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [cluster_resources] updated. No update handler registered.","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"service.go:76"},"level":"info","msg":"setting metrics keys to [project domain wf task phase tasktype runtime_type runtime_version app_name]","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"service.go:335"},"level":"info","msg":"Serving Flyte Admin Insecure","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"service.go:87"},"level":"info","msg":"Registering default middleware with blanket auth validation","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"service.go:121"},"level":"info","msg":"Creating gRPC server without authentication","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"server.go:96"},"level":"info","msg":"Starting profiling server on port [10254]","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
{"json":{"src":"serve.go:32"},"level":"panic","msg":"Failed to Start profiling and Metrics server. Error, listen tcp :10254: bind: address already in use","ts":"2025-01-08T17:16:51Z"}
Show syntax highlighted
panic: (*logrus.Entry) 0xc000030620
goroutine 69 [running]:
<http://github.com/sirupsen/logrus.(*Entry).log(0xc000030540|github.com/sirupsen/logrus.(*Entry).log(0xc000030540>, 0x0, {0xc0004c4070, 0x64})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:260 +0x491
<http://github.com/sirupsen/logrus.(*Entry).Log(0xc000030540|github.com/sirupsen/logrus.(*Entry).Log(0xc000030540>, 0x0, {0xc000851ed0?, 0x1?, 0x1?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:304 +0x48
<http://github.com/sirupsen/logrus.(*Entry).Logf(0xc000030540|github.com/sirupsen/logrus.(*Entry).Logf(0xc000030540>, 0x0, {0x2ce9f43?, 0xc000851de8?}, {0xc000f5a3a0?, 0x0?, 0x0?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:349 +0x7c
<http://github.com/sirupsen/logrus.(*Entry).Panicf(0x3474b18|github.com/sirupsen/logrus.(*Entry).Panicf(0x3474b18>?, {0x2ce9f43?, 0x411f65?}, {0xc000f5a3a0?, 0x2650de0?, 0x2801?})
	/go/pkg/mod/github.com/sirupsen/logrus@v1.9.3/entry.go:387 +0x2e
<http://github.com/flyteorg/flyte/flytestdlib/logger.Panicf({0x3474b18|github.com/flyteorg/flyte/flytestdlib/logger.Panicf({0x3474b18>?, 0x4d75f20?}, {0x2ce9f43, 0x37}, {0xc000f5a3a0, 0x1, 0x1})
	/go/src/github.com/flyteorg/flytestdlib/logger/logger.go:189 +0x5e
<http://github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.init.func7.1()|github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.init.func7.1()>
	/go/src/github.com/flyteorg/flyteadmin/cmd/entrypoints/serve.go:32 +0xd2
created by <http://github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.init.func7|github.com/flyteorg/flyte/flyteadmin/cmd/entrypoints.init.func7> in goroutine 1
	/go/src/github.com/flyteorg/flyteadmin/cmd/entrypoints/serve.go:28 +0x85
time="2025-01-08T17:16:52Z" level=info msg="Using config file: [/etc/flyte/config/cluster_resources.yaml /etc/flyte/config/clusters.yaml /etc/flyte/config/database.yaml /etc/flyte/config/domains.yaml /etc/flyte/config/logger.yaml /etc/flyte/config/namespace_mapping.yaml /etc/flyte/config/registration.yaml /etc/flyte/config/remoteData.yaml /etc/flyte/config/server.yaml /etc/flyte/config/storage.yaml /etc/flyte/config/task_resources.yaml]"
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [flyteadmin] updated. No update handler registered.","ts":"2025-01-08T17:16:52Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [scheduler] updated. No update handler registered.","ts":"2025-01-08T17:16:52Z"}
Show syntax highlighted
{"json":{"src":"viper.go:389"},"level":"debug","msg":"Config section [clusters] updated. No update handler registered.","ts":"2025-01-08T17:16:52Z"}
Show syntax highlighted
{"json":{"src":"database.go:162"},"level":"info","msg":"Set connection pool values to [{MaxOpenConnections:100 OpenConnections:1 InUse:0 Idle:1 WaitCount:0 WaitDuration:0s MaxIdleClosed:0 MaxIdleTimeClosed:0 MaxLifetimeClosed:0}]","ts":"2025-01-08T17:16:52Z"}
Show syntax highlighted