Hi team our Flyte deploy is acting up today Flyteadmin and F

Question

Hi team our Flyte deploy is acting up today Flyteadmin and Flytescheduler pods keep going down randomly We ve not touched the deployment itself hence quite confused about how to root cause this In the scheduler logs we see this `rpc error code = Unavailable desc = connection error desc = transport Error while dialing dial tcp 172 20 71 119 81 connect connection refused ` Not sure what component it s trying to connect to and not sure what s causing the flakiness Replicas are starting and stopping somewhat arbitrarily Any insight would be appreciated

David Espejo (he/him) · Answer

HI < Rupsha Chaudhuri> Is it possible to identify to which component is `172 20 71 119` assigned to Probably if you issue `kubectl get service` `kubectl get endpoints`

Rupsha Chaudhuri · Answer

ok let me try that

João Lobo Guerra Neto · Answer

I have same error

João Lobo Guerra Neto · Answer

` kubectl get service n flyte` `NAME TYPE CLUSTER IP EXTERNAL IP PORT S AGE` `datacatalog ClusterIP 10 0 208 201 lt none gt 88 TCP 89 TCP 15h` `flyte pod webhook ClusterIP 10 0 72 200 lt none gt 443 TCP 15h` `flyteadmin ClusterIP 10 0 221 40 lt none gt 87 TCP 80 TCP 81 TCP 15h` `flyteconsole ClusterIP 10 0 36 237 lt none gt 80 TCP 15h` `minio ClusterIP 10 0 102 150 lt none gt 9000 TCP 9001 TCP 15h` `minio direct NodePort 10 0 141 140 lt none gt 9000 30084 TCP 15h` `postgres ClusterIP 10 0 63 11 lt none gt 5432 TCP 15h` `postgres direct NodePort 10 0 111 195 lt none gt 5432 30083 TCP 15h`

João Lobo Guerra Neto · Answer

` kubectl get endpoints n flyte` `NAME ENDPOINTS AGE` `datacatalog 10 224 2 126 8089 10 224 2 126 8088 15h` `flyte pod webhook 10 224 16 102 9443 15h` `flyteadmin 10 224 0 248 8089 10 224 0 248 8088 10 224 0 248 8087 15h` `flyteconsole 10 224 31 209 8080 15h` `minio 10 224 6 229 9000 10 224 6 229 9001 15h` `minio direct 10 224 6 229 9000 15h` `postgres 10 224 9 110 5432 15h` `postgres direct 10 224 9 110 5432 15h`

João Lobo Guerra Neto · Answer

`Using config file etc flyte config admin yaml etc flyte config db yaml etc flyte config logger yaml ` ` json src client go 181 level error msg failed to initialize token source provider Err failed to fetch auth metadata Error rpc error code = Unavailable desc = connection error desc = transport Error while dialing dial tcp 10 0 221 40 81 connect connection refused ts 2023 03 08T01 03 59Z ` ` json src client go 186 level warning msg Starting an unauthenticated client because can t create authenticated channel without a TokenSourceProvider ts 2023 03 08T01 03 59Z ` ` json src client go 65 level info msg Initialized Admin client ts 2023 03 08T01 03 59Z ` ` json src precheck go 32 level error msg Attempt failed due to rpc error code = Unavailable desc = connection error desc = transport Error while dialing dial tcp 10 0 221 40 81 connect connection refused ts 2023 03 08T01 03 59Z ` ` json src client go 65 level info msg Initialized Admin client ts 2023 03 08T01 04 00Z ` ` json src precheck go 32 level error msg Attempt failed due to rpc error code = Unavailable desc = connection error desc = transport Error while dialing dial tcp 10 0 221 40 81 connect connection refused ts 2023 03 08T01 04 01Z `