Flyteconsole is now giving 502 Bad Gateway errors How do I i

Question

Flyteconsole is now giving 502 Bad Gateway errors How do I interrogate Flyte to find out what the problem is And what are solutions to likely causes I figure something that runs the process for that functionality is dead and I don t know how to query whether it s dead nor how to restart it FWIW it s running on top of eks at AWS Sorry for the naive question My team has been using Flyte for the past year or so Unfortunately the leading engineers who introduced Flyte have left our company and I know only a little about it This isn t a prod cluster so it s not 100 important that I get this right

Fabio Grätz · Answer

Can you pls connect to the k8s cluster and post the result of `kubectl n flyte get pods`

Stephen Fromm · Answer

I get `No resources found in flyte namespace ` Though again I don t know much about these things there s stuff that looks flyte ish ```NAMESPACE NAME READY STATUS RESTARTS AGE lt snipped irrelevant content gt great falls flyte pod webhook 67b567f698 pbkg7 1 1 Running 0 170d great falls flyteadmin 6bfb478fdf 6hq8m 1 1 Running 0 176d great falls flyteadmin 6bfb478fdf fw4z8 1 1 Running 0 176d great falls flyteconsole 66d7468bf9 99tkk 1 1 Running 27 9d ago 176d great falls flyteconsole 66d7468bf9 lj82g 1 1 Running 24 3d17h ago 176d great falls flytepropeller c888bf854 zpcgx 1 1 Running 0 176d great falls flytescheduler 58dcdf9b96 8t4wd 1 1 Running 0 170d```

Fabio Grätz · Answer

Ah ok not running in the default namespace

Fabio Grätz · Answer

When you do `kubectl get pods all namespaces` is ther anything that is not running Like completed or failed

Stephen Fromm · Answer

```$ kubectl n great falls get pods NAME READY STATUS RESTARTS AGE datacatalog c485d95d4 75vgn 1 1 Running 0 176d datacatalog c485d95d4 pbkfk 1 1 Running 0 176d flyte pod webhook 67b567f698 pbkg7 1 1 Running 0 170d flyteadmin 6bfb478fdf 6hq8m 1 1 Running 0 176d flyteadmin 6bfb478fdf fw4z8 1 1 Running 0 176d flyteconsole 66d7468bf9 99tkk 1 1 Running 27 9d ago 176d flyteconsole 66d7468bf9 lj82g 1 1 Running 24 3d17h ago 176d flytepropeller c888bf854 zpcgx 1 1 Running 0 176d flytescheduler 58dcdf9b96 8t4wd 1 1 Running 0 170d syncresources 5d9df9b699 gjdqx 1 1 Running 0 170d```

Fabio Grätz · Answer

or crashloopbackoff

Fabio Grätz · Answer

Some restarts in flyte console but a few days old and seems to be running now

Stephen Fromm · Answer

Everything is `Running` except there s some prometheus stuff that is `Pending` which doesn t seem relevant

Stephen Fromm · Answer

No I m still getting `502 Bad Gateway` Was working yesterday

Fabio Grätz · Answer

Are there any nginx pods in your namespaces

Fabio Grätz · Answer

Or do you know how flyte console is exposed

Fabio Grätz · Answer

Does the error when accessing flyte console through the browser say anything about nginx

Stephen Fromm · Answer

I don t see anything in the output of `get pods` or in the error message or in the local documentation at my company about nginx

Fabio Grätz · Answer

any other ingress How do you access flyte console

Fabio Grätz · Answer

Is it exposed in the public internet Or local network

Stephen Fromm · Answer

We use VPN

Stephen Fromm · Answer

Under `kubectl describe ep` I see some things which are a little suspicious ```Annotations <http endpoints kubernetes io last change trigger time|endpoints kubernetes io last change trigger time> 2023 07 27T17 52 46Z``` and ```Subsets Addresses lt none gt NotReadyAddresses 5 0 12 118 Ports Name Port Protocol http 7979 TCP``` Those are both under `Name external dns`

Fabio Grätz · Answer

What s the name of the corresponding service

Stephen Fromm · Answer

Not sure Only thing I see associated is `external dns` Is there a command I should type to get that

Fabio Grätz · Answer

Mh I still don t fully understand how your deployment looks like

Fabio Grätz · Answer

What is the output of `kubectl get service n great falls`

Fabio Grätz · Answer

Or also in all namespaces

Fabio Grätz · Answer

There must be some kind of ingress or reverse proxy I d say

Stephen Fromm · Answer

```$ kubectl get service n great falls NAME TYPE CLUSTER IP EXTERNAL IP PORT S AGE datacatalog LoadBalancer 10 100 244 46 8089 32312 TCP 88 31369 TCP 89 32734 TCP 176d flyte pod webhook ClusterIP 10 100 53 162 lt none gt 443 TCP 176d flyteadmin LoadBalancer 10 100 205 238 80 31117 TCP 81 30601 TCP 87 30822 TCP 10254 31187 TCP 176d flyteconsole LoadBalancer 10 100 150 24 80 30722 TCP 176d```

Stephen Fromm · Answer

```$ kubectl get service NAME TYPE CLUSTER IP EXTERNAL IP PORT S AGE external dns ClusterIP 10 100 185 214 lt none gt 7979 TCP 177d kubernetes ClusterIP 10 100 0 1 lt none gt 443 TCP 177d```

Fabio Grätz · Answer

Are these IPs all reachable only from within the VPN Otherwise maybe redact them

Stephen Fromm · Answer

```$ kubectl get service A | grep v prometheus NAMESPACE NAME TYPE CLUSTER IP EXTERNAL IP PORT S AGE actions runner system actions runner controller metrics service ClusterIP 10 100 186 11 lt none gt 8443 TCP 136d actions runner system actions runner controller webhook ClusterIP 10 100 193 93 lt none gt 443 TCP 136d cert manager cert manager ClusterIP 10 100 159 173 lt none gt 9402 TCP 136d cert manager cert manager webhook ClusterIP 10 100 183 161 lt none gt 443 TCP 136d default external dns ClusterIP 10 100 185 214 lt none gt 7979 TCP 177d default kubernetes ClusterIP 10 100 0 1 lt none gt 443 TCP 177d great falls datacatalog LoadBalancer 10 100 244 46 8089 32312 TCP 88 31369 TCP 89 32734 TCP 176d great falls flyte pod webhook ClusterIP 10 100 53 162 lt none gt 443 TCP 176d great falls flyteadmin LoadBalancer 10 100 205 238 80 31117 TCP 81 30601 TCP 87 30822 TCP 10254 31187 TCP 176d great falls flyteconsole LoadBalancer 10 100 150 24 80 30722 TCP 176d kube system aws load balancer webhook service ClusterIP 10 100 154 57 lt none gt 443 TCP 177d kube system kube dns ClusterIP 10 100 0 10 lt none gt 53 UDP 53 TCP 177d monitoring alertmanager operated ClusterIP None lt none gt 9093 TCP 9094 TCP 9094 UDP 177d```

Stephen Fromm · Answer

I assume nothing is reachable unless I m in the VPN but I don t really know

Fabio Grätz · Answer

`<http a1f320958d4844bd081a46xxxxxx485 1851160411 us east 1 elb amazonaws com console|a1f320958d4844bd081a46xxxxxx485 1851160411 us east 1 elb amazonaws com console>` is how you would try to reach the console

Stephen Fromm · Answer

No It s something like <https somethingsomething flyte great falls dev companyname com console>

Fabio Grätz · Answer

and there is a dns record somwhere configured in aws that maps this domain to `<http a1f320958d4844bd081a46e0c3fd4485 1851160411 us east 1 elb amazonaws com|a1f320958d4844bd081a46e0c3fd4485 1851160411 us east 1 elb amazonaws com> `

Fabio Grätz · Answer

`kubectl get ingress all namespaces`

Fabio Grätz · Answer

Does this give anything

Stephen Fromm · Answer

```$ kubectl get ingress all namespaces NAMESPACE NAME CLASS HOSTS ADDRESS PORTS AGE great falls flyte core lt none gt 80 176d great falls flyte core grpc lt none gt 80 176d monitoring prometheus stack grafana lt none gt 80 177d```

Fabio Grätz · Answer

Ok and `kubectl n great falls describe ingress flyte core grpc `

Stephen Fromm · Answer

Route 53 record for the seems to point at

Fabio Grätz · Answer

Makes sense

Fabio Grätz · Answer

that is the external address of the ingress

Stephen Fromm · Answer

```$ kubectl n great falls describe ingress flyte core Name flyte core Labels Namespace great falls Address Ingress Class lt none gt Default backend lt default gt Rules Host Path Backends ssl redirect use annotation lt error endpoints ssl redirect not found gt console flyteconsole 80 console flyteconsole 80 api flyteadmin 80 api flyteadmin 80 healthcheck flyteadmin 80 v1 flyteadmin 80 well known flyteadmin 80 well known flyteadmin 80 login flyteadmin 80 login flyteadmin 80 logout flyteadmin 80 logout flyteadmin 80 callback flyteadmin 80 callback flyteadmin 80 me flyteadmin 80 config flyteadmin 80 config flyteadmin 80 oauth2 flyteadmin 80 oauth2 flyteadmin 80 Annotations Type redirect RedirectConfig Protocol HTTPS Port 443 StatusCode HTTP 301 arn aws acm us east 1 11111111111111 certificate UUID ish ID flyte HTTP 80 HTTPS 443 internal service instance=production alb great falls great falls console Events lt none gt ```

Stephen Fromm · Answer

```$ kubectl n great falls describe ingress flyte core Name flyte core Labels Namespace great falls Address Ingress Class lt none gt Default backend lt default gt Rules Host Path Backends ssl redirect use annotation lt error endpoints ssl redirect not found gt console flyteconsole 80 console flyteconsole 80 api flyteadmin 80 api flyteadmin 80 healthcheck flyteadmin 80 v1 flyteadmin 80 well known flyteadmin 80 well known flyteadmin 80 login flyteadmin 80 login flyteadmin 80 logout flyteadmin 80 logout flyteadmin 80 callback flyteadmin 80 callback flyteadmin 80 me flyteadmin 80 config flyteadmin 80 config flyteadmin 80 oauth2 flyteadmin 80 oauth2 flyteadmin 80 Annotations Type redirect RedirectConfig Protocol HTTPS Port 443 StatusCode HTTP 301 arn aws acm us east 1 1111111111 certificate UUID ish ID flyte HTTP 80 HTTPS 443 internal service instance=production alb great falls great falls console Events lt none gt base ip xx xx xx xx flyte deployment sfromm$ kubectl n great falls describe ingress flyte core grpc Name flyte core grpc Labels Namespace great falls Address Ingress Class lt none gt Default backend lt default gt Rules Host Path Backends flyteidl service AdminService flyteadmin 81 flyteidl service AdminService flyteadmin 81 flyteidl service DataProxyService flyteadmin 81 flyteidl service DataProxyService flyteadmin 81 flyteidl service AuthMetadataService flyteadmin 81 flyteidl service AuthMetadataService flyteadmin 81 flyteidl service IdentityService flyteadmin 81 flyteidl service IdentityService flyteadmin 81 grpc health v1 Health flyteadmin 81 grpc health v1 Health flyteadmin 81 Annotations Type redirect RedirectConfig Protocol HTTPS Port 443 StatusCode HTTP 301 HTTP2 arn aws acm us east 1 11111111111 certificate UUID ish ID flyte HTTP 80 HTTPS 443 internal service instance=production alb great falls great falls console GRPC Events lt none gt ```

Fabio Grätz · Answer

I ve never done this with AWS but in GCP there is a page in the UI where one can see the health of the ingress

Fabio Grätz · Answer

Can you pls check whether this is the case in aws

Stephen Fromm · Answer

I ll have to look at the docs Might be away from keyboard for awhile for other reasons It d be great if you checked this thread later today but if you re too busy I understand since you re doing this for free

Fabio Grätz · Answer

I ll try to check tonight slightly smiling face 5 30 pm here

Stephen Fromm · Answer

Unfortunately I ve done some looking and cannot find anything more about ingress health

Fabio Grätz · Answer

I unfortunately also don t know where this is found in AWS The fact that `Events lt none gt ` at least on GCP would be a sign that the ingress is not doing so well But can t tell on AWS sorry

Stephen Fromm · Answer

My colleagues and I looked at it more It appears the pods were in the Ready state but the nodes were in Not Ready I would have thought that can t happen though I did find a github issue for k8s where it happens at least for older versions of k8s and we re using an outdated version So now we re cordoning and draining the nodes

Stephen Fromm · Answer

We figured out what happened In an attempt to give me access to read the cluster state using k9s kubectl etc we added an identity mapping with me as the user which shadowed the system user which basically halted all communication within the network Whoops

Fabio Grätz · Answer

Good to hear you figured it out slightly smiling face

Stephen Fromm · Answer

Sorry to have wasted so much of your time cry

Fabio Grätz · Answer

No worries slightly smiling face