Hello. Need some help as our production env web broke for some reason. Last I remember I tried registering a workflow which registered successfully and a few mins later when I tried running it - it shows this error. Can someone lead me to what could be the issue? I asked my other colleagues - there was no operation on the cluster during that time. We have flyte deployed on our GCP k8s cluster.

Looks like something is wrong with the auth callback. I would check flyte admin logs for errors.

I did check. It says EMPTY_AUTH_TOKEN

@great-hair-77803 can you share full logs from the flyteadmin Pod?

Hello @average-finland-92144 We were able to solve the issue after adding some annotations to nginx ingress.

What is the root cause for how it broke all of a sudden?

We found the error in our nginx-controller upstream sent too big error while reading response header from upstream We modified a few annotations on our side like below I added some more annotations into nginx-ingress. annotations: nginx.ingress.kubernetes.io/proxy-buffer-size: "256k" nginx.ingress.kubernetes.io/proxy-buffers-number: "4" nginx.ingress.kubernetes.io/proxy-buffering: "on"

Ah yeah. We have run into that as well. Must have only happened on certain pages where the error message was large?

No this blocked our login itself on all pages afair!

Strange