Hello. Need some help as our production env web br...
# flyte-support
g
Hello. Need some help as our production env web broke for some reason. Last I remember I tried registering a workflow which registered successfully and a few mins later when I tried running it - it shows this error. Can someone lead me to what could be the issue? I asked my other colleagues - there was no operation on the cluster during that time. We have flyte deployed on our GCP k8s cluster.
Also, the launchplans and workflows are running fine. I can see the tasks running fine.
c
Looks like something is wrong with the auth callback. I would check flyte admin logs for errors.
g
I did check. It says EMPTY_AUTH_TOKEN
and that the flyte_idt names cookie is missing
a
@great-hair-77803 can you share full logs from the flyteadmin Pod?
how's auth configured? is it internal or external auth server?
g
Hello @average-finland-92144 We were able to solve the issue after adding some annotations to nginx ingress.
c
What is the root cause for how it broke all of a sudden?
g
We found the error in our nginx-controller upstream sent too big error while reading response header from upstream We modified a few annotations on our side like below I added some more annotations into nginx-ingress. annotations: nginx.ingress.kubernetes.io/proxy-buffer-size: "256k" nginx.ingress.kubernetes.io/proxy-buffers-number: "4" nginx.ingress.kubernetes.io/proxy-buffering: "on"
c
Ah yeah. We have run into that as well. Must have only happened on certain pages where the error message was large?
g
No this blocked our login itself on all pages afair!
c
Strange