Hi, I am facing a very strange issue, and I am out of ideas. We have 4 instances of Flyte, all of which are configured the same and run the latest version. Each of them is running on a different cluster, and we route traffic using Ingress Nginx Controller, which is configured in exactly the same way on all clusters. All instances use Azure AD SSO, and all use the same App Registration/credentials. However, for some reason, one of these 4 instances does not work. The issue is that when I access the URL, I get to the login page, and then successfully log in using the Azure AD SSO but after that, every request fails on 400 error

400 Bad Request
Request Header Or Cookie Too Large
nginx

I tried different browsers, incognito mode, wiping cookies, everything. This only happens on that one instance, and it works without any issues on the other 3. Any ideas?

This happens if you have JWT token in the cookie which might be large for nginx to process. If clearing data from browser for the domain does not work try incresing the header size on nginx side: Below is a medium article: https://medium.com/brain-second/when-large-client-header-buffers-and-client-header-buffer-size-arent-enough-7d5763aa99b8 Try with the below config:

client_header_buffer_size 8k;
large_client_header_buffers 4 32k;

Remeber increasing the header size increases the memory consumption

I am hesitant to do this because it does not explain why it only happens in one of 4 environments. I already checked the nginx.conf file in the ingress nginx pod on all clusters, and these 2 fields have the same value on all of them so I just can't wrap my head around that

If you read the medium article pasted, it has some reasons why this might happen. Try it once

I've read it and I understand your point, but Flyte is the only application in that cluster that has these issues, and I can't justify tweaking our ingress controller just to fix an issue in one of many apps that we host on it. Thanks anyway, it's a good solution, but unfortunately does not fit our use case.

Maybe its something related to the cluster embedded in the token. Do the clusters have names of different lengths?

@clean-glass-36808, I also thought that might be the case, but we have another instance in the same region that works, and that one uses flyte.accept.nl.cloud while the non-working one uses flyte.test.nl.cloud, so the non-working one is shorter. Even the cluster name is shorter for the non-working instance. I will try to poke around the token from one of the working instances and compare it with the non-working one, and see if I can spot something