Flyte Team - we're still having difficult problems...
# announcements
a
Flyte Team - we're still having difficult problems in between CORS with our gloo ingress, virtual services, etc. Is there any way we can just completely disable CORS for Flyte Admin / Flyte Console?
m
cc: @Jason Porter
a
The CORS errors in our Chrome browser seem to be a red herring; the issue seems to be that admin is hammering our database and throwing an error back up to Flyte Console
m
cc: @Ketan (kumare3)
a
@Ketan (kumare3) our RDS writer instance is taking 100% of the traffic, our reader is down to zero.
It looks like the query that is busted is queries like https://avflyteconsole.pdx.l5.woven-planet.tech/console/projects/avdataetl/workflows?domain=dev for large projects. Somehow the queries to load workflows (and the last 10 executions for each workflow) that are made from this page are being made in a way that is directing the query to our RDS writer instance only (and also perhaps not using any available indexes).
n
As a temporary workaround you can Install Chrome plugin Allow CORS: Access-Control-Allow-Origin and activate it in separate Chrome window with opened flyteconsole (toggle to on).
a
While our writer instance has accumulated ~500 connections it actually has only about 15% CPU usage ==> indicates most threads are blocked on an index, row or table lock.
Perhaps a change was introduced to do something like write to a "metrics" table to record Flyte usage along with every read query?
k
@Alex Bain did you rollback console
also @katrina has a fix to add index to the most important query
k
k
but remember this is bleeding edge (not a backport) cc @Alex Bain. Please test before you deploy
a
@Ketan (kumare3) and @katrina we rolled back console this morning and our RDS went back to normal. Thanks for jumping on the call yesterday to discuss it.
Thanks for publishing admin
v0.6.128
with the index fix. We'll plan to test it soon!
k
awesome
165 Views