Flyte Team - we're still having difficult problems...
# announcementsa
Alex Bain
04/04/2022, 10:02 PMFlyte Team - we're still having difficult problems in between CORS with our gloo ingress, virtual services, etc. Is there any way we can just completely disable CORS for Flyte Admin / Flyte Console?
m
Miggy
04/04/2022, 10:05 PMcc: @Jason Porter
a
Alex Bain
04/04/2022, 10:28 PMThe CORS errors in our Chrome browser seem to be a red herring; the issue seems to be that admin is hammering our database and throwing an error back up to Flyte Console
m
Miggy
04/04/2022, 10:29 PMcc: @Ketan (kumare3)
a
Alex Bain
04/04/2022, 10:29 PM@Ketan (kumare3) our RDS writer instance is taking 100% of the traffic, our reader is down to zero.
Alex Bain
04/04/2022, 10:41 PMIt looks like the query that is busted is queries like https://avflyteconsole.pdx.l5.woven-planet.tech/console/projects/avdataetl/workflows?domain=dev for large projects. Somehow the queries to load workflows (and the last 10 executions for each workflow) that are made from this page are being made in a way that is directing the query to our RDS writer instance only (and also perhaps not using any available indexes).
n
Nastya Rusina
04/04/2022, 10:44 PMAs a temporary workaround you can Install Chrome plugin Allow CORS: Access-Control-Allow-Origin and activate it in separate Chrome window with opened flyteconsole (toggle to on).
a
Alex Bain
04/04/2022, 10:44 PMWhile our writer instance has accumulated ~500 connections it actually has only about 15% CPU usage ==> indicates most threads are blocked on an index, row or table lock.
Alex Bain
04/04/2022, 10:45 PMPerhaps a change was introduced to do something like write to a "metrics" table to record Flyte usage along with every read query?
k
Ketan (kumare3)
04/05/2022, 12:08 AM
@Alex Bain did you rollback console
Ketan (kumare3)
04/05/2022, 12:09 AM
also @katrina has a fix to add index to the most important query
k
katrina
04/05/2022, 4:02 AM
should be released https://github.com/flyteorg/flyteadmin/releases/tag/v0.6.128
k
Ketan (kumare3)
04/05/2022, 5:05 AM
but remember this is bleeding edge (not a backport) cc @Alex Bain. Please test before you deploy
a
Alex Bain
04/05/2022, 10:27 PM@Ketan (kumare3) and @katrina we rolled back console this morning and our RDS went back to normal. Thanks for jumping on the call yesterday to discuss it.
Alex Bain
04/05/2022, 10:29 PMThanks for publishing admin
v0.6.128k
Ketan (kumare3)
04/05/2022, 10:33 PM
awesome
166 Views