https://flyte.org logo
#announcements
Title
# announcements
a

Alex Bain

04/04/2022, 10:02 PM
Flyte Team - we're still having difficult problems in between CORS with our gloo ingress, virtual services, etc. Is there any way we can just completely disable CORS for Flyte Admin / Flyte Console?
m

Miggy

04/04/2022, 10:05 PM
cc: @Jason Porter
a

Alex Bain

04/04/2022, 10:28 PM
The CORS errors in our Chrome browser seem to be a red herring; the issue seems to be that admin is hammering our database and throwing an error back up to Flyte Console
m

Miggy

04/04/2022, 10:29 PM
cc: @Ketan (kumare3)
a

Alex Bain

04/04/2022, 10:29 PM
@Ketan (kumare3) our RDS writer instance is taking 100% of the traffic, our reader is down to zero.
It looks like the query that is busted is queries like https://avflyteconsole.pdx.l5.woven-planet.tech/console/projects/avdataetl/workflows?domain=dev for large projects. Somehow the queries to load workflows (and the last 10 executions for each workflow) that are made from this page are being made in a way that is directing the query to our RDS writer instance only (and also perhaps not using any available indexes).
n

Nastya Rusina

04/04/2022, 10:44 PM
As a temporary workaround you can Install Chrome plugin Allow CORS: Access-Control-Allow-Origin and activate it in separate Chrome window with opened flyteconsole (toggle to on).
a

Alex Bain

04/04/2022, 10:44 PM
While our writer instance has accumulated ~500 connections it actually has only about 15% CPU usage ==> indicates most threads are blocked on an index, row or table lock.
Perhaps a change was introduced to do something like write to a "metrics" table to record Flyte usage along with every read query?
k

Ketan (kumare3)

04/05/2022, 12:08 AM
@Alex Bain did you rollback console
also @katrina has a fix to add index to the most important query
k

katrina

04/05/2022, 4:02 AM
k

Ketan (kumare3)

04/05/2022, 5:05 AM
but remember this is bleeding edge (not a backport) cc @Alex Bain. Please test before you deploy
a

Alex Bain

04/05/2022, 10:27 PM
@Ketan (kumare3) and @katrina we rolled back console this morning and our RDS went back to normal. Thanks for jumping on the call yesterday to discuss it.
Thanks for publishing admin
v0.6.128
with the index fix. We'll plan to test it soon!
k

Ketan (kumare3)

04/05/2022, 10:33 PM
awesome
6 Views