Flyte Team - we're still having difficult problems...
# announcementsa
astonishing-lizard-78628
04/04/2022, 10:02 PMFlyte Team - we're still having difficult problems in between CORS with our gloo ingress, virtual services, etc. Is there any way we can just completely disable CORS for Flyte Admin / Flyte Console?
g
glamorous-father-92994
04/04/2022, 10:05 PMcc: @late-eye-50215
a
astonishing-lizard-78628
04/04/2022, 10:28 PMThe CORS errors in our Chrome browser seem to be a red herring; the issue seems to be that admin is hammering our database and throwing an error back up to Flyte Console
g
glamorous-father-92994
04/04/2022, 10:29 PMcc: @freezing-airport-6809
a
astonishing-lizard-78628
04/04/2022, 10:29 PM@freezing-airport-6809 our RDS writer instance is taking 100% of the traffic, our reader is down to zero.
astonishing-lizard-78628
04/04/2022, 10:41 PMIt looks like the query that is busted is queries like https://avflyteconsole.pdx.l5.woven-planet.tech/console/projects/avdataetl/workflows?domain=dev for large projects. Somehow the queries to load workflows (and the last 10 executions for each workflow) that are made from this page are being made in a way that is directing the query to our RDS writer instance only (and also perhaps not using any available indexes).
c
cold-lighter-9610
04/04/2022, 10:44 PMAs a temporary workaround you can Install Chrome plugin Allow CORS: Access-Control-Allow-Origin and activate it in separate Chrome window with opened flyteconsole (toggle to on).
a
astonishing-lizard-78628
04/04/2022, 10:44 PMWhile our writer instance has accumulated ~500 connections it actually has only about 15% CPU usage ==> indicates most threads are blocked on an index, row or table lock.
astonishing-lizard-78628
04/04/2022, 10:45 PMPerhaps a change was introduced to do something like write to a "metrics" table to record Flyte usage along with every read query?
f
freezing-airport-6809
04/05/2022, 12:08 AM
@astonishing-lizard-78628 did you rollback console
freezing-airport-6809
04/05/2022, 12:09 AM
also @acceptable-policeman-57188 has a fix to add index to the most important query
a
acceptable-policeman-57188
04/05/2022, 4:02 AM
should be released https://github.com/flyteorg/flyteadmin/releases/tag/v0.6.128
f
freezing-airport-6809
04/05/2022, 5:05 AM
but remember this is bleeding edge (not a backport) cc @astonishing-lizard-78628. Please test before you deploy
a
astonishing-lizard-78628
04/05/2022, 10:27 PM@freezing-airport-6809 and @acceptable-policeman-57188 we rolled back console this morning and our RDS went back to normal. Thanks for jumping on the call yesterday to discuss it.
astonishing-lizard-78628
04/05/2022, 10:29 PMThanks for publishing admin
v0.6.128f
freezing-airport-6809
04/05/2022, 10:33 PM
awesome
168 Views