Hi flyte folks was there any significant performance change

Question

Hi flyte folks was there any significant performance change in Flyte console we recently swtiched to flyte 1 2 milestone version and its correspodning flyte backend Unfortunately now its taking a lot of time display the workflow tasks as opposed to what previously and from the browser log ` ```vendor a9fbc36b js 2 GET https fizlxcwy n4 0 dn1 0 n1 limit=10000 net ERR INSUFFICIENT RESOURCES``` it might have been showing previously as well but never seem to be noticable it takes solid min or 2 to even show what tasks were executed previous version was flyteconsole `1 1 6` propeller `1 1 15` and admin on `1 1 29` any idea on what have changed

Jason Porter · Answer

Hey < Jay Ganbat> could you try pulling the latest from flyteconsole We were in the process of a refactor on those views and `v 1 4 0` includes the full refactor and should load faster than the release version I should note he refactor does add a slight increase in the initial load of the node list as we combined all the requests into one the benefit purpose of this refactor is that now the other views graph timeline are much faster as they no longer require separate requests

Jay Ganbat · Answer

hmm i see yeah its hard to test the latest version on our shared cluster and dev cluster doesnt have the above issue sweat smile but i will try to test it out

Jay Ganbat · Answer

I think it also might be flyteadmin I am seeing this logs ```2022 11 07 17 36 05 go src github com flyteorg flyteadmin pkg repositories gormimpl task execution repo go 121 SLOW SQL gt = 200ms 204 218ms rows 1 SELECT task executions id task executions created at task executions updated at task executions deleted at task executions project task executions domain task executions name task executions version task executions execution project task executions execution domain task executions execution name task executions node id task executions retry attempt task executions phase task executions phase version task executions input uri task executions closure task executions started at task executions task execution created at task executions task execution updated at task executions duration FROM task executions LEFT JOIN tasks ON task executions project = tasks project AND task executions domain = tasks domain AND task executions name = tasks name AND task executions version = tasks version INNER JOIN node executions ON task executions node id = node executions node id AND task executions execution project = node executions execution project AND task executions execution domain = node executions execution domain AND task executions execution name = node executions execution name INNER JOIN executions ON node executions execution project = executions execution project AND node executions execution domain = executions execution domain AND node executions execution name = executions execution name WHERE executions execution project = balrog AND executions execution domain = production AND executions execution name = f54bcr4c4l3v5s AND node executions node id = n0 LIMIT 10000```

Jay Ganbat · Answer

maybe how the SQL query generated got changed in recent version

Jay Ganbat · Answer

could be related to this issue Hi < katrina> do you know when did the issue described started happening in flyteadmin

katrina · Answer

hey < Jay Ganbat> I m not sure it s something I observed on our internal deployment are you running into this now

Jay Ganbat · Answer

yeah we have started noticing significant slowdown in flyteconsole to show task executions I looked through the flyteadmin and seeing bunch of these `SLOW SQL gt = 200ms` logs Previously we were on flyteadmin `1 1 29` and upgraded to `1 1 46`

katrina · Answer

oof cc < Eduardo Apolinario eapolinario > I can try taking a look at this at some point but not sure if anyone on OSS has bandwidth

Jay Ganbat · Answer

i see do you recognize this version `v1 1 29 hotfix sha256 4de1d9e93cbb9d93a659b49bf51d7d9cc0ffe8fad8408d4cc16378930bb1de4b` we were using it before but couldnt figure out where it came from its base was the `name <http cr flyte org flyteorg flyteadmin|cr flyte org flyteorg flyteadmin>` which is pointing to the flyte s image registry

katrina · Answer

not sure has to do with execution abort logic and has a fix for Dockerfile vulnerabilities maybe it was a security patch but doesn t appear to be related to any sql changes

katrina · Answer

hey < Jay Ganbat> can you share which queries exactly are slow on your side is it just the list task executions one you pasted above

Jay Ganbat · Answer

yeah pretty much all of the list execution or SELECT queries this one is for node execution ```2022 11 07 23 19 24 go src github com flyteorg flyteadmin pkg repositories gormimpl node execution repo go 47 SLOW SQL gt = 200ms 587 680ms rows 1 SELECT FROM node executions WHERE node executions execution project = pineapple AND node executions execution domain = development AND node executions execution name = fpz5pyuqbaiuuo AND node executions node id = n0 0 dn1 1 dn82 LIMIT 1```

Jay Ganbat · Answer

which is odd this query looks relatively simple

Eduardo Apolinario (eapolinario) · Answer

this is interesting Is this database instance shared in any way Can you double check that it s not under some sort of resource crunch at the time you re running those queries

Jay Ganbat · Answer

hmm there could be something wrong with the backend database thinking face i just reverted to our old version and started to seeing the slowness grimacing ill let you know if i figure it out

Eduardo Apolinario (eapolinario) · Answer

Ok let us know what you find out

Jay Ganbat · Answer

does Datacatalog has the DB stuff in it imnot sure what DB flyteadmin uses in the backend

Jay Ganbat · Answer

ok i think the issue is coming from flyteconsole I downgraded flyteconsole to `1 1 6` and no longer have the issue I think the newer flyteconsole tries to pull everything in one request so if the workflow has a lot of dynamic tasks it slows down considerably i observed this before when trying to fetch execution using FlyteRemote and using sync operation

Eduardo Apolinario (eapolinario) · Answer

Cool thanks for your patience < Jay Ganbat> Let me talk to the team about this

Jay Ganbat · Answer

thanks i think it started with v1 2 6 i tested v1 2 5 and that doesnt have the problem i think below PR might have introduced the issue looking at the description of the PR <https github com flyteorg flyteconsole commit 9b10e5f58363e3ece29748677c65bfab7f7812ed>

Jay Ganbat · Answer

created an issue <https github com flyteorg flyteconsole issues 638>

Eduardo Apolinario (eapolinario) · Answer

thank you < Jay Ganbat> First of all sorry for the bad experience The team had a meeting this morning and we ll prioritize the lazy loading of the nodes going forward The work to enable this is slated to happen early next week I ll ping here once that lands

Jay Ganbat · Answer

whoohoo tada thank you for the quick turn around btw was my guess correct sweat smile in the meantime we could use 1 25 for now

Eduardo Apolinario (eapolinario) · Answer

yes you re absolutely on point

Eduardo Apolinario (eapolinario) · Answer

thank you for understanding We ll get this fixed as soon as possible

Jay Ganbat · Answer

awesome thank you very much pray

Jay Ganbat · Answer

btw would you recommend we use 1 25 or should we revert back to 1 1 6 version

Eduardo Apolinario (eapolinario) · Answer

Your pick Looking at <https github com flyteorg flyteconsole compare v1 1 6 v1 2 5> you will probably not see a huge difference So how about you give 1 2 5 a spin

Jay Ganbat · Answer

cool will do