#ask-the-community We have been using Flyte in our production cluster for the past 6 months and now we are experiencing slowness in the system which we can see in the logs with the below messages
SLOW SQL >= 200ms
*[237.313ms] [rows:0] SELECT * FROM "node_executions" WHERE "node_executions"."execution_project" = 'samuel-predictive-handover' AND "node_executions"."execution_domain" = 'development' AND "node_executions"."execution_name" = 'a8ktq27nxlfzg2xd95w7' AND "node_executions"."node_id" = 'n3' LIMIT 1*
Are there any housekeeping instructions to delete workflows off of the S3 as well as postgres DB for workflow executions older than 30 days so that we could manage the above issue???
07/04/2023, 6:39 AM
How many namespaces do you have now?
07/04/2023, 7:38 AM
we have around 202 projects and 3 regions in flyte, dev,staging and production. which means 606 namespaces just for flyte
07/05/2023, 11:40 AM
That’s not too many, I think the cluster sync controller is serial and stores no state so might take some time to replicate
Cc @katrina / @Haytham Abuelfutuh fyi.
@Sujith Samuel please feel free to contribute
07/05/2023, 6:34 PM
oh this is useful to know re: number of distinct projects. I think syncing namespaces like Ketan mentioned might be slow but if you're not frequently updating/manipulating projects there shouldn't be concerns for cluster resource controller since it should be eventually consistent
there aren't any recommendations for deleting older executions but if you wanted to introduce a cron to run a postgres command to delete executions tables entries older than 30 days (or some configurable threshold) and add that as an optional sidecar or flyteadmin command option (like we do for syncing the cluster resource controller) that would be super appreciated!
07/06/2023, 5:45 PM
I will check this and see how I can configure the sidecar to do executions cleanup