#4538 [Core feature] Allow setting of limit for how many executions the catch up system can launch Issue created by rxraghu ### Motivation: Why do you think this is important? If for some reason Flyte is down and the components are not able to communicate with each other to launch executions, then executions get backed up in the pipeline and when the system comes back online, then the catchup system executes all the pending requests. This may undue stress on different components like the database or API because usually these systems have a limit of how many requests they can serve at a time. This caused an incident in our production system where there was an issue with kubernetes and the executions created by our launch plan were not being executed, and when the system became healthy, all the executions were launched at once which brought down the database instance. We had to restart the database to make it drop the excess requests. ### Goal: What should the final outcome look like, ideally? Ideally if the number of executions that can be launched by the catchup system, then we will be able to configure this based on available resources so that it does not bring the other systems down. Also, if we can specify if the executions from the beginning of the queue (earliest) should be executed or from the end of the queue(latest) that would be great, but this is a nice to have. ### Describe alternatives you've considered We tried archiving the LP when the system was apparently down using flytectl, but it did not work, i.e the LP did not stop creating executions. We archived the LP after the system came back online, but by then hundreds of executions were already created. ### Propose: Link/Inline OR Additional context Discussion on slack - https://flyte-org.slack.com/archives/CP2HDHKE1/p1701084955940149 ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte