Hello, is there any method of Rate Limiting the nu...
# ask-the-communityr
Robert Everson
09/19/2022, 9:03 PMHello, is there any method of Rate Limiting the number of resources that can get created on a k8s cluster (or just the number of tasks/nodes that start up in general)? We’re hitting an issue with crio on our clusters that is causing some of the nodes to fall over, but the nodes stay up in k8s so the resources don’t get returned to the cluster
k
Ketan (kumare3)
09/19/2022, 9:07 PM
@Robert Everson yes Flyte does use resourceQuotas so you can adjust them per project
Ketan (kumare3)
09/19/2022, 9:08 PM
Also at Lyft this is controlled through a config file AFAIR
Ketan (kumare3)
09/19/2022, 9:08 PM
there is also a way to restrict using resource manager and max parallelism
r
Robert Everson
09/19/2022, 9:26 PMWhat do you mean by resource manager and max parallelism? We have resourceQuotas, but I think the issue we have is that we have a bunch of very small jobs that all start up at once and overload crio on the nodes
k
Ketan (kumare3)
09/19/2022, 9:29 PM
are they across namespaces?
r
Robert Everson
09/19/2022, 9:30 PMNo, mostly in the same namespace from what I’ve seen at least today
k
Ketan (kumare3)
09/19/2022, 9:30 PM
A quick mitigation can be to increase the requested mem/cpu etc
Ketan (kumare3)
09/19/2022, 9:30 PM
this seems to be a crio problem
r
Robert Everson
09/19/2022, 9:31 PM100%, just wasn’t sure if we could mitigate this in Flyte
k
Ketan (kumare3)
09/19/2022, 10:12 PM
hmm we can mitigate by doing 2 things as i said
1. increase resource requests
2. reduce quota (which may not be a good idea)
r
Robert Everson
09/19/2022, 10:36 PM@Eli Schachar
Robert Everson
09/19/2022, 10:36 PM@Paul Lee
Robert Everson
09/19/2022, 10:51 PM@Ketan (kumare3) do you know if there’s a way to increase the default resource request of a job? I found a way to do this for spark jobs, just wondering if there’s a way to do this for k8s jobs as well?
k
Ketan (kumare3)
09/19/2022, 11:20 PM
@Robert Everson 2 ways - https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_task-resource-attribute.html#id1 (which may not work for your older version, ensure it does)
update in flyteadmin default config - https://docs.flyte.org/en/latest/deployment/cluster_config/flyteadmin_config.html#section-task-resources
r
Robert Everson
09/19/2022, 11:32 PMthanks!
p
Paul Lee
09/21/2022, 7:00 PMhey @Ketan (kumare3) is my assumption right that if projects define something lower than the defaults we assign they will use the project resource requests rather than the default?
Paul Lee
09/21/2022, 7:01 PMthink i found my answer: https://docs.flyte.org/en/v1.0.0/deployment/cluster_config/general.html
Paul Lee
09/21/2022, 7:02 PM Copy code
In the absence of an override, the global default values in the FlyteAdmin config are used.
The override values from the database are assigned at execution, rather than registration time.
To update individual project-domain attributes, first define a tra.yaml file with your overrides:156 Views