Hello, is there any method of Rate Limiting the nu...
# flyte-supports
salmon-waitress-82501
09/19/2022, 9:03 PMHello, is there any method of Rate Limiting the number of resources that can get created on a k8s cluster (or just the number of tasks/nodes that start up in general)? We’re hitting an issue with crio on our clusters that is causing some of the nodes to fall over, but the nodes stay up in k8s so the resources don’t get returned to the cluster
f
freezing-airport-6809
09/19/2022, 9:07 PM
@salmon-waitress-82501 yes Flyte does use resourceQuotas so you can adjust them per project
freezing-airport-6809
09/19/2022, 9:08 PM
Also at Lyft this is controlled through a config file AFAIR
freezing-airport-6809
09/19/2022, 9:08 PM
there is also a way to restrict using resource manager and max parallelism
s
salmon-waitress-82501
09/19/2022, 9:26 PMWhat do you mean by resource manager and max parallelism? We have resourceQuotas, but I think the issue we have is that we have a bunch of very small jobs that all start up at once and overload crio on the nodes
f
freezing-airport-6809
09/19/2022, 9:29 PM
are they across namespaces?
s
salmon-waitress-82501
09/19/2022, 9:30 PMNo, mostly in the same namespace from what I’ve seen at least today
f
freezing-airport-6809
09/19/2022, 9:30 PM
A quick mitigation can be to increase the requested mem/cpu etc
freezing-airport-6809
09/19/2022, 9:30 PM
this seems to be a crio problem
s
salmon-waitress-82501
09/19/2022, 9:31 PM100%, just wasn’t sure if we could mitigate this in Flyte
f
freezing-airport-6809
09/19/2022, 10:12 PM
hmm we can mitigate by doing 2 things as i said
1. increase resource requests
2. reduce quota (which may not be a good idea)
s
salmon-waitress-82501
09/19/2022, 10:36 PM@ambitious-apartment-43458
salmon-waitress-82501
09/19/2022, 10:36 PM@witty-balloon-62264
salmon-waitress-82501
09/19/2022, 10:51 PM@freezing-airport-6809 do you know if there’s a way to increase the default resource request of a job? I found a way to do this for spark jobs, just wondering if there’s a way to do this for k8s jobs as well?
f
freezing-airport-6809
09/19/2022, 11:20 PM
@salmon-waitress-82501 2 ways - https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_task-resource-attribute.html#id1 (which may not work for your older version, ensure it does)
update in flyteadmin default config - https://docs.flyte.org/en/latest/deployment/cluster_config/flyteadmin_config.html#section-task-resources
👍 2
s
salmon-waitress-82501
09/19/2022, 11:32 PMthanks!
w
witty-balloon-62264
09/21/2022, 7:00 PMhey @freezing-airport-6809 is my assumption right that if projects define something lower than the defaults we assign they will use the project resource requests rather than the default?
witty-balloon-62264
09/21/2022, 7:01 PMthink i found my answer: https://docs.flyte.org/en/v1.0.0/deployment/cluster_config/general.html
witty-balloon-62264
09/21/2022, 7:02 PM Copy code
In the absence of an override, the global default values in the FlyteAdmin config are used.
The override values from the database are assigned at execution, rather than registration time.
To update individual project-domain attributes, first define a tra.yaml file with your overrides:👍 1
157 Views