Hello, is there any method of Rate Limiting the number of resources that can get created on a k8s cl...

salmon-waitress-82501

09/19/2022, 9:03 PM

Hello, is there any method of Rate Limiting the number of resources that can get created on a k8s cluster (or just the number of tasks/nodes that start up in general)? We’re hitting an issue with crio on our clusters that is causing some of the nodes to fall over, but the nodes stay up in k8s so the resources don’t get returned to the cluster

freezing-airport-6809

09/19/2022, 9:07 PM

@salmon-waitress-82501 yes Flyte does use resourceQuotas so you can adjust them per project

freezing-airport-6809

09/19/2022, 9:08 PM

Also at Lyft this is controlled through a config file AFAIR

freezing-airport-6809

09/19/2022, 9:08 PM

there is also a way to restrict using resource manager and max parallelism

salmon-waitress-82501

09/19/2022, 9:26 PM

What do you mean by resource manager and max parallelism? We have resourceQuotas, but I think the issue we have is that we have a bunch of very small jobs that all start up at once and overload crio on the nodes

freezing-airport-6809

09/19/2022, 9:29 PM

are they across namespaces?

salmon-waitress-82501

09/19/2022, 9:30 PM

No, mostly in the same namespace from what I’ve seen at least today

freezing-airport-6809

09/19/2022, 9:30 PM

A quick mitigation can be to increase the requested mem/cpu etc

freezing-airport-6809

09/19/2022, 9:30 PM

this seems to be a crio problem

salmon-waitress-82501

09/19/2022, 9:31 PM

100%, just wasn’t sure if we could mitigate this in Flyte

freezing-airport-6809

09/19/2022, 10:12 PM

hmm we can mitigate by doing 2 things as i said 1. increase resource requests 2. reduce quota (which may not be a good idea)

salmon-waitress-82501

09/19/2022, 10:36 PM

@ambitious-apartment-43458

salmon-waitress-82501

09/19/2022, 10:36 PM

@witty-balloon-62264

salmon-waitress-82501

09/19/2022, 10:51 PM

@freezing-airport-6809 do you know if there’s a way to increase the default resource request of a job? I found a way to do this for spark jobs, just wondering if there’s a way to do this for k8s jobs as well?

freezing-airport-6809

09/19/2022, 11:20 PM

@salmon-waitress-82501 2 ways - https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_task-resource-attribute.html#id1 (which may not work for your older version, ensure it does) update in flyteadmin default config - https://docs.flyte.org/en/latest/deployment/cluster_config/flyteadmin_config.html#section-task-resources

👍 2

salmon-waitress-82501

09/19/2022, 11:32 PM

thanks!

witty-balloon-62264

09/21/2022, 7:00 PM

hey @freezing-airport-6809 is my assumption right that if projects define something lower than the defaults we assign they will use the project resource requests rather than the default?

witty-balloon-62264

09/21/2022, 7:01 PM

think i found my answer: https://docs.flyte.org/en/v1.0.0/deployment/cluster_config/general.html

witty-balloon-62264

09/21/2022, 7:02 PM

Copy code

In the absence of an override, the global default values in the FlyteAdmin config are used.

The override values from the database are assigned at execution, rather than registration time.

To update individual project-domain attributes, first define a tra.yaml file with your overrides:

👍 1

156 Views

Open in Slack

Previous Next

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.