Channels
announcements
ask-the-community
auth
conference-talks
contribute
databricks-integration
datahub-flyte
deployment
ecosystem-unionml
engineeringlabs
events
feature-discussions
flyte-bazel
flyte-build
flyte-console
flyte-deployment
flyte-documentation
flyte-github
flyte-on-gcp
flyte-ui-ux
flytekit
flytekit-java
flytelab
great-content
hacktoberfest-2022
helsing-flyte
in-flyte-conversations
integrations
introductions
jobs
konan-integration
linkedin-flyte
random
ray-integration
ray-on-flyte
release
scipy-2022-sprint
show-and-tell
sig-large-models
torch-elastic
workflow-building-ui-proj
writing-w-sfloris
Title
r
Robert Everson
09/19/2022, 9:03 PMHello, is there any method of Rate Limiting the number of resources that can get created on a k8s cluster (or just the number of tasks/nodes that start up in general)? We’re hitting an issue with crio on our clusters that is causing some of the nodes to fall over, but the nodes stay up in k8s so the resources don’t get returned to the cluster
k
Ketan (kumare3)
09/19/2022, 9:07 PM@Robert Everson yes Flyte does use resourceQuotas so you can adjust them per project
Also at Lyft this is controlled through a config file AFAIR
there is also a way to restrict using resource manager and max parallelism
r
Robert Everson
09/19/2022, 9:26 PMWhat do you mean by resource manager and max parallelism? We have resourceQuotas, but I think the issue we have is that we have a bunch of very small jobs that all start up at once and overload crio on the nodes
k
Ketan (kumare3)
09/19/2022, 9:29 PMare they across namespaces?
r
Robert Everson
09/19/2022, 9:30 PMNo, mostly in the same namespace from what I’ve seen at least today
k
Ketan (kumare3)
09/19/2022, 9:30 PMA quick mitigation can be to increase the requested mem/cpu etc
this seems to be a crio problem
r
Robert Everson
09/19/2022, 9:31 PM100%, just wasn’t sure if we could mitigate this in Flyte
k
Ketan (kumare3)
09/19/2022, 10:12 PMhmm we can mitigate by doing 2 things as i said
1. increase resource requests
2. reduce quota (which may not be a good idea)
r
Robert Everson
09/19/2022, 10:36 PM@Eli Schachar
@Paul Lee
@Ketan (kumare3) do you know if there’s a way to increase the default resource request of a job? I found a way to do this for spark jobs, just wondering if there’s a way to do this for k8s jobs as well?
k
Ketan (kumare3)
09/19/2022, 11:20 PM@Robert Everson 2 ways - https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_update_task-resource-attribute.html#id1 (which may not work for your older version, ensure it does)
update in flyteadmin default config - https://docs.flyte.org/en/latest/deployment/cluster_config/flyteadmin_config.html#section-task-resources
r
Robert Everson
09/19/2022, 11:32 PMthanks!
p
Paul Lee
09/21/2022, 7:00 PMhey @Ketan (kumare3) is my assumption right that if projects define something lower than the defaults we assign they will use the project resource requests rather than the default?
think i found my answer: https://docs.flyte.org/en/v1.0.0/deployment/cluster_config/general.html
In the absence of an override, the global default values in the FlyteAdmin config are used.
The override values from the database are assigned at execution, rather than registration time.
To update individual project-domain attributes, first define a tra.yaml file with your overrides: