https://flyte.org logo
#ask-the-community
Title
# ask-the-community
l

Lukas Bommes

05/17/2023, 10:55 AM
Dear all, I am having some troubles configuring the project resource quotas in my deployment. I am using the flyte-binary Helm chart on Kubernetes (on-premise) and keep getting the error message below when trying to register my workflow. I tried several things none of which changed the error I am getting. I also confirmed that the Helm deployment is actually updated and I can see changes in the resources created (helm get ...) 1. Add the following to helm values.yaml
configuration:
inline:
task_resource_defaults:
task_resources:
defaults:
cpu: 500m
memory: 500Mi
gpu: 0
limits:
cpu: 8
memory: 32Gi
gpu: 1
2. Add the following to helm values.yaml
configuration:
inline:
cluster_resources:
customData:
- production:
- projectQuotaCpu:
value: "5"
- projectQuotaMemory:
value: "4000Mi"
- staging:
- projectQuotaCpu:
value: "2"
- projectQuotaMemory:
value: "3000Mi"
- development:
- projectQuotaCpu:
value: "48"
- projectQuotaMemory:
value: "100Gi"
3. Update task-resource-attributes through flytectl as described here and here. Any ideas on this? Would be greatly appreciated.
f

Felix Ruess

05/17/2023, 10:58 AM
which chart are you using?
if you are using flyte-core, then the values are not correct.
it should be
configmap.task_resource_defaults
not
configuration.inline.task_resource_defaults
check the contents of the
task_resources_defaults.yaml
in flyte-admin or so configmap in your cluster...
l

Lukas Bommes

05/17/2023, 11:23 AM
Thanks for your quick answer, Felix. I am using the flyte-binary Chart. I'm not sure if your suggestion also applies to this chart. But I'll give it a try now.
b

Babis Kiosidis

05/17/2023, 11:27 AM
Hey, flyteadmin has a config that applies default resources to any task, and the upper limit that any task can use. This is in place to avoid setting limits higher than the nodes available in the Kubernetes Cluster, thus making the task impossible to schedule. Also these limits apply when registering the tasks on flyteadmin, before interacting with the actual kubernetes clusters or anything. It should be configurable from the helm charts. Here is a reference I found https://github.com/flyteorg/flyte/blob/6fd4ddb5354314038a85ddb32375da1aa077bd3b/rsts/community/troubleshoot.rst#terminated-with-[…]-reason-oomkilled (ignore the headline here) For EKS you could add the config here: https://github.com/flyteorg/flyte/blob/d60c9af85a59ebb4c2265f76cb082b992078a309/charts/flyte-binary/eks-production.yaml#L30 something like:
Copy code
inline:
  task_resources:
    limits:
      memory: 10Gi
projectQuotas are applied on the project/domain as a total. It's a metric used during runtime. If for example multiple launch plans are running in a specific namespace at one time, they will all be competing for the available resources defined by the project quotas. This might not be a problem if the values are high, but it's best to configure them higher than the task limits
l

Lukas Bommes

05/17/2023, 11:40 AM
Thanks also for your suggestion, Babis. I am trying this out now. @Felix Ruess I tried out `configmap.task_resource_defaults`instead of `configuration.inline.task_resource_defaults`but that didn't solve the problem for me. Also, in the readme there is no key `configmap`mentioned. So, this does not seem to apply to the flyte-binary Helm Chart.
@Babis Kiosidis I could get it to work now using your suggested syntax.
configuration:
inline:
task_resources:
defaults:
cpu: 500m
memory: 500Mi
gpu: 0
limits:
cpu: 16
memory: 32Gi
gpu: 1
Thank you so much for your help!
d

David Espejo (he/him)

05/17/2023, 5:08 PM
Thank you @Babis Kiosidis and @Felix Ruess!
b

Babis Kiosidis

05/17/2023, 8:52 PM
you are welcome 👍
74 Views