Dear all, I am having some troubles configuring the project resource quotas in my deployment. I am using the flyte-binary Helm chart on Kubernetes (on-premise) and keep getting the error message below when trying to register my workflow. I tried several things none of which changed the error I am getting. I also confirmed that the Helm deployment is actually updated and I can see changes in the resources created (helm get ...) 1. Add the following to helm values.yaml

configuration:

inline:

task_resource_defaults:

task_resources:

defaults:

cpu: 500m

memory: 500Mi

gpu: 0

limits:

cpu: 8

memory: 32Gi

gpu: 1

2. Add the following to helm values.yaml

configuration:

inline:

cluster_resources:

customData:

- production:

- projectQuotaCpu:

value: "5"

- projectQuotaMemory:

value: "4000Mi"

- staging:

- projectQuotaCpu:

value: "2"

- projectQuotaMemory:

value: "3000Mi"

- development:

- projectQuotaCpu:

value: "48"

- projectQuotaMemory:

value: "100Gi"

3. Update task-resource-attributes through flytectl as described here and here. Any ideas on this? Would be greatly appreciated.

which chart are you using?

Thanks for your quick answer, Felix. I am using the flyte-binary Chart. I'm not sure if your suggestion also applies to this chart. But I'll give it a try now.

Hey, flyteadmin has a config that applies default resources to any task, and the upper limit that any task can use. This is in place to avoid setting limits higher than the nodes available in the Kubernetes Cluster, thus making the task impossible to schedule. Also these limits apply when registering the tasks on flyteadmin, before interacting with the actual kubernetes clusters or anything. It should be configurable from the helm charts. Here is a reference I found https://github.com/flyteorg/flyte/blob/6fd4ddb5354314038a85ddb32375da1aa077bd3b/rsts/community/troubleshoot.rst#terminated-with-[…]-reason-oomkilled (ignore the headline here) For EKS you could add the config here: https://github.com/flyteorg/flyte/blob/d60c9af85a59ebb4c2265f76cb082b992078a309/charts/flyte-binary/eks-production.yaml#L30 something like:

inline:
  task_resources:
    limits:
      memory: 10Gi

Thanks also for your suggestion, Babis. I am trying this out now. @Felix Ruess I tried out `configmap.task_resource_defaults`instead of `configuration.inline.task_resource_defaults`but that didn't solve the problem for me. Also, in the readme there is no key `configmap`mentioned. So, this does not seem to apply to the flyte-binary Helm Chart.

Thank you @Babis Kiosidis and @Felix Ruess!

you are welcome 👍