https://flyte.org logo
#ask-the-community
Title
# ask-the-community
n

Nandakumar Raghu

01/05/2024, 12:33 PM
Hi all, Is there a way to configure resource requests and limits for
flyte-binary
pods? We are seeing the flyte-binary pods failing healthchecks sometimes causing the pod to be terminated and a new one brought up which causes some downtime. One recommendation from our platform team is to set requests and limits for memory and CPU. But I am not sure how to do that for flyte-binary. We are setting these requests and limits for tasks via
task_resources
in the values.yaml though but not sure it works for the main
flyte-binary
pods.
k

Ketan (kumare3)

01/05/2024, 3:32 PM
You can this is in the helm
n

Nandakumar Raghu

01/05/2024, 3:45 PM
Can you please point where to put the resource requests and limits in the values.yaml? I tried putting it under deployment, but it didn't show up in the terraform plan output:
Copy code
deployment:
  annotations:
    downscaler/exclude: "true"
  readinessProbe:
    tcpSocket:
      port: 8088
    initialDelaySeconds: 300
    periodSeconds: 30
  image:
    repository: "${flyte_repo}"
    tag: "${flyte_tag}"
  resources:
    requests:
      memory: 256Mi
      cpu: 100m
    limits:
      memory: 8Gi
      cpu: 500m
n

Nandakumar Raghu

01/06/2024, 12:06 PM
Hi @Samhita Alla we have already set the requests and limits for task pods via
task_resources
I want to know if this applies to the
flyte-binary
pods as well. The current problem we are facing is the
flyte-binary
pods (that host the UI) being evicted / reloaded over and over, due to which the UI is inaccessible, which our platform team feels might be due to requests / limits not being set. Hope that clarifies.
s

Samhita Alla

01/06/2024, 4:08 PM
sorry, my bad. @David Espejo (he/him) any idea why adding resources config to deployment isn't working?
d

David Espejo (he/him)

01/09/2024, 3:29 PM
@Nandakumar Raghu according to the template definition, the rendered deployment should include the resources you define under
deployment.resources
Are you using a Helm resource with Terraform?
n

Nandakumar Raghu

01/09/2024, 3:43 PM
Thanks @David Espejo (he/him), found it and were able to define resource requests and limits for flyte-binary.
d

David Espejo (he/him)

01/09/2024, 3:45 PM
Ok, but just adding the resources to that section worked? Or did you have to do something else?
n

Nandakumar Raghu

01/09/2024, 3:50 PM
Yes, we added resource limits and requests. Karpenter was evicting the flyte-binary pods as the resource requests and limits were not defined and we were using spot instances.