New RunLLM response generated: >>>*# Question* Hi!, I’m encountering an issue with memory allocation in my Flyte deployment using the Helm chart for flyte-core. Despite specifying requests and limits in the Flyte task decorator, these settings do not seem to affect the pod memory. The pods always allocate memory (and cpu) based on the default values defined in the flyte-admin-base-config under task_resource_defaults, like this: values.yaml:

task_resource_defaults:
      task_resources:
        defaults:
          cpu: 250m
          memory: 2Gi
        limits:
          cpu: 5000m
          memory: 64Gi

Here’s what I’ve done so far: • Set a low default memory request and a high memory limit in task_resource_defaults. • Observed that changing the defaults in task_resource_defaults affects the memory available to the pods. • Observed that changing the requests and limits in the task decorator (inside or outside the limits of task_resource_defaults) affects nothing. I have been checking the memory available for the pod with kubectl describe pod/node where it always shows the requests and limist as the exact same as the defaults in task_resource_defaults. Also, when the memory usage of the pod goes over the limit I get OOMKilled. For some reason the Flyte task decorator seem to be ignored, and only the defaults are applied. Could anyone help me understand what might be missing or misconfigured? Any insights or suggestions would be greatly appreciated! # Answer It seems like the issue is that the task resource settings specified in the Flyte task decorator are not being applied, and only the defaults from

task_resource_defaults

are being used. This can happen due to several reasons, including misconfiguration or the way Flyte handles resource requests and limits. Steps to Diagnose and Fix: 1. Verify Configuration Application: Ensure that the configuration changes in

values.yaml

are correctly applied to the Flyte cluster. After modifying the configuration file, apply the changes using the appropriate Helm command:

helm upgrade --install flyte-core flyteorg/flyte-core -f values.yaml

2. Check Task Resource Overrides: Ensure that the task resource settings in the task decorator are correctly specified. Here’s an example of how to set requests and limits in the task decorator:

from flytekit import task, Resources

   @task(requests=Resources(cpu="500m", mem="4Gi"), limits=Resources(cpu="1000m", mem="8Gi"))
   def my_task():
       # Task implementation
       pass

3. Inspect Pod Resource Allocation: After deploying the workflow, inspect the actual resource allocation of the task pod using `kubectl describe pod`:

kubectl describe pod <pod-name> -n <namespace>

Check the

spec.containers.resources

section to see if the requests and limits match what you specified in the task decorator. 4. FlyteAdmin Configuration: