The task resources cab be defined in a few locations in our flyte binary deployment 1. in the helm chart

task_resource_defaults = {
          limits = {
            cpu    = 16
            memory = "16Gi"
          }
          defaults = {
            cpu    = 1
            memory = "1Gi"
          }
        }
        task_resources = {
          limits = {
            cpu    = 16
            memory = "16Gi"
          }
          defaults = {
            cpu    = 1
            memory = "1Gi"
          }
        }

2. in the task decorator or ShellTask arguments

requests=Resources(cpu="4", mem="16Gi")

@task(
        requests=Resources(mem="16Gi", cpu=4), 
        limits=Resources(mem="16Gi", cpu=4)
)
def my_task

3. through

flytectl update task-resources-attribute

command. This was not set in our case

{
  "json": {},
  "level": "error",
  "msg": "attribute not found",
  "ts": "00"
}

4. Project level settings through Cluster resources

flytectl update cluster-resource-attribute.

This was not set in our case

{"json":{},"level":"error","msg":"attribute not found","ts":"00"}

5. Finally, We tested removing the limits from the deployment chart definition and keeping a default of 0.5 mem and 1 cpu. Rerunning the task we got an error of exceeding the limits but in this case it said the limit did not match the default. In the absence of a defined limit where did flyte get the limit from?

details: Requested CPU default [4] is greater than current limit set in the platform configuration [2]. Please contact Flyte Admins to change these limits or consult the configuration

Questions 1. In the absence of a set limit at the deployment, does the the default = the limit ? 2. What is the order of evaluation between flytectl task update, flytectl cluster update, deployment helm chart, task decorator override. How does flyte determine the limit and default 3. Is our helm chart configuration above correct ?

1. In the Helm chart a. At the end of the day, what you set here ends up landing in one of the base flyteadmin/propeller config sections. In this case, its task_resources b. As it's indicated here (but I haven't tried it myself),

defaults

are used in case no request or limit was specified. These are the platform-level defaults. 2. At the task level. Looking at the validation logic, what if you only specify

requests

but not a

limit

in the task decorator? It should make

request

= `limit`which is good for the K8s scheduler. There seem to be defaults set here that you effectively override with the Helm config

3. I think

task-resource-attribute

is set at the project/domain level. In terms of hierarchy it's not clear to me either from the docs:

This will take precedence over any other resource attribute defined at project domain level.

I'll need to investigate more 4. This is the proposal that updated the experience of handling overridable resources at different levels

Is our helm chart configuration above correct ?

I think you only need the latter section

task_resources = {
          limits = {
            cpu    = 16
            memory = "16Gi"
          }
          defaults = {
            cpu    = 1
            memory = "1Gi"
          }
}