@Glime, I am using

.with_overrides(requests=Resources(cpu="4000m", mem="8000Mi")

to override resources available to a task. My defaults are ``cpu="1000m", mem="1000Mi"` . The override is not working though - created pods have default resources. Here are relevant log entries which I don't understand:

{
    "exec_id": "a8gqqxhxw86r82xbr88f",
    "node": "n0",
    "ns": "forecasts-production",
    "res_ver": "141234539",
    "routine": "worker-8",
    "src": "container_helper.go:310",
    "tasktype": "python-task",
    "wf": "forecasts:production:workflows.data.sync_service_data"
},
"level": "info",
"msg": "ApplyResourceOverrides with Resources [&ResourceRequirements{Limits:ResourceList{},Requests:ResourceList{cpu: {{2 0} {<nil>} 2 DecimalSI},memory: {{4194304000 0} {<nil>}  BinarySI},},Claims:[]ResourceClaim{},}], Platform Resources [&ResourceRequirements{Limits:ResourceList{cpu: {{16 0} {<nil>} 16 DecimalSI},ephemeral-storage: {{0 0} {<nil>} 0 DecimalSI},memory: {{34359738368 0} {<nil>}  BinarySI},<http://nvidia.com/gpu|nvidia.com/gpu>: {{1 0} {<nil>} 1 DecimalSI},},Requests:ResourceList{cpu: {{1 0} {<nil>} 1 DecimalSI},ephemeral-storage: {{0 0} {<nil>} 0 DecimalSI},memory: {{1073741824 0} {<nil>} 1Gi BinarySI},<http://nvidia.com/gpu|nvidia.com/gpu>: {{0 0} {<nil>} 0 DecimalSI},},Claims:[]ResourceClaim{},}] and Container Resources [{map[cpu:{{1 0} {<nil>} 1 DecimalSI} memory:{{1073741824 0} {<nil>} 1Gi BinarySI}] map[cpu:{{1 0} {<nil>} 1 DecimalSI} memory:{{1073741824 0} {<nil>} 1Gi BinarySI}] []}] with mode [MergeExistingResources]",
"ts": "2024-04-22T12:57:21Z"
{
    "exec_id": "a8gqqxhxw86r82xbr88f",
    "node": "n0",
    "ns": "forecasts-production",
    "res_ver": "141234539",
    "routine": "worker-8",
    "src": "container_helper.go:329",
    "tasktype": "python-task",
    "wf": "forecasts:production:workflows.data.sync_service_data"
},
"level": "info",
"msg": "Adjusted container resources [{map[cpu:{{1 0} {<nil>} 1 DecimalSI} memory:{{1073741824 0} {<nil>} 1Gi BinarySI}] map[cpu:{{1 0} {<nil>} 1 DecimalSI} memory:{{1073741824 0} {<nil>} 1Gi BinarySI}] []}]",
"ts": "2024-04-22T12:57:21Z"
}

Do you have any advice?

It appears that the

.with_overrides()

method you are using to set the resource requests for your task is not effectively overriding the default resource settings as expected. The logs indicate that the

ApplyResourceOverrides

function is called, but the final adjusted container resources still reflect the default values (

cpu: 1

,

memory: 1Gi

), not the overridden values you specified (

cpu: 4000m

,

mem: 8000Mi

). This issue might be due to several reasons: 1. Platform Limits: The platform might have set limits on the maximum resources that can be requested by any task. If your override values exceed these limits, they will not be applied. 2. Configuration Precedence: There might be other configurations at the project or domain level that take precedence over the overrides specified at the task level. 3. Incorrect Usage or Bug: There could be an incorrect usage of the

.with_overrides()

method or a potential bug in how overrides are being applied or processed in the system. To troubleshoot and resolve this issue, you can try the following steps: • Check Platform Limits: Verify the maximum resource limits set at the platform, project, or domain level. You can use

flytectl

to check these configurations. • Review Task Configuration: Ensure that the task is correctly configured to accept overrides. This might involve checking the task definition and how it's integrated into the workflow. • Consult Documentation or Community: Review the Flyte documentation on resource customization and consult the Flyte community forums or GitHub issues for similar problems and solutions. If these steps do not resolve the issue, it might be beneficial to raise a ticket or issue in the Flyte community with detailed logs and configuration details for further assistance.