Hey guys I m using helm flyte to launch jobs in k8s and I ne Flyte #flyte-support

Hey guys, I'm using helm flyte to launch jobs in k...

strong-soccer-41351

02/27/2025, 11:42 AM

Hey guys, I'm using helm flyte to launch jobs in k8s and I need to bump the resources. I have a task with resources specified in the headers. I bumped from 1cpu to 50cpu and 40Gi to 128Gi

Copy code

@task(
    requests=Resources(cpu="50", mem="128Gi"),
    limits=Resources(cpu="50", mem="128Gi"),
)

and when i launch it sets the cpu to

as desired, but memory is always at

38Gi

requested and

40Gi

limit

Copy code

resources:
      limits:
        cpu: "50"
        memory: 40Gi
      requests:
        cpu: "50"
        memory: 32Gi

i already set the

task_resource_defaults

flyte-admin-base-config

which previously was not allowing me to exceed the limits

Copy code

task_resource_defaults:
    task_resources:
      limits:
        cpu: 50
        memory: 128Gi
        storage: 40Gi
        ephemeral-storage: 100Gi

and im seeing that we do not have project quotas enabled

Copy code

cluster_resource_manager:
  enabled: false

i'm quite stumped as to why it's insisting on only giving me 40Gi memory even though the task specifies more and there's not quotas applied

gentle-tomato-480

02/27/2025, 12:19 PM

Use the flytectl tool to check for all

tra

? https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_get_task-resource-attribute.html And delete them if any are set?

square-carpet-13590

02/27/2025, 12:19 PM

seems like your workflow might not be updated, you can check the cr Flyteworkflows with execution id and make sure default limits and task req. limits are updated.

strong-soccer-41351

02/27/2025, 3:52 PM

@gentle-tomato-480 like this?

Copy code

eshan@LUSQQHDH42MHC ~ % flytectl get task-resource-attribute -p navigation -d development

Error: attribute not found
{"json":{},"level":"error","msg":"attribute not found","ts":"2025-02-27T07:51:57-08:00"}

gentle-tomato-480

02/27/2025, 4:08 PM

Indeed. But looks like there are none, not even the default 🤔

strong-soccer-41351

02/27/2025, 4:10 PM

is it strange that you don't specify a workflow/task name for retrieving

task-resource-attributes

? i would imagine within a single project/environment there be many many tasks with various resource configurations attached

gentle-tomato-480

02/27/2025, 4:13 PM

You can specify the workflow task it seems:

flytectl get task-resource-attribute -p flytesnacks -d development core.control_flow.merge_sort.merge_sort

gentle-tomato-480

02/27/2025, 4:13 PM

From the second example they give

strong-soccer-41351

02/27/2025, 4:17 PM

oh god sorry guys

strong-soccer-41351

02/27/2025, 4:18 PM

the task specified is called from a dynamic workflow which calls it from a map_task and it passes in

.with_overrides

Copy code

map_task(
                # pass multiple inputs to flyte map_task
                functools.partial(
                    <task_name>,
                    inputs=inputs
                ),
                concurrency=processing_concurrency,
                metadata=TaskMetadata(
                    cache=True,
                    cache_version="1.0.3",
                    retries=NUM_RETRIES,
                    interruptible=False,
                ),
            )(partition=partition).with_overrides(
                requests=Resources(mem="32Gi"),
                limits=Resources(mem="40Gi"),
            )

gentle-tomato-480

02/27/2025, 4:18 PM

heh

gentle-tomato-480

02/27/2025, 4:18 PM

That 'd do it

gentle-tomato-480

02/27/2025, 4:18 PM

Nice catch

strong-soccer-41351

02/27/2025, 4:18 PM

it quite would

strong-soccer-41351

02/27/2025, 4:18 PM

thanks for the help 🙏

average-finland-92144

02/27/2025, 4:59 PM

Thank you @gentle-tomato-480 🙂

5 Views

Open in Slack

Previous Next