https://flyte.org
Join Slack
Hey guys, I'm using helm flyte to launch jobs in k8s and I need to bump the resources. I have a task...
# flyte-support
s
Hey guys, I'm using helm flyte to launch jobs in k8s and I need to bump the resources. I have a task with resources specified in the headers. I bumped from 1cpu to 50cpu and 40Gi to 128Gi
Copy code
@task(
    requests=Resources(cpu="50", mem="128Gi"),
    limits=Resources(cpu="50", mem="128Gi"),
)
and when i launch it sets the cpu to 
50
as desired, but memory is always at 
38Gi
requested and 
40Gi
limit
Copy code
resources:
      limits:
        cpu: "50"
        memory: 40Gi
      requests:
        cpu: "50"
        memory: 32Gi
i already set the 
task_resource_defaults
in 
flyte-admin-base-config
which previously was not allowing me to exceed the limits
Copy code
task_resource_defaults:
    task_resources:
      limits:
        cpu: 50
        memory: 128Gi
        storage: 40Gi
        ephemeral-storage: 100Gi
and im seeing that we do not have project quotas enabled
Copy code
cluster_resource_manager:
  enabled: false
i'm quite stumped as to why it's insisting on only giving me 40Gi memory even though the task specifies more and there's not quotas applied
g
Use the flytectl tool to check for all 
tra
? https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_get_task-resource-attribute.html And delete them if any are set?
s
seems like your workflow might not be updated, you can check the cr Flyteworkflows with execution id and make sure default limits and task req. limits are updated.
s
@gentle-tomato-480 like this?
Copy code
eshan@LUSQQHDH42MHC ~ % flytectl get task-resource-attribute -p navigation -d development

Error: attribute not found
{"json":{},"level":"error","msg":"attribute not found","ts":"2025-02-27T07:51:57-08:00"}
g
Indeed. But looks like there are none, not even the default 🤔
s
is it strange that you don't specify a workflow/task name for retrieving 
task-resource-attributes
? i would imagine within a single project/environment there be many many tasks with various resource configurations attached
g
You can specify the workflow task it seems: 
flytectl get task-resource-attribute -p flytesnacks -d development core.control_flow.merge_sort.merge_sort
From the second example they give
s
oh god sorry guys
the task specified is called from a dynamic workflow which calls it from a map_task and it passes in 
.with_overrides
Copy code
map_task(
                # pass multiple inputs to flyte map_task
                functools.partial(
                    <task_name>,
                    inputs=inputs
                ),
                concurrency=processing_concurrency,
                metadata=TaskMetadata(
                    cache=True,
                    cache_version="1.0.3",
                    retries=NUM_RETRIES,
                    interruptible=False,
                ),
            )(partition=partition).with_overrides(
                requests=Resources(mem="32Gi"),
                limits=Resources(mem="40Gi"),
            )
g
heh
That 'd do it
Nice catch
s
it quite would
thanks for the help 🙏
a
Thank you @gentle-tomato-480 🙂
5 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by