Hi, I ran into the default memory limit: `Requeste...
# ask-the-community
f
Hi, I ran into the default memory limit:
Requested MEMORY default [2Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration
And I already increased the task resource limits but I still get the same error. Any pointers?
l
Im not part of the flyte team but I did run into this myself and found that the resource limit was in multiple locations.
If you are running on k8s you can look at your resourcequata
f
yeah, already checked that one as well
k
@Louis DiNatale you are part of the Flyte team. It’s a community effort ❤️
l
be still my heart
k
This is bad if it’s in multiple parts
But task resources is right you can change in config restart FlyteAdmin or use flytectl to create per project
f
I used flytectl to create it per project and verified that it's there in the API... no dice... or do I still need to restart flyte admin?
k
No need to restart admin, for flytectl. Cc @Yee can you give tips here
f
flytectl get task-resource-attribute -p flytesnacks -d development
:
Copy code
{
  "project": "flytesnacks",
  "domain": "development",
  "defaults": {
    "cpu": "1",
    "memory": "2Gi"
  },
  "limits": {
    "cpu": "6",
    "memory": "10Gi"
  }
}
flytectl get cluster-resource-attribute -p flytesnacks -d development <http://cadmatch_train.wf|cadmatch_train.wf>
:
Copy code
{
  "project": "flytesnacks",
  "domain": "development",
  "workflow": "<http://cadmatch_train.wf|cadmatch_train.wf>",
  "attributes": {
    "projectQuotaCpu": "20",
    "projectQuotaMemory": "64G"
  }
}
so resource limits seem to be fine, but I still get the error that current platform limit is 1Gi
k
Sorry I have been busy for a couple days. Cc @Eduardo Apolinario (eapolinario) / @Yee / @katrina any tips here
f
I changed the
task_resource_defaults
now and did a helm upgrade.. at least I can continue testing, but I don't get why my wf was rejected, since the resource limits for flytesnacks/development should have been fine already according to what I read back via flytectl
y
when you say task resource defaults you mean this bit right?
i almost always restart actually rather than rely on the file change detection mechanism.
f
well, I guess that is where the defaults end up, I overwrote them in the values for the helm chart: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L539
y
got it yeah
f
but since flytectl was reporting the updated values before (and I also checked via API, but that end up being the same), the new values must have been applied/available already... no?
y
and what does
Copy code
flytectl get task-resource-attribute -p flytesnacks -d development
give you?
f
just like I wrote above:
{"project":"flytesnacks","domain":"development","defaults":{"cpu":"1","memory":"2Gi"},"limits":{"cpu":"6","memory":"10Gi"}}
y
oh sorry didn’t see that
which admin version are you on?
click on the i in the top right
f
y
and does
<https://flyte.corp/api/v1/project_attributes/flytesnacks?resource_type=1>
give you anything?
sorry
(edited)
f
just
{"attributes":{"project":"flytesnacks","matching_attributes":{"workflow_execution_config":{"max_parallelism":25}}}}
y
should not be relevant
f
domain/api/v1/matchable_attributes?resource_type=0
gives me the same as flytectl get task-resource-attribute ..
y
so literally there is no 1Gi limit anywhere
f
yeah, the only 1Gi limit I could find was in the defaults... so it seemed like it was checking against the defaults and not the actually currently set ones... After I increased the defaults as well it worked
y
okay let me dig into this more.
will need to repro and debug I think. skimming the code is not enough
f
thanks!
let me know if I can help with that somehow
y
not able to repro this…
i’m running flyteadmin locally, requesting 900Mi. setting admin limit to 899 fails, 900 works. defaults much lower than that
f
hm... anything else I can check?
y
i think we’re going to have to add debug logging, and try to understand from there
maybe we can hop on a screenshare tomorrow first though?
let me see it happening, maybe that will help
f
sure, I can show you via screenshare, pm me when..
y
hey
have time now?
hey @Felix Ruess - talked with katrina about this. i think you’re right, flyte admin should be picking the updates up from the project/domain setting instead of using the server default.
would you mind submitting a ticket for this?
f
k
Is this really happening
f
yeah...
279 Views