Hi, I ran into the default memory limit: `Requeste...
# flyte-support
q
Hi, I ran into the default memory limit:
Requested MEMORY default [2Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration
And I already increased the task resource limits but I still get the same error. Any pointers?
n
Im not part of the flyte team but I did run into this myself and found that the resource limit was in multiple locations.
If you are running on k8s you can look at your resourcequata
q
yeah, already checked that one as well
f
@nice-zebra-99977 you are part of the Flyte team. It’s a community effort ❤️
n
be still my heart
💯 1
f
This is bad if it’s in multiple parts
But task resources is right you can change in config restart FlyteAdmin or use flytectl to create per project
q
I used flytectl to create it per project and verified that it's there in the API... no dice... or do I still need to restart flyte admin?
f
No need to restart admin, for flytectl. Cc @thankful-minister-83577 can you give tips here
q
flytectl get task-resource-attribute -p flytesnacks -d development
:
Copy code
{
  "project": "flytesnacks",
  "domain": "development",
  "defaults": {
    "cpu": "1",
    "memory": "2Gi"
  },
  "limits": {
    "cpu": "6",
    "memory": "10Gi"
  }
}
flytectl get cluster-resource-attribute -p flytesnacks -d development <http://cadmatch_train.wf|cadmatch_train.wf>
:
Copy code
{
  "project": "flytesnacks",
  "domain": "development",
  "workflow": "<http://cadmatch_train.wf|cadmatch_train.wf>",
  "attributes": {
    "projectQuotaCpu": "20",
    "projectQuotaMemory": "64G"
  }
}
so resource limits seem to be fine, but I still get the error that current platform limit is 1Gi
f
Sorry I have been busy for a couple days. Cc @high-accountant-32689 / @thankful-minister-83577 / @acceptable-policeman-57188 any tips here
q
I changed the
task_resource_defaults
now and did a helm upgrade.. at least I can continue testing, but I don't get why my wf was rejected, since the resource limits for flytesnacks/development should have been fine already according to what I read back via flytectl
t
when you say task resource defaults you mean this bit right?
i almost always restart actually rather than rely on the file change detection mechanism.
q
well, I guess that is where the defaults end up, I overwrote them in the values for the helm chart: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L539
t
got it yeah
q
but since flytectl was reporting the updated values before (and I also checked via API, but that end up being the same), the new values must have been applied/available already... no?
t
and what does
Copy code
flytectl get task-resource-attribute -p flytesnacks -d development
give you?
q
just like I wrote above:
{"project":"flytesnacks","domain":"development","defaults":{"cpu":"1","memory":"2Gi"},"limits":{"cpu":"6","memory":"10Gi"}}
t
oh sorry didn’t see that
which admin version are you on?
click on the i in the top right
q
t
and does
<https://flyte.corp/api/v1/project_attributes/flytesnacks?resource_type=1>
give you anything?
sorry
(edited)
q
just
{"attributes":{"project":"flytesnacks","matching_attributes":{"workflow_execution_config":{"max_parallelism":25}}}}
t
should not be relevant
q
domain/api/v1/matchable_attributes?resource_type=0
gives me the same as flytectl get task-resource-attribute ..
t
so literally there is no 1Gi limit anywhere
q
yeah, the only 1Gi limit I could find was in the defaults... so it seemed like it was checking against the defaults and not the actually currently set ones... After I increased the defaults as well it worked
t
okay let me dig into this more.
will need to repro and debug I think. skimming the code is not enough
q
thanks!
let me know if I can help with that somehow
t
not able to repro this…
i’m running flyteadmin locally, requesting 900Mi. setting admin limit to 899 fails, 900 works. defaults much lower than that
q
hm... anything else I can check?
t
i think we’re going to have to add debug logging, and try to understand from there
maybe we can hop on a screenshare tomorrow first though?
let me see it happening, maybe that will help
q
sure, I can show you via screenshare, pm me when..
t
hey
have time now?
hey @quaint-diamond-37493 - talked with katrina about this. i think you’re right, flyte admin should be picking the updates up from the project/domain setting instead of using the server default.
would you mind submitting a ticket for this?
q
f
Is this really happening
q
yeah...
493 Views