https://flyte.org logo
#ask-the-community
Title
# ask-the-community
f

Felix Ruess

11/04/2022, 3:51 PM
Hi, I ran into the default memory limit:
Requested MEMORY default [2Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration
And I already increased the task resource limits but I still get the same error. Any pointers?
l

Louis DiNatale

11/04/2022, 4:15 PM
Im not part of the flyte team but I did run into this myself and found that the resource limit was in multiple locations.
If you are running on k8s you can look at your resourcequata
f

Felix Ruess

11/04/2022, 4:48 PM
yeah, already checked that one as well
k

Ketan (kumare3)

11/04/2022, 5:20 PM
@Louis DiNatale you are part of the Flyte team. It’s a community effort ❤️
l

Louis DiNatale

11/04/2022, 5:20 PM
be still my heart
k

Ketan (kumare3)

11/04/2022, 5:22 PM
This is bad if it’s in multiple parts
But task resources is right you can change in config restart FlyteAdmin or use flytectl to create per project
f

Felix Ruess

11/04/2022, 6:05 PM
I used flytectl to create it per project and verified that it's there in the API... no dice... or do I still need to restart flyte admin?
k

Ketan (kumare3)

11/05/2022, 1:34 AM
No need to restart admin, for flytectl. Cc @Yee can you give tips here
f

Felix Ruess

11/05/2022, 10:25 AM
flytectl get task-resource-attribute -p flytesnacks -d development
:
Copy code
{
  "project": "flytesnacks",
  "domain": "development",
  "defaults": {
    "cpu": "1",
    "memory": "2Gi"
  },
  "limits": {
    "cpu": "6",
    "memory": "10Gi"
  }
}
flytectl get cluster-resource-attribute -p flytesnacks -d development <http://cadmatch_train.wf|cadmatch_train.wf>
:
Copy code
{
  "project": "flytesnacks",
  "domain": "development",
  "workflow": "<http://cadmatch_train.wf|cadmatch_train.wf>",
  "attributes": {
    "projectQuotaCpu": "20",
    "projectQuotaMemory": "64G"
  }
}
so resource limits seem to be fine, but I still get the error that current platform limit is 1Gi
k

Ketan (kumare3)

11/05/2022, 2:39 PM
Sorry I have been busy for a couple days. Cc @Eduardo Apolinario (eapolinario) / @Yee / @katrina any tips here
f

Felix Ruess

11/05/2022, 2:53 PM
I changed the
task_resource_defaults
now and did a helm upgrade.. at least I can continue testing, but I don't get why my wf was rejected, since the resource limits for flytesnacks/development should have been fine already according to what I read back via flytectl
y

Yee

11/05/2022, 9:17 PM
when you say task resource defaults you mean this bit right?
i almost always restart actually rather than rely on the file change detection mechanism.
f

Felix Ruess

11/05/2022, 9:19 PM
well, I guess that is where the defaults end up, I overwrote them in the values for the helm chart: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L539
y

Yee

11/05/2022, 9:20 PM
got it yeah
f

Felix Ruess

11/05/2022, 9:21 PM
but since flytectl was reporting the updated values before (and I also checked via API, but that end up being the same), the new values must have been applied/available already... no?
y

Yee

11/05/2022, 9:23 PM
and what does
Copy code
flytectl get task-resource-attribute -p flytesnacks -d development
give you?
f

Felix Ruess

11/05/2022, 9:25 PM
just like I wrote above:
{"project":"flytesnacks","domain":"development","defaults":{"cpu":"1","memory":"2Gi"},"limits":{"cpu":"6","memory":"10Gi"}}
y

Yee

11/05/2022, 9:27 PM
oh sorry didn’t see that
which admin version are you on?
click on the i in the top right
f

Felix Ruess

11/05/2022, 9:36 PM
y

Yee

11/05/2022, 9:39 PM
and does
<https://flyte.corp/api/v1/project_attributes/flytesnacks?resource_type=1>
give you anything?
sorry
(edited)
f

Felix Ruess

11/05/2022, 9:42 PM
just
{"attributes":{"project":"flytesnacks","matching_attributes":{"workflow_execution_config":{"max_parallelism":25}}}}
y

Yee

11/05/2022, 9:42 PM
should not be relevant
f

Felix Ruess

11/05/2022, 9:44 PM
domain/api/v1/matchable_attributes?resource_type=0
gives me the same as flytectl get task-resource-attribute ..
y

Yee

11/05/2022, 9:45 PM
so literally there is no 1Gi limit anywhere
f

Felix Ruess

11/05/2022, 9:46 PM
yeah, the only 1Gi limit I could find was in the defaults... so it seemed like it was checking against the defaults and not the actually currently set ones... After I increased the defaults as well it worked
y

Yee

11/05/2022, 9:51 PM
okay let me dig into this more.
will need to repro and debug I think. skimming the code is not enough
f

Felix Ruess

11/05/2022, 9:52 PM
thanks!
let me know if I can help with that somehow
y

Yee

11/07/2022, 10:45 PM
not able to repro this…
i’m running flyteadmin locally, requesting 900Mi. setting admin limit to 899 fails, 900 works. defaults much lower than that
f

Felix Ruess

11/08/2022, 7:52 AM
hm... anything else I can check?
y

Yee

11/08/2022, 5:09 PM
i think we’re going to have to add debug logging, and try to understand from there
maybe we can hop on a screenshare tomorrow first though?
let me see it happening, maybe that will help
f

Felix Ruess

11/09/2022, 10:46 AM
sure, I can show you via screenshare, pm me when..
y

Yee

11/09/2022, 6:01 PM
hey
have time now?
hey @Felix Ruess - talked with katrina about this. i think you’re right, flyte admin should be picking the updates up from the project/domain setting instead of using the server default.
would you mind submitting a ticket for this?
f

Felix Ruess

11/11/2022, 10:15 AM
k

Ketan (kumare3)

11/11/2022, 4:07 PM
Is this really happening
f

Felix Ruess

11/11/2022, 5:18 PM
yeah...
108 Views