Hi, I ran into the default memory limit:

Requested MEMORY default [2Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration

And I already increased the task resource limits but I still get the same error. Any pointers?

Im not part of the flyte team but I did run into this myself and found that the resource limit was in multiple locations.

yeah, already checked that one as well

@Louis DiNatale you are part of the Flyte team. It’s a community effort ❤️

be still my heart

This is bad if it’s in multiple parts

I used flytectl to create it per project and verified that it's there in the API... no dice... or do I still need to restart flyte admin?

No need to restart admin, for flytectl. Cc @Yee can you give tips here

flytectl get task-resource-attribute -p flytesnacks -d development

:

{
  "project": "flytesnacks",
  "domain": "development",
  "defaults": {
    "cpu": "1",
    "memory": "2Gi"
  },
  "limits": {
    "cpu": "6",
    "memory": "10Gi"
  }
}

flytectl get cluster-resource-attribute -p flytesnacks -d development <http://cadmatch_train.wf|cadmatch_train.wf>

:

{
  "project": "flytesnacks",
  "domain": "development",
  "workflow": "<http://cadmatch_train.wf|cadmatch_train.wf>",
  "attributes": {
    "projectQuotaCpu": "20",
    "projectQuotaMemory": "64G"
  }
}

so resource limits seem to be fine, but I still get the error that current platform limit is 1Gi

Sorry I have been busy for a couple days. Cc @Eduardo Apolinario (eapolinario) / @Yee / @katrina any tips here

I changed the

task_resource_defaults

now and did a helm upgrade.. at least I can continue testing, but I don't get why my wf was rejected, since the resource limits for flytesnacks/development should have been fine already according to what I read back via flytectl

when you say task resource defaults you mean this bit right?

well, I guess that is where the defaults end up, I overwrote them in the values for the helm chart: https://github.com/flyteorg/flyte/blob/master/charts/flyte-core/values.yaml#L539

got it yeah

but since flytectl was reporting the updated values before (and I also checked via API, but that end up being the same), the new values must have been applied/available already... no?

and what does

flytectl get task-resource-attribute -p flytesnacks -d development

give you?

just like I wrote above:

{"project":"flytesnacks","domain":"development","defaults":{"cpu":"1","memory":"2Gi"},"limits":{"cpu":"6","memory":"10Gi"}}

oh sorry didn’t see that

and does

<https://flyte.corp/api/v1/project_attributes/flytesnacks?resource_type=1>

give you anything?

just

{"attributes":{"project":"flytesnacks","matching_attributes":{"workflow_execution_config":{"max_parallelism":25}}}}

should not be relevant

domain/api/v1/matchable_attributes?resource_type=0

gives me the same as flytectl get task-resource-attribute ..

so literally there is no 1Gi limit anywhere

yeah, the only 1Gi limit I could find was in the defaults... so it seemed like it was checking against the defaults and not the actually currently set ones... After I increased the defaults as well it worked

okay let me dig into this more.

thanks!

not able to repro this…

hm... anything else I can check?

i think we’re going to have to add debug logging, and try to understand from there

sure, I can show you via screenshare, pm me when..

hey

sure: https://github.com/flyteorg/flyte/issues/3065

Is this really happening

yeah...