Hey folks anyone know how to override task default and limit

Question

Hey folks anyone know how to override task default and limit resources using the `flyte binary` helm chart with `eks starter yaml`

Reda Oulbacha · Answer

I am getting this error pretty much ``` Requested MEMORY limit 2Gi is greater than current limit set in the platform configuration 1Gi Please contact Flyte Admins to change these limits or consult the configuration ```

Ketan (kumare3) · Answer

The defaults are in backed config and in the helm chart But you can also update it for a project using flytectl <https docs flyte org projects flytectl en latest gen flytectl update task resource attribute html|https docs flyte org projects flytectl en latest gen flytectl update task resource attribute html>

Reda Oulbacha · Answer

I think there is a bug with that <https github com flyteorg flyte issues 3065>

Reda Oulbacha · Answer

I ran it but the limit still shows at 1Gi

Reda Oulbacha · Answer

```flytectl get task resource attribute p flytesnacks d development project flytesnacks domain development defaults cpu 2 memory 2Gi limits cpu 4 memory 8Gi ```

Reda Oulbacha · Answer

Then using `pyflyte run remote `

Reda Oulbacha · Answer

`debug error string = UNKNOWN Error received from peer ipv6 5B 1 5D 8089 grpc message Requested MEMORY limit 2Gi is greater than current limit set in the platform configuration 1Gi Please contact Flyte Admins to change these limits or consult the configuration grpc status 3 created time 2023 02 24T15 51 00 19866+01 00 `

Ketan (kumare3) · Answer

Please delete the default

Reda Oulbacha · Answer

```flytectl get task resource attribute p flytesnacks d development project flytesnacks domain development limits cpu 4 memory 8Gi ```

Reda Oulbacha · Answer

Still getting the same error as before

Reda Oulbacha · Answer

looks like project domain resources don t overwrite the default that lives somewhere I can t find

Ketan (kumare3) · Answer

Did you restart the backed pod

Reda Oulbacha · Answer

I can try but is that just a workaround

Reda Oulbacha · Answer

Yep I did it isn t working

Reda Oulbacha · Answer

Please let me know if there is a way to control the global default limits from the flyte binary chart

Ketan (kumare3) · Answer

I meant in the Flyte binary helm chart

Reda Oulbacha · Answer

This is what I used ```kubectl n flyte scale deployment flyte backend flyte binary replicas=0 Wait 1 minute kubectl n flyte scale deployment flyte backend flyte binary replicas=1```

Reda Oulbacha · Answer

I don t think `flytectl` changes anything in the helm chart itself

Ketan (kumare3) · Answer

You are right give me a moment I am trying to find The problem is it is not shown in th default values It is absolutely possible but I want to show you an example

Reda Oulbacha · Answer

Yes please anything that would work

Ketan (kumare3) · Answer

This is what you need set <https docs flyte org en latest deployment configuration generated flyteadmin config html|https docs flyte org en latest deployment configuration generated flyteadmin config html>

Ketan (kumare3) · Answer

Here is how it is set in the non binary helm chart <https github com flyteorg flyte blob f94f8830815318af907874a9cab1bf54f371f2be charts flyte core values sandbox yaml L54|https github com flyteorg flyte blob f94f8830815318af907874a9cab1bf54f371f2be charts flyte core values sandbox yaml L54>

Reda Oulbacha · Answer

Is there a way for me to change it for the flyte binary chart

Ketan (kumare3) · Answer

There is since I have not played much with the binary hom chart bear with me

Reda Oulbacha · Answer

absolutely no worries I appreciate you working with me on this either way

Ketan (kumare3) · Answer

I think it should be under here Cc < Yee> < jeev>

Ketan (kumare3) · Answer

No we should I actually wanted no defaults by default grinning

Reda Oulbacha · Answer

Yeah default limits are limiting

Ketan (kumare3) · Answer

Now you got it stuck out tongue closed eyes

Ketan (kumare3) · Answer

Jokes aside sorry for the trouble

Reda Oulbacha · Answer

no worries

Reda Oulbacha · Answer

So would that look something like this ```configuration task resource defaults task resources defaults cpu 1 memory 2Gi limits cpu 4 memory 8Gi```

Ketan (kumare3) · Answer

I think so not on my computer else would have tried

Reda Oulbacha · Answer

ok one sec I can try now

Ketan (kumare3) · Answer

It s still early for me have not had my morning coffee

Ketan (kumare3) · Answer

Go for it

Ketan (kumare3) · Answer

If not then someone better will help in a bit

Reda Oulbacha · Answer

no worries at all I can still test it out and let you know then find out later just testing things out not in a hurry

Ketan (kumare3) · Answer

This is where the defaults are coming from <https github com flyteorg flyteadmin blob fb003b72493cec3ca3e541c5e9356c6cba94c834 pkg runtime task resource provider go L11|https github com flyteorg flyteadmin blob fb003b72493cec3ca3e541c5e9356c6cba94c834 pkg runtime task resource provider go L11>

jeev · Answer

hi < Reda Oulbacha>

jeev · Answer

it should be under configuration inline

Reda Oulbacha · Answer

sweet I m about to try trying to fix a few issues along the way

jeev · Answer

now that im at a computer ```configuration inline task resource defaults task resources defaults cpu 1 memory 2Gi limits cpu 4 memory 8Gi```

Reda Oulbacha · Answer

yep I have the same thing here ran a `helm upgrade flyte backend ` but the changes weren t reflected even though the deployment restarted

jeev · Answer

you still have the same error

jeev · Answer

ah my bad

Reda Oulbacha · Answer

so I m looking at the logs of the service and I see this

Reda Oulbacha · Answer

```2023 02 24 16 39 13 go pkg mod gorm io driver postgres v1 2 3 migrator go 106 0 495ms rows 1 SELECT count FROM pg indexes WHERE tablename = artifacts AND indexname = artifacts dataset uuid idx AND schemaname = CURRENT SCHEMA metrics prefix flyte certDir var run flyte certs localCert true listenPort 9443 serviceName flyte backend flyte binary webhook servicePort 443 secretName flyte backend flyte binary webhook secret secretManagerType K8s awsSecretManager sidecarImage <http docker io amazon aws secrets manager secret sidecar v0 1 4|docker io amazon aws secrets manager secret sidecar v0 1 4> resources limits cpu 200m memory 500Mi requests cpu 200m memory 500Mi vaultSecretManager role flyte kvVersion 2 ```

jeev · Answer

```configuration inline task resources defaults cpu 1 memory 2Gi limits cpu 4 memory 8Gi```

jeev · Answer

try that

Reda Oulbacha · Answer

oh ok one sec

jeev · Answer

guilty of copy pasta slightly smiling face

Reda Oulbacha · Answer

```configuration inline task resources defaults cpu 1 memory 2Gi limits cpu 4 memory 8Gi```

Reda Oulbacha · Answer

I have that but

Reda Oulbacha · Answer

one moment

jeev · Answer

<https flyte org slack com archives CP2HDHKE1 p1674749335646479 thread ts=1674675922 181559 amp cid=CP2HDHKE1> was the example that worked in the past

Reda Oulbacha · Answer

It looks like it worked

Reda Oulbacha · Answer

When I inspect the pod I see ``` Limits cpu 1 memory 2Gi Requests cpu 1 memory 2Gi```

Reda Oulbacha · Answer

thank you

jeev · Answer

+1

Reda Oulbacha · Answer

I tried to run the hello world workflow after deploying `flyte binary` as per the doc and ran into `OOMKilled` then tried to increase the resources with the ` task` decorator and then ran into this resource limit

Reda Oulbacha · Answer

I feel like everyone trying to go the same path as me will end up here

jeev · Answer

indeed i think the fix here is to not have flyte inject a default value in code if not specified in the configuration

Reda Oulbacha · Answer

Yes that would be amazing

Ketan (kumare3) · Answer

gt Yes this was indeed the idea cc < Yee> < Eduardo Apolinario eapolinario > we should do this

Yee · Answer

yeah this was the idea we will look into this

Ketan (kumare3) · Answer

< Yee> currently flyteadmin has some hardcoded defaults

Yee · Answer

Yee · Answer

< Ketan kumare3 > can you take a look

Yee · Answer

the only thing is that when you launch a pod manually via kubectl and you don t pass resources you get `resources `

Yee · Answer

I m only like 99 sure that 0s are equivalent to ` `

Brian Tang · Answer

< jeev> lt this no longer works for me i had this in a shell script whenever i startup a flytectl demo sandbox and i m now getting ```Error from server NotFound configmaps sandbox flyte binary config not found``` and when checking ``` flyte briant Ashriels MacBook Pro flyte test k get cm n flyte NAME DATA AGE flyte sandbox cluster resource templates 1 16m flyte sandbox config 5 16m flyte sandbox docker registry config 1 16m flyte sandbox extra cluster resource templates 0 16m flyte sandbox extra config 0 16m flyte sandbox proxy config 1 16m kubernetes dashboard settings 0 16m kube root ca crt 1 16m flyte briant Ashriels MacBook Pro flyte test k get deployment n flyte NAME READY UP TO DATE AVAILABLE AGE flyte sandbox proxy 1 1 1 1 17m flyte sandbox kubernetes dashboard 1 1 1 1 17m flyte sandbox docker registry 1 1 1 1 17m flyte sandbox minio 1 1 1 1 17m flyte sandbox 1 1 1 1 17m```

Brian Tang · Answer

i suppose once <https github com flyteorg flyteadmin pull 530|this> gets merged and released we wouldn t have to manually increase the resource limits anymore

jeev · Answer

right it s just a name change < Brian Tang> you want flyte sandbox config that s the new sandbox flyte binary config but also if you are using the latest sandbox and flytectl there is a much easier way now to do this you can keep a config file at ~ flyte sandbox config yaml that will get loaded at sandbox initialization basically move the contents of 100 overrides yaml from the snippet above into ~ flyte sandbox config yaml and do flytectl demo reload once if your sandbox is already up or do flytectl demo start