I'm using Flyte-demo-sandbox. I have build the workflow which training with FasterRCNN_ResNet50_FPN. I just want to testing with CPU pod first. But i catch the error on Kubenetes-dashboard that the pod which exec my wf had been killed with status ( OOMKilled), and i saw the resource of each pod is only 2Gb.I have 2 question: My first question is What is limit resource of @task, cuz i have set up @task(limits=Resources(mem="20000Mi")) ? Thus i, think that if i set up like this, the limit memory for my pod should be 20000Mi. And the second question is how to up the limit of resource of the pod.
g
gray-ocean-62145
10/26/2023, 7:35 AM
You probably need to increase the cluster resource limits, see this thread
👀 1
t
thankful-minister-83577
10/27/2023, 6:34 AM
you should also just set requests. on the backend flyte will set requests=limits. this is how most containers are run at least.
thankful-minister-83577
10/27/2023, 6:34 AM
if you want to set both you of course can.
thankful-minister-83577
10/27/2023, 6:36 AM
just keep in mind when requests are not equal to limits, k8s uses a different eviction policy for the pod.