We're running into `terminated with exit code (137...
# ask-the-communityt
Taylor Stout
02/15/2023, 6:06 PMWe're running into
terminated with exit code (137). Reason [OOMKilled].d
David Espejo (he/him)
02/15/2023, 6:34 PMHi Taylor!
You could use PodTemplate:
https://docs.flyte.org/en/latest/deployment/configuration/general.html#using-default-k8s-podtemplates
y
Yee
02/15/2023, 6:37 PM
you can use pod template yeah but i think that’s not the root cause here - how is your flyte deployment configured? you’re using the flyte-binary helm chart?
Yee
02/15/2023, 6:41 PM
the old charts had task resources specified
Yee
02/15/2023, 6:41 PM
but we removed that in the new chart
Yee
02/15/2023, 6:42 PM
basically those would’ve ended up here in the admin config
Yee
02/15/2023, 6:44 PM
but we should’ve removed that in the new chart.
Yee
02/15/2023, 6:51 PM
you can add that back in the “inline” section of the new chart which shows up here
Yee
02/15/2023, 6:52 PM
you’ll just have to namespace it under `admin:`https://github.com/flyteorg/flyte/blob/d60c9af85a59ebb4c2265f76cb082b992078a309/charts/flyte-binary/templates/configmap.yaml#L23
Yee
02/15/2023, 6:52 PM
since for single binary we now have to specify which of the flyte components config entries are for
t
Taylor Stout
02/15/2023, 8:07 PMHi @Yee, yes we're using the flyte-binary chart. I tried adding the task_resources object to the inline object in the values.yaml:
Copy code
inline:
task_resources:
defaults:
cpu: 100m
memory: 100Mi
storage: 100Mi
limits:
memory: 1GiTaylor Stout
02/15/2023, 8:07 PMthis appears to have set the defaults appropriate but the limit for memory doesn't appear to be set to 1Gi and instead just gets set to 100Mi to match default request.
Taylor Stout
02/15/2023, 8:08 PMimage.png
Taylor Stout
02/15/2023, 8:09 PMIdeally I would be able to set lower CPU/Memory requests compared to their limits
y
Yee
02/15/2023, 9:31 PM
actually a bit confused… you didn’t have to add the
admin:t
Taylor Stout
02/15/2023, 9:31 PMNo, when I did that it didn't work. Only worked when added like I showed above
y
Yee
02/15/2023, 9:32 PM
i see
Yee
02/15/2023, 9:32 PM
Yee
02/15/2023, 9:32 PM
so about the other issue…
Default values get injected as the task requests and limits when a task definition omits a specific resource. Limit values are only used as validation. Neither a task request nor limit can exceed the limit for a resource type.
t
Taylor Stout
02/15/2023, 9:37 PMAH I see.
Taylor Stout
02/15/2023, 9:39 PMOn a side note, any idea how I can speed up task latency? After resolving the memory request issue, these very simple hellow world tasks take ~2 mins to complete. I can see the pod gets spun up and completes but Flyte appears to have a latency in completing a task built in.
y
Yee
02/15/2023, 9:46 PM
i’m gonna queue @Dan Rammer (hamersaw) on this one.
Yee
02/15/2023, 9:46 PM
dan is there a beginner’s guide to flyte tuning somewhere?
t
Taylor Stout
02/15/2023, 9:59 PMi've tried lowering both workflow-reeval-duration and downstream-eval-duration, but regardless a task, even run on its own, takes ~2 minutes no matter what
Taylor Stout
02/16/2023, 2:23 PMI'm going to spin this q up in another thread since it veered off the original ask. Thank you for the help!
d
Dan Rammer (hamersaw)
02/16/2023, 2:45 PM@Taylor Stout can you direct me to the new thread? Would be happy to discuss!
t
Taylor Stout
02/16/2023, 3:06 PMSure. I just tagged you in a new thread.
229 Views