https://flyte.org
Join Slack
We're running into `terminated with exit code (137). Reason [OOMKilled].` pod failures when trying ...
# flyte-support
s
We're running into 
terminated with exit code (137). Reason [OOMKilled].
pod failures when trying to run simple hello world tasks. We have flyte-binary deployed via its helm chart. I see the task pod is being spun up with a default memory limit of 200mb. I'm having trouble tracking down how to set the default pod resource spec to adjust the memory limit of the pod.
a
t
you can use pod template yeah but i think that’s not the root cause here - how is your flyte deployment configured? you’re using the flyte-binary helm chart?
the old charts had task resources specified
but we removed that in the new chart
basically those would’ve ended up here in the admin config
but we should’ve removed that in the new chart.
you can add that back in the “inline” section of the new chart which shows up here
since for single binary we now have to specify which of the flyte components config entries are for
s
Hi @thankful-minister-83577, yes we're using the flyte-binary chart. I tried adding the task_resources object to the inline object in the values.yaml:
Copy code
inline: 
    task_resources:
      defaults:
        cpu: 100m
        memory: 100Mi
        storage: 100Mi
      limits:
        memory: 1Gi
this appears to have set the defaults appropriate but the limit for memory doesn't appear to be set to 1Gi and instead just gets set to 100Mi to match default request.
message has been deleted
Ideally I would be able to set lower CPU/Memory requests compared to their limits
t
actually a bit confused… you didn’t have to add the 
admin:
key to the config?
s
No, when I did that it didn't work. Only worked when added like I showed above
t
i see
so about the other issue…
Default values get injected as the task requests and limits when a task definition omits a specific resource. Limit values are only used as validation. Neither a task request nor limit can exceed the limit for a resource type.
s
AH I see.
On a side note, any idea how I can speed up task latency? After resolving the memory request issue, these very simple hellow world tasks take ~2 mins to complete. I can see the pod gets spun up and completes but Flyte appears to have a latency in completing a task built in.
t
i’m gonna queue @hallowed-mouse-14616 on this one.
dan is there a beginner’s guide to flyte tuning somewhere?
s
i've tried lowering both workflow-reeval-duration and downstream-eval-duration, but regardless a task, even run on its own, takes ~2 minutes no matter what
I'm going to spin this q up in another thread since it veered off the original ask. Thank you for the help!
👍 1
h
@swift-coat-31700 can you direct me to the new thread? Would be happy to discuss!
s
Sure. I just tagged you in a new thread.
👍 1
294 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by