https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
t

Taylor Stout

02/15/2023, 6:06 PM
We're running into 
terminated with exit code (137). Reason [OOMKilled].
pod failures when trying to run simple hello world tasks. We have flyte-binary deployed via its helm chart. I see the task pod is being spun up with a default memory limit of 200mb. I'm having trouble tracking down how to set the default pod resource spec to adjust the memory limit of the pod.
d

David Espejo (he/him)

02/15/2023, 6:34 PM
y

Yee

02/15/2023, 6:37 PM
you can use pod template yeah but i think that’s not the root cause here - how is your flyte deployment configured? you’re using the flyte-binary helm chart?
the old charts had task resources specified
but we removed that in the new chart
basically those would’ve ended up here in the admin config
but we should’ve removed that in the new chart.
you can add that back in the “inline” section of the new chart which shows up here
since for single binary we now have to specify which of the flyte components config entries are for
t

Taylor Stout

02/15/2023, 8:07 PM
Hi @Yee, yes we're using the flyte-binary chart. I tried adding the task_resources object to the inline object in the values.yaml:
Copy code
inline: 
    task_resources:
      defaults:
        cpu: 100m
        memory: 100Mi
        storage: 100Mi
      limits:
        memory: 1Gi
this appears to have set the defaults appropriate but the limit for memory doesn't appear to be set to 1Gi and instead just gets set to 100Mi to match default request.
image.png
Ideally I would be able to set lower CPU/Memory requests compared to their limits
y

Yee

02/15/2023, 9:31 PM
actually a bit confused… you didn’t have to add the 
admin:
key to the config?
t

Taylor Stout

02/15/2023, 9:31 PM
No, when I did that it didn't work. Only worked when added like I showed above
y

Yee

02/15/2023, 9:32 PM
i see
so about the other issue…
Default values get injected as the task requests and limits when a task definition omits a specific resource. Limit values are only used as validation. Neither a task request nor limit can exceed the limit for a resource type.
t

Taylor Stout

02/15/2023, 9:37 PM
AH I see.
On a side note, any idea how I can speed up task latency? After resolving the memory request issue, these very simple hellow world tasks take ~2 mins to complete. I can see the pod gets spun up and completes but Flyte appears to have a latency in completing a task built in.
y

Yee

02/15/2023, 9:46 PM
i’m gonna queue @Dan Rammer (hamersaw) on this one.
dan is there a beginner’s guide to flyte tuning somewhere?
t

Taylor Stout

02/15/2023, 9:59 PM
i've tried lowering both workflow-reeval-duration and downstream-eval-duration, but regardless a task, even run on its own, takes ~2 minutes no matter what
I'm going to spin this q up in another thread since it veered off the original ask. Thank you for the help!
d

Dan Rammer (hamersaw)

02/16/2023, 2:45 PM
@Taylor Stout can you direct me to the new thread? Would be happy to discuss!
t

Taylor Stout

02/16/2023, 3:06 PM
Sure. I just tagged you in a new thread.
29 Views
Powered by