https://flyte.org
Join Slack
Hi Community, whenever I try to launch the workflow, I get an error "OOMKilled". How to solve this? ...
# flyte-support
b
Hi Community, whenever I try to launch the workflow, I get an error "OOMKilled". How to solve this? I am using demo sandbox (union sandbox). Earlier, this issue was not there. Since, I migrated from one server to another, I start facing these issues.
w
I have not encountered this error before, but I’m assuming that OOM is out of memory. Have you tried to increase the resources for the cluster?
b
How can I increase the resources?
I tried to override the resources limit through task decorator. But I get an error stating "Contact flyteadmin to increase the resources quota"
I hope there is a way to edit the resources using flytectl config?
t
b
Thanks @tall-lock-23197. I am trying this out
admin:
# For GRPC endpoints you might want to use dns:///flyte.myexample.com
endpoint: localhost:30080
authType: Pkce
insecure: true
console:
endpoint: <http://localhost:30080>
logger:
show-source: true
level: 0
task_resources:
defaults:
cpu: 1
memory: 8Gi
limits:
cpu: 4
memory: 8Gi
After starting the union sanbox I am doing "export FLYTECTL_CONFIG=/root/.flyte/config.yaml". The config-sandbox has above configuration. If I edit the config-sandbox.yaml then I get an "failed strict mode check". So, I make changes in config.yaml file and then export it after the union sandbox gets started.
Copy code
"UNKNOWN:Error received from peer ipv4:127.0.0.1:30080 {grpc_message:"Requested MEMORY default [8Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration", grpc_status:3, created_time:"2023-05-29T15:24:42.288425399+05:30"}"
I am getting the above error
@tall-lock-23197 Thanks in advanced!
t
Have you run 
flytectl demo reload
command?
b
I followed the following steps: 1. flytectl demo start 2. export FLYTECTL_CONFIG=/root/.flyte/config.yaml
I am trying "flytectl demo reload"
I get this error if I do "flytectl demo reload" after adding the task_resources in the config.yaml and doing "export FLYTECTL_CONFIG"
t
Can you try editing the config map 
kubectl -n flyte edit cm flyte-sandbox-config
?
b
Ohkay. I am a beginner at Kubernetes but I can try this. How should I edit precisely?
Oh. It's a vim editor. What are the changes I should make to the config map? I don't see a task_resources key or a way to define the default config file to be used while starting the union sandbox.
Got it. I need to add the task_resources to the configmap. Right? As described here: https://flyte-org.slack.com/archives/CP2HDHKE1/p1677672917236769?thread_ts=1677661973.390819&amp;cid=CP2HDHKE1
t
Yeah, I think so! Can you add that block and give it a try?
b
Thank you so much.
It worked.
Summary: To add resource limits, we can just add the task_resources block to the config map which can be accessed by "kubectl -n flyte edit cm flyte-sandbox-config". Reference: https://flyte-org.slack.com/archives/CP2HDHKE1/p1677672917236769?thread_ts=1677661973.390819&amp;cid=CP2HDHKE1
t
If you could add the same to our troubleshooting guide, that'd be awesome!
b
I will do that.
🙏 1
157 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by