Hi Community, whenever I try to launch the workflo...
# ask-the-communitya
Akshat D.
05/29/2023, 9:15 AMHi Community, whenever I try to launch the workflow, I get an error "OOMKilled". How to solve this? I am using demo sandbox (union sandbox). Earlier, this issue was not there. Since, I migrated from one server to another, I start facing these issues.
e
Erik Dao
05/29/2023, 9:20 AMI have not encountered this error before, but I’m assuming that OOM is out of memory. Have you tried to increase the resources for the cluster?
a
Akshat D.
05/29/2023, 9:24 AMHow can I increase the resources?
Akshat D.
05/29/2023, 9:25 AMI tried to override the resources limit through task decorator. But I get an error stating "Contact flyteadmin to increase the resources quota"
Akshat D.
05/29/2023, 9:25 AMI hope there is a way to edit the resources using flytectl config?
s
Samhita Alla
05/29/2023, 9:36 AM
@Akshat D., check this out: https://flyte-org.slack.com/archives/CP2HDHKE1/p1679097188086029?thread_ts=1678826532.980439&cid=CP2HDHKE1
a
Akshat D.
05/29/2023, 9:41 AMThanks @Samhita Alla. I am trying this out
Akshat D.
05/29/2023, 9:55 AMadmin:# For GRPC endpoints you might want to use dns:///flyte.myexample.comendpoint: localhost:30080authType: Pkceinsecure: trueconsole:endpoint: <http://localhost:30080>logger:show-source: truelevel: 0task_resources:defaults:cpu: 1memory: 8Gilimits:cpu: 4memory: 8GiAkshat D.
05/29/2023, 9:56 AMAfter starting the union sanbox I am doing "export FLYTECTL_CONFIG=/root/.flyte/config.yaml". The config-sandbox has above configuration. If I edit the config-sandbox.yaml then I get an "failed strict mode check". So, I make changes in config.yaml file and then export it after the union sandbox gets started.
Akshat D.
05/29/2023, 9:57 AM Copy code
"UNKNOWN:Error received from peer ipv4:127.0.0.1:30080 {grpc_message:"Requested MEMORY default [8Gi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration", grpc_status:3, created_time:"2023-05-29T15:24:42.288425399+05:30"}"Akshat D.
05/29/2023, 9:57 AMI am getting the above error
Akshat D.
05/29/2023, 9:57 AM@Samhita Alla Thanks in advanced!
s
Samhita Alla
05/29/2023, 10:30 AM
Have you run
flytectl demo reloada
Akshat D.
05/29/2023, 10:33 AMI followed the following steps:
1. flytectl demo start
2. export FLYTECTL_CONFIG=/root/.flyte/config.yaml
Akshat D.
05/29/2023, 10:34 AMI am trying "flytectl demo reload"
Akshat D.
05/29/2023, 10:38 AMI get this error if I do "flytectl demo reload" after adding the task_resources in the config.yaml and doing "export FLYTECTL_CONFIG"
s
Samhita Alla
05/29/2023, 10:45 AM
Can you try editing the config map
kubectl -n flyte edit cm flyte-sandbox-configa
Akshat D.
05/29/2023, 10:47 AMOhkay. I am a beginner at Kubernetes but I can try this. How should I edit precisely?
Akshat D.
05/29/2023, 10:53 AMOh. It's a vim editor. What are the changes I should make to the config map? I don't see a task_resources key or a way to define the default config file to be used while starting the union sandbox.
Akshat D.
05/29/2023, 11:27 AMGot it. I need to add the task_resources to the configmap. Right? As described here: https://flyte-org.slack.com/archives/CP2HDHKE1/p1677672917236769?thread_ts=1677661973.390819&cid=CP2HDHKE1
s
Samhita Alla
05/29/2023, 11:40 AM
Yeah, I think so! Can you add that block and give it a try?
a
Akshat D.
05/29/2023, 11:40 AMThank you so much.
Akshat D.
05/29/2023, 11:40 AMIt worked.
Akshat D.
05/29/2023, 11:42 AMSummary: To add resource limits, we can just add the task_resources block to the config map which can be accessed by "kubectl -n flyte edit cm flyte-sandbox-config". Reference: https://flyte-org.slack.com/archives/CP2HDHKE1/p1677672917236769?thread_ts=1677661973.390819&cid=CP2HDHKE1
s
Samhita Alla
05/29/2023, 11:44 AM
If you could add the same to our troubleshooting guide, that'd be awesome!
a
Akshat D.
05/30/2023, 4:59 AMI will do that.
154 Views