Hi I got the `OOMKilled` error and I found many similar ques Flyte #ask-the-community

Hi, I got the `OOMKilled` error, and I found many ...

Ryo M

03/20/2023, 5:30 AM

Hi, I got the

OOMKilled

error, and I found many similar questions in this channel. According to the discussion, the reason is resource shortage, so I set

@task(limits=Resources(storage="20Gi", ephemeral_storage="20Gi", mem="40Gi", cpu="6"))

which is enough for the error task. But I still got the same error. I do not use GPU. Any other reasons for this?

Copy code

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f52215af773e248b0af3-n2-0] terminated with exit code (1). Reason [OOMKilled]. Message: 
asctime": "2023-03-20 05:09:46,909", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
tar: Removing leading `/' from member names
{"asctime": "2023-03-20 05:09:48,309", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
...

Samhita Alla

03/20/2023, 7:02 AM

Can you increase mem to an even bigger value and see if that's resolving the issue?

Ryo M

03/20/2023, 7:05 AM

Ok, I will set 64Gi which is max size.

Ryo M

03/20/2023, 10:06 AM

It does not solved.

Ryo M

03/20/2023, 10:08 AM

How can I get more information about the error? Is there options such as verbose or v? It is hard to debug only with

OOMKilled

Björn

03/20/2023, 11:04 AM

Is there enough memory on the worker node? You could use request instead of limit that guarantees the allocation of those resources to the process

Ryo M

03/20/2023, 11:05 AM

I run it locally.

Ryo M

03/20/2023, 11:05 AM

Yes, my machine has enough memory.

Björn

03/20/2023, 11:09 AM

But you are deploying to a local k8s cluster?

Ryo M

03/20/2023, 11:10 AM

Yes, I installed

minikube

. flyte use it?

Ryo M

03/20/2023, 11:11 AM

Do you know the namespace for flyte? I do not know these are for flyte.

Copy code

ryo@ryo:~$ kubectl get po
NAME          READY   STATUS      RESTARTS   AGE
py39-cacher   0/1     Completed   0          25m
ryo@ryo:~$ kubectl get node
NAME           STATUS   ROLES                  AGE   VERSION
344a31d3f672   Ready    control-plane,master   26m   v1.24.4+k3s1

Ryo M

03/20/2023, 11:13 AM

Copy code

ryo@ryo:~$ kubectl get po -n flyte
NAME                                                  READY   STATUS    RESTARTS   AGE
flyte-sandbox-docker-registry-b5c57c55-rtc4q          1/1     Running   0          29m
flyte-sandbox-kubernetes-dashboard-6757db879c-4q6cm   1/1     Running   0          29m
flyte-sandbox-proxy-d95874857-8t8h7                   1/1     Running   0          29m
flyte-sandbox-postgresql-0                            1/1     Running   0          29m
flyte-sandbox-75c5d88454-tpm9c                        1/1     Running   0          29m
flyte-sandbox-minio-645c8ddf7c-9f9zg                  1/1     Running   0          29m

Ryo M

03/20/2023, 11:14 AM

Copy code

ryo@ryo:~$ kubectl get node -n flyte
NAME           STATUS   ROLES                  AGE   VERSION
344a31d3f672   Ready    control-plane,master   30m   v1.24.4+k3s1

Ryo M

03/20/2023, 11:16 AM

@Björn My minikube cluster use default value... I will set larger one. Thank you very much.

Copy code

ryo@ryo:~$ kubectl top node -n flyte
W0320 20:15:45.557393  176553 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
344a31d3f672   185m         2%     1797Mi          2%

Ryo M

03/20/2023, 11:22 AM

flyte uses minikube? When I tried to delete minikube cluster then I got the error.

ryo@ryo:~$ minikube delete

🙄 "minikube" profile does not exist, trying anyways.

💀 Removed all traces of the "minikube" cluster.

Björn

03/20/2023, 11:25 AM

what's the error here - that minikube profile doesn't exist? seems to work regardless?

Björn

03/20/2023, 11:27 AM

If you use the demo cluster:

flytectl demo start

flyte will set up a k8s environment for you... https://docs.flyte.org/projects/flytectl/en/latest/gen/flytectl_demo.html

Ryo M

03/20/2023, 11:27 AM

But I still can see flyte node.

Copy code

ryo@ryo:~$ kubectl top node -n flyte
W0320 20:26:20.436047  179361 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
344a31d3f672   167m         2%     1793Mi          2%

Björn

03/20/2023, 11:27 AM

sorry, haven't used minikube 😞

Ryo M

03/20/2023, 11:28 AM

No problem. I am also confused with minikube and flytectl🙂

Björn

03/20/2023, 11:29 AM

🙂 maybe list the profiles with

minikube profile list

Björn

03/20/2023, 11:29 AM

https://stackoverflow.com/questions/72147700/deleting-minikube-cluster-so-i-can-create-a-larger-cluster-with-more-cpus

Ryo M

03/20/2023, 11:30 AM

🙂 maybe list the profiles with
minikube profile list
?

yes, I already got this suggestion and tried it.

ryo@ryo:~$ minikube stop

🤷 Profile "minikube" not found. Run "minikube profile list" to view all profiles.

👉 To start a cluster, run: "minikube start"

ryo@ryo:~$ minikube profile list

🤹 Exiting due to MK_USAGE_NO_PROFILE: No minikube profile was found.

💡 Suggestion:

You can create one using 'minikube start'.

Ryo M

03/20/2023, 11:31 AM

minikube is related to this issue?

Ketan (kumare3)

03/20/2023, 2:18 PM

You are using limits. Limits is like optimistic allocation- k8s will schedule and kill if not available. Use request. This is forced. So on minikube you do not have the mem, it will not get scheduled

Ketan (kumare3)

03/20/2023, 2:19 PM

Cc @Samhita Alla ^ - increasing limits is futile

Ryo M

03/20/2023, 2:24 PM

Thank you. I found the document. I will try it tomorrow.https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/customizing_resources.html

Samhita Alla

03/20/2023, 3:47 PM

increasing limits is futile

My bad. Haven't closely looked at what's being assigned.

Samhita Alla

03/20/2023, 3:48 PM

@Ryo M, let us know if you're able to resolve the issue.

Ryo M

03/20/2023, 11:25 PM

I got

Request rejected by the API, due to Invalid input.

when I set

requests

instead of

limits

Copy code

$ pyflyte run --remote --image <http://gcr.io/myimage:mytag|gcr.io/myimage:mytag> src/train.py training_workflow
{"asctime": "2023-03-21 08:22:47,049", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Request rejected by the API, due to Invalid input.
        Reason: 
        Input Request: {
  "id": {
...

Ryo M

03/20/2023, 11:36 PM

This is the same error message. But the reason may be different. https://flyte-org.slack.com/archives/CP2HDHKE1/p1678462238468779

Ryo M

03/20/2023, 11:39 PM

Also https://github.com/flyteorg/flyte/issues/3404#issuecomment-1468327645

Ryo M

03/20/2023, 11:49 PM

I realized the new version of flytekit was released. I will try it.

Ryo M

03/20/2023, 11:51 PM

The same error even after 1.4.1 -> 1.4.2 of flyte

Ryo M

03/21/2023, 12:04 AM

Even small requests, I got the same error.

Ryo M

03/21/2023, 12:22 AM

I tried

minikube start --cpus 6 --memory 40G --disk-size 50G --kubernetes-version v1.21.14 --driver=kvm2

but nothing changed.

Samhita Alla

03/21/2023, 5:13 AM

@Ryo M, could you share your task decorator arguments?

Ryo M

03/21/2023, 8:02 AM

For example, this one

Copy code

@task(requests=Resources(storage="20Gi",  mem="50Gi", cpu="6"))

Samhita Alla

03/21/2023, 9:37 AM

Could you share the full error msg?

Samhita Alla

03/21/2023, 9:40 AM

Are you sure you're able to run the workflow when you set

limits

instead of

requests

Ryo M

03/21/2023, 11:18 PM

I am not able to run even with both

Samhita Alla

03/23/2023, 5:07 AM

How have you set up your Flyte cluster?

166 Views

Open in Slack

Previous Next