Hi, I got the

OOMKilled

error, and I found many similar questions in this channel. According to the discussion, the reason is resource shortage, so I set

@task(limits=Resources(storage="20Gi", ephemeral_storage="20Gi", mem="40Gi", cpu="6"))

which is enough for the error task. But I still got the same error. I do not use GPU. Any other reasons for this?

[1/1] currentAttempt done. Last Error: USER::Pod failed. No message received from kubernetes.
[f52215af773e248b0af3-n2-0] terminated with exit code (1). Reason [OOMKilled]. Message: 
asctime": "2023-03-20 05:09:46,909", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
tar: Removing leading `/' from member names
{"asctime": "2023-03-20 05:09:48,309", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
...

Can you increase mem to an even bigger value and see if that's resolving the issue?

Ok, I will set 64Gi which is max size.

Is there enough memory on the worker node? You could use request instead of limit that guarantees the allocation of those resources to the process

I run it locally.

But you are deploying to a local k8s cluster?

Yes, I installed

minikube

. flyte use it?

what's the error here - that minikube profile doesn't exist? seems to work regardless?

But I still can see flyte node.

ryo@ryo:~$ kubectl top node -n flyte
W0320 20:26:20.436047  179361 top_node.go:119] Using json format to get metrics. Next release will switch to protocol-buffers, switch early by passing --use-protocol-buffers flag
NAME           CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%   
344a31d3f672   167m         2%     1793Mi          2%

sorry, haven't used minikube 😞

No problem. I am also confused with minikube and flytectl🙂

🙂 maybe list the profiles with

minikube profile list

?

🙂 maybe list the profiles with

minikube profile list

?

yes, I already got this suggestion and tried it.

ryo@ryo:~$ minikube stop

🤷 Profile "minikube" not found. Run "minikube profile list" to view all profiles.

👉 To start a cluster, run: "minikube start"

ryo@ryo:~$ minikube profile list

🤹 Exiting due to MK_USAGE_NO_PROFILE: No minikube profile was found.

💡 Suggestion:

You can create one using 'minikube start'.

You are using limits. Limits is like optimistic allocation- k8s will schedule and kill if not available. Use request. This is forced. So on minikube you do not have the mem, it will not get scheduled

Thank you. I found the document. I will try it tomorrow.https://docs.flyte.org/projects/cookbook/en/latest/auto/deployment/customizing_resources.html

increasing limits is futile

My bad. Haven't closely looked at what's being assigned.

I got

Request rejected by the API, due to Invalid input.

when I set

requests

instead of

limits

.

$ pyflyte run --remote --image <http://gcr.io/myimage:mytag|gcr.io/myimage:mytag> src/train.py training_workflow
{"asctime": "2023-03-21 08:22:47,049", "name": "flytekit", "levelname": "WARNING", "message": "FlyteSchema is deprecated, use Structured Dataset instead."}
Request rejected by the API, due to Invalid input.
        Reason: 
        Input Request: {
  "id": {
...

@enough-notebook-15673, could you share your task decorator arguments?

For example, this one

@task(requests=Resources(storage="20Gi",  mem="50Gi", cpu="6"))

Could you share the full error msg?

I am not able to run even with both

How have you set up your Flyte cluster?