I tried running a task that needs a GPU on my local sandbox cluster, to test it out first. In the console, it shows as RUNNING, in kubectl, the pod shows as PENDING (for several hours even). I tried adding a simple print statement at the start, but don't see see it with kubectl logs, so I assume it's some scheduling or setup issue. I wonder what's a good way to debug this sort of issue?

There are no gpus available in sandbox

I setup the flytesnacks resources as follows:

project: flytesnacks 
domain: development
defaults:
  cpu: "8"
  gpu: "1"
  memory: 32Gi 
  storage: "32Gi"
limits:
  cpu: "8"
  gpu: "1"
  memory: 32Gi 
  storage: "32Gi"

That is ok, but sandbox is probably running on your local machine with no gpus right

image.png

IMG_2900.jpg

Ah, indeed, the exclamation mark is clear:

8/29/2023 2:22:28 PM UTC Unschedulable:0/1 nodes are available: 1 Insufficient <http://nvidia.com/gpu|nvidia.com/gpu>. preemption: 0/1 nodes are available: 1 No preemption victims found for incoming pod.

This PR has a helpful relevant discussion: https://github.com/flyteorg/flyte/pull/3256 My coworker set up sandbox with GPUs yesterday, he couldn’t do it with flytectl so he used

k3d

directly, and ran into gpu issues for which this was the solution . Our local sandbox gpu clusters are running well now, so while sandbox does not officially support GPUs you can work around it in the same way From kubernetes’ side, to get a pod to schedule you need your pod to have a toleration for the nvidia.com/gpu taint (if the node is tainted), which you can set in the pod spec as you shared already Then you need to install the nvidia device plugin daemonset on the cluster, here we had issues with this daemonset’s pods being stuck in ContainerCreating status because the nvidia container runtime was not used, which we fixed according to this discussion

Awesome, thanks so much for the pointers Victor, building the image right now and will report back!:)

@Petr Pilař You can try this tutorial to run

flyte-binary

on a local K3d environment Then, you can adjust taints and tolerations as described here to enable task Pods to consume GPUs