<#4340 Added GPU enabled sandbox image. (v2?)> Pul...
# flyte-github
c
#4340 Added GPU enabled sandbox image. (v2?) Pull request opened by danpf Preface: Combining work done by @ahlgol and @Future-Outlier with some extra testing/eval/a bunch of nvidia-headache-fixes to get it working fully on ubuntu server. #3256 If ahlgol merges this into the previous PR, this one will close, otherwise we can just use this one (I kept the previous PR's commits) ## Setup / testing # 0. Prerequisites Ensure you have installed them and you can run them all • Installing the NVIDIA Container Toolkit: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/install-guide.html • Nvidia container-toolkit sample-workload: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/sample-workload.html • Support for Container Device Interface: https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/cdi-support.html • NVIDIA device plugin for Kubernetes (Finish All the Quick Start) https://github.com/NVIDIA/k8s-device-plugin#quick-start • A public docker server may be necessary (I pushed just to make sure), so login with
docker login
• general reqs: • kustomize • helm • kubectl • docker • flyte envd reqs: •
pip install flytekitplugins-envd
• see below: create_envd_context.sh and run that My env: (may or may not be necessary) • /etc/docker/daemon.json
Copy code
{
  "default-runtime": "nvidia",
  "runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
  }
}
•
docker context list
Copy code
NAME        DESCRIPTION                               DOCKER ENDPOINT               ERROR
default *   Current DOCKER_HOST based configuration   unix:///var/run/docker.sock
• /etc/containerd/config.toml
Copy code
version = 2

[plugins]

  [plugins."io.containerd.grpc.v1.cri"]

    [plugins."io.containerd.grpc.v1.cri".containerd]

      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes]

        [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia]
          privileged_without_host_devices = false
          runtime_engine = ""
          runtime_root = ""
          runtime_type = "io.containerd.runc.v2"

          [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.nvidia.options]
            BinaryName = "/usr/bin/nvidia-container-runtime"
            SystemdCgroup = true
•
lsb_release -a
Copy code
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 22.04.3 LTS
Release:        22.04
Codename:       jammy
•
nvidia-smi
Copy code
Wed Nov  1 03:52:14 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.06              Driver Version: 545.23.06    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Tesla T4                       On  | 00000000:00:04.0 Off |                    0 |
| N/A   33C    P8               9W /  70W |      2MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  Tesla T4                       On  | 00000000:00:05.0 Off |                    0 |
| N/A   32C    P8               9W /  70W |      2MiB / 15360MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
# 1. Get branch Download the branch, build the dockerfile, tag the image, and push it:
Copy code
git clone <https://github.com/danpf/flyte>
git checkout danpf-sandbox-gpu
cd flyte/docker/sandbox-bundled
make build-gpu
docker tag flyte-sandbox-gpu:latest dancyrusbio/flyte-sandbox-gpu:latest
docker login
docker push dancyrusbio/flyte-sandbox-gpu:latest
# 2. Start the cluster
Copy code
flytectl demo start --image dancyrusbio/flyte-sandbox-gpu:latest --disable-agent --force
# 3. See if you can use the gpu
Copy code
$ kubectl describe node | grep -i gpu
  <http://nvidia.com/gpu|nvidia.com/gpu>:     2
  <http://nvidia.com/gpu|nvidia.com/gpu>:     2
  <http://nvidia.com/gpu|nvidia.com/gpu>     0           0
# 4. run the final job: create the runme.py script shown below, and then run
Copy code
pyflyte run --remote runme.py  check_if_gpu_available
## Testing scripts
Copy code
# create_envd_context.sh
envd context create --name flyte-sandbox --builder tcp --builder-address localhost:30003 --use
quickly rebuild and push your docker image (change the name obviously)
Copy code
# rebuild.sh
make build-gpu && docker tag flyte-sandbox-gpu dancyrusbio/flyte-sandbox-gpu && docker push dancyrusbio/flyte-sandbox-gpu
start a new flyte sandbox cluster
Copy code
# start_new_flyte_cluster.sh
flytectl demo start --image dancyrusbio/flyte-sandbox-gpu:latest --disable-agent --force
This is the final flyte script to check if your gpu is working
Copy code
# runme.py
from flytekit import ImageSpec, Resources, task

gpu = "1"

@task(
    retries=2,
    cache=True,
    cache_version="1.0",
    requests=Resources(gpu=gpu),
    environment={"PYTHONPATH": "/root"},
    container_image=ImageSpec(
            cuda="11.8.0",
            python_version="3.9.13",
            packages=["flytekit", "torch"],
            apt_packages=["git"],
            registry="localhost:30000",
    )
)
def check_if_gpu_available() -> bool:
    import torch
    return torch.cuda.is_available()
## Proof! <https://private-user-images.githubusercontent.com/16297104/279567405-031e90ea-67c0-4b42-9b72-d6e60e8fe077.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ9.eyJpc3MiOiJnaXRodWIuY29tIiwiYXVkIjoicmF3LmdpdGh1YnVzZXJjb250ZW50LmNvbSIsImtleSI6ImtleTUiLCJleHAiOjE3NTYwODA4OTgsIm5iZiI6MTc1NjA4MDU5OCwicGF0aCI6Ii8xNjI5NzEwNC8yNzk1Njc0MDUtMDMxZTkwZWEtNjdjMC00YjQyLTliNzItZDZlNjBlOGZlMDc3LnBuZz9YLUFtei1BbGdvcml0aG09QVdTNC1ITUFDLVNIQTI1NiZYLUFtei1DcmVkZW50aWFsPUFLSUFWQ09EWUxTQTUzUFFLNFpBJTJGMjAyNTA4MjUlMkZ1cy1lYXN0LTElMkZzMyUyRmF3czRfcmVxdWVzdCZYLUFtei1EYXRlPTIwMjUwODI1VDAwMDk1OFomWC1BbXotRXhwaXJlcz0zMDAmWC1BbXotU2lnbmF0dXJlPTdlODkwMTg3ODMzNjEwMTY3NjNiYWU3OTVhZWUwY2Y5NWJkOGJkZTg1YmJjZDk2MWRhOTc2MDMyMjA2OGIxMjEmWC1BbXotU2lnbmVkSGVhZGVycz1ob3N0In0.qlrihr-_2NSUrfecntPxipUPCPOcgiRIdM9m8pJ0KT4|[image](https://private-user-images.githubusercontent.com/16297104/279567405-031e90ea-67c0-4b42-9b72-d6e60e8fe077.png?jwt=eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiJ… flyteorg/flyte