Hi community, is there a way to use gpus in the sandbox cluster?

Here is a pr to build a gpu image for the sandbox.

tried building the image but got the following error. not sure if I missed something

=> ERROR [flytebuilder 6/7] COPY --from=flyteconsole /app/dist cmd/single/dist                                                                                                                                                                                                            0.0s
------                                                                                                                                                                                                                                                                                          
 > [flytebuilder 6/7] COPY --from=flyteconsole /app/dist cmd/single/dist:
------
Dockerfile:15
--------------------
  13 |     RUN go mod download
  14 |     COPY cmd cmd
  15 | >>> COPY --from=flyteconsole /app/dist cmd/single/dist
  16 |     RUN --mount=type=cache,target=/root/.cache/go-build --mount=type=cache,target=/root/go/pkg/mod \
  17 |         go build -tags console -v -o dist/flyte cmd/main.go
--------------------
ERROR: failed to solve: failed to compute cache key: failed to calculate checksum of ref o9gpa8wxdy93rhm0ln129nnas::jkz102t1mu3qkv65xr1hyjjzi: "/app/dist": not found

need to update line 15. https://github.com/flyteorg/flyte/commit/401973c72dd59497f03ca6efd2267fe850ccb5c3

is there also a limit on memory request in sandbox? I am getting rejections when requesting more than 1Gi of mem

yeah, there is a limit in the flyte-sandbox config map.

task_resource_defaults.yaml: |
    task_resources:
      defaults:
        cpu: 500m
        memory: 1Gi
        storage: 500Mi
      limits:
        cpu: 2
        gpu: 5
        memory: 4Gi
        storage: 20Mi

is there a way to override that when creating the sandbox cluster?

You can increase the mem values by running

kubectl -n flyte edit cm flyte-admin-base-config

command.

Updated the GPU PR; hopefully it works with a current demo cluster now

thanks! I will give it a try

Oh yeah. GHCR is down. https://www.githubstatus.com/

@Björn I was able to build gpu image with updated PR. but the the sandbox container still immediately exited with code 1 (same with docker run). what version of flytectl did you test with? I am on

{
  "App": "flytectl",
  "Build": "29da288",
  "Version": "0.6.34",
  "BuildTime": "2023-03-15 10:45:13.597115631 -0500 CDT m=+0.041086554"
}

Maybe it's better to continue the discussion here, rather than in the PR ^_^

here are the logs:

[2023-03-15T17:53:37+00:00] Running k3d entrypoints...
[2023-03-15T17:53:37+00:00] Running /bin/k3d-entrypoint-cgroupv2.sh
[2023-03-15T17:53:37+00:00] Running /bin/k3d-entrypoint-flyte-sandbox-bootstrap.sh
2023/03/15 17:53:37 failed to apply transformations: lookup host.docker.internal on 8.8.8.8:53: no such host

yeah, I get that same error when running the entrypoint with v1.4, so it's something else... check with the stopped container after a failed

flytectl demo start

or maybe @jeev can help?

flytectl does that for you

then run the entrypoint with parameters like the Dockerfile:

/bin/k3d-entrypoint.sh server --disable=traefik --disable=servicelb

ah

here is what I did:

docker run -it --gpus all --entrypoint bash --add-host host.docker.internal:host-gateway flyte-sandbox-gpu:latest

in the container

/bin/k3d-entrypoint.sh server --disable=traefik --disable=servicelb

part of the logs:

INFO[0010] certificate CN=system:node:293d72719bcd,O=system:nodes signed by CN=k3s-client-ca@1678906975: notBefore=2023-03-15 19:02:55 +0000 UTC notAfter=2024-03-14 19:03:05 +0000 UTC 
INFO[0010] Waiting to retrieve agent configuration; server is not ready: "overlayfs" snapshotter cannot be enabled for "/var/lib/rancher/k3s/agent/containerd", try using "fuse-overlayfs" or "native": failed to mount overlay: operation not permitted 
INFO[0011] Waiting for control-plane node 293d72719bcd startup: nodes "293d72719bcd" not found 
INFO[0012] Waiting for control-plane node 293d72719bcd startup: nodes "293d72719bcd" not found 
INFO[0013] Waiting for control-plane node 293d72719bcd startup: nodes "293d72719bcd" not found 
INFO[0014] Waiting for control-plane node 293d72719bcd startup: nodes "293d72719bcd" not found 
I0315 19:03:10.389468      14 range_allocator.go:83] Sending events to api server.

Try starting the container in privileged mode, I think that might be needed:

docker run -it --gpus all --privileged --entrypoint bash --add-host host.docker.internal:host-gateway flyte-sandbox-gpu:latest

I updated makefile:

.PHONY: start
start: FLYTE_SANDBOX_IMAGE := flyte-sandbox-gpu:latest
start: FLYTE_DEV := False
start:
	[ -n "$(shell docker volume ls --filter name=^flyte-sandbox$$ --format {{.Name}})" ] || \
		docker volume create flyte-sandbox
	[ -n "$(shell docker ps --filter name=^flyte-sandbox$$ --format {{.Names}})" ] || \
		docker run -it --rm --privileged --name flyte-sandbox \
			--gpus all \
			--add-host "host.docker.internal:host-gateway" \
			--env FLYTE_DEV=$(FLYTE_DEV) \
			--env K3S_KUBECONFIG_OUTPUT=/.kube/kubeconfig \
			--volume $(PWD)/.kube:/.kube \
			--volume $(HOME)/.flyte/sandbox:/var/lib/flyte/config \
			--volume flyte-sandbox:/var/lib/flyte/storage \
			--publish "6443":"6443" \
			--publish "30000:30000" \
			--publish "30001:30001" \
			--publish "30002:30002" \
			--publish "30080:30080" \
			$(FLYTE_SANDBOX_IMAGE)
		export KUBECONFIG=.kube/kubeconfig

and it is working with

make start

!

Surprising that

flytectl demo start

didn't work then... 🤔

I committed the failed container. Here are log files in

/var/log

alternatives.log  apt  bootstrap.log  btmp  dpkg.log  faillog  lastlog  wtmp

do you want to take a look at any of these files?

source <(make kubeconfig)

@Nan Qin weird there aren't any k3d-entrypoint-logs... not sure to look for in the others :-/

if you use the dry-run flag with flytectl it will print out commands

Aha, thanks - Please give it a go @Nan Qin 🙂

❇️ Run the following command to create new sandbox container
        docker create --privileged -p 0.0.0.0:30000:30000 -p 0.0.0.0:30001:30001 -p 0.0.0.0:30002:30002 -p 0.0.0.0:6443:6443 -p 0.0.0.0:30080:30080 --env SANDBOX=1 --env KUBERNETES_API_PORT=30086 --env FLYTE_HOST=localhost:30081 --env FLYTE_AWS_ENDPOINT=<http://localhost:30084> --env K3S_KUBECONFIG_OUTPUT=/var/lib/flyte/config/kubeconfig --mount type=bind,source=/home/nan/.flyte,target=/etc/rancher/ --mount type=bind,source=/home/nan/.flyte/sandbox,target=/var/lib/flyte/config --mount type=volume,source=flyte-sandbox,target=/var/lib/flyte/storage --name flyte-sandbox flyte-sandbox-gpu:latest

doesn't have

--add-host "host.docker.internal:host-gateway"

hmm that might just be a bug in rendering the line. is everything else the same?

hmm that might just be a bug in rendering the line. is everything else the same?

the same as the make target?

right. you can try running the command directly with the add-host arg

yeah I added

--add-host host.docker.internal:host-gateway --gpus all

and it works

hmm

Just to verify if it's really missing @Nan Qin, you can

docker inspect

the failed container from flytectl and check if it has a section such as this one

"ExtraHosts": [

"host.docker.internal:host-gateway"

]

yeah the failed one has it

"ExtraHosts": [
            "host.docker.internal:host-gateway"
        ],

Focusing the the

--gpus all

... I realise now that I have a non-standard flag in my docker-file needed for gpus to be passed to docker build.

i was curious about —gpus all as well

Wouldn't have found it without your help @jeev - it was a group effort 🙂 Hope it works for @Nan Qin

yes, after setting the default runtime to nvidia, flytectl demo start works for me.

we need better docs for this: https://flyte-org.slack.com/archives/CP2HDHKE1/p1678454867658579?thread_ts=1677247518.480459&amp;cid=CP2HDHKE1

@jeev is there a way to increase /dev/shm in the pod? Pytorch dataloaders easily run out of shared memory as in this issue. Below is output of df -h from the pod

overlay         1.8T  932G  808G  54% /
tmpfs            64M     0   64M   0% /dev
tmpfs            32G     0   32G   0% /sys/fs/cgroup
/dev/nvme0n1p2  1.8T  932G  808G  54% /etc/hosts
shm              64M  8.0K   64M   1% /dev/shm
tmpfs            32G   12K   32G   1% /run/secrets/kubernetes.io/serviceaccount
tmpfs            32G   12K   32G   1% /proc/driver/nvidia
tmpfs            32G     0   32G   0% /proc/acpi
tmpfs            32G     0   32G   0% /proc/scsi
tmpfs            32G     0   32G   0% /sys/firmware

yea using a pod template seems like the best bet. there is no easier way to add volumes to task pods outside of using sidecar tasks. but not sure how this works within docker though.

hmm, it didn't pick up the podTemplate. Here is what I did 1. kubectl -n flyte edit cm flyte-sandbox-config, add template-name so it is now

010-inline-config.yaml: |
    plugins:
      k8s:
        default-env-vars:
        - FLYTE_AWS_ENDPOINT: <http://flyte-sandbox-minio.flyte:9000>
        - FLYTE_AWS_ACCESS_KEY_ID: minio
        - FLYTE_AWS_SECRET_ACCESS_KEY: miniostorage
        default-pod-template-name: flyte-template

2. kubectl apply -f podTemplate.yaml which is

apiVersion: v1
kind: PodTemplate
metadata:
  name: flyte-template
  namespace: flyte
template:
  spec:
    volumes:
    - name: dshm
      emptyDir:
        medium: Memory
        sizeLimit: 64000Mi
    containers:
      - name: default
        image: <http://docker.io/rwgrim/docker-noop|docker.io/rwgrim/docker-noop>
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
        env:
          - name: FOO
            value: BAR

3. wait for a few mins and start a workflow. Neither the volume nor the env vars are in the task container.

you don’t have to modify the configmap directly

also tried setting pod_template_name in @task as in this pr, but got error:

Workflow[flytesnacks:development:workflows.workflow.baby_training_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [BadTaskSpecification] PodTemplate 'flyte-template' does not exist

however I have the podTemplate in both

flyte

and

flytesnacks-development

ns

just add it to ~/.flyte/sandbox/config.yaml

~/.flyte/sandbox/config.yaml

?

yea

should it look like this?

task_resources:
  defaults:
    cpu: 1
    memory: 2Gi
    storage: 32Gi
  limits:
    cpu: 8
    memory: 128Gi
    storage: 512Gi
inline:
  plugins:
    k8s:
      default-pod-template-name: flyte-template

yes

ok, reloaded

did the flyte-sandbox deployment restart?

yes, reload

ok. did the pod restart and pick up the changes?

still getting this

Workflow[flytesnacks:development:workflows.workflow.baby_training_wf] failed. RuntimeExecutionError: max number of system retry attempts [11/10] exhausted. Last known status message: failed at Node[n0]. RuntimeExecutionError: failed during plugin execution, caused by: failed to execute handle for plugin [container]: [BadTaskSpecification] PodTemplate 'flyte-template' does not exist

very helpful thread - helped me get going on testing using a GPU on the demo sandbox. thanks @Nan Qin for raising this, @Björn@jeevfor the guidance. looking forward to when we get all this as part of the official image and docs