Hi community is there a way to use gpus in the sandbox clust

Question

Hi community is there a way to use gpus in the sandbox cluster

Kevin Su · Answer

Here is a <https github com flyteorg flyte pull 3256|pr> to build a gpu image for the sandbox

Nan Qin · Answer

tried building the image but got the following error not sure if I missed something ``` = gt ERROR flytebuilder 6 7 COPY from=flyteconsole app dist cmd single dist 0 0s gt flytebuilder 6 7 COPY from=flyteconsole app dist cmd single dist Dockerfile 15 13 | RUN go mod download 14 | COPY cmd cmd 15 | gt gt gt COPY from=flyteconsole app dist cmd single dist 16 | RUN mount=type=cache target= root cache go build mount=type=cache target= root go pkg mod 17 | go build tags console v o dist flyte cmd main go ERROR failed to solve failed to compute cache key failed to calculate checksum of ref o9gpa8wxdy93rhm0ln129nnas jkz102t1mu3qkv65xr1hyjjzi app dist not found```

Nan Qin · Answer

besides sandbox are there other on prem deployment options

Kevin Su · Answer

need to update line 15 <https github com flyteorg flyte commit 401973c72dd59497f03ca6efd2267fe850ccb5c3>

Nan Qin · Answer

is there also a limit on memory request in sandbox I am getting rejections when requesting more than 1Gi of mem

Kevin Su · Answer

yeah there is a limit in the flyte sandbox config map ``` task resource defaults yaml | task resources defaults cpu 500m memory 1Gi storage 500Mi limits cpu 2 gpu 5 memory 4Gi storage 20Mi```

Nan Qin · Answer

is there a way to override that when creating the sandbox cluster

Samhita Alla · Answer

You can increase the mem values by running `kubectl n flyte edit cm flyte admin base config` command

Björn · Answer

Updated the GPU PR hopefully it works with a current demo cluster now

Nan Qin · Answer

thanks I will give it a try

Nan Qin · Answer

getting 503 from ghcr Something wrong on github thinking face ```ERROR failed to solve <http ghcr io flyteorg flyteconsole latest|ghcr io flyteorg flyteconsole latest> failed to authorize failed to fetch anonymous token unexpected status 503 Service Unavailable```

Samhita Alla · Answer

Oh yeah GHCR is down <https www githubstatus com >

Nan Qin · Answer

< Björn> I was able to build gpu image with updated PR but the the sandbox container still immediately exited with code 1 same with docker run what version of flytectl did you test with I am on ``` App flytectl Build 29da288 Version 0 6 34 BuildTime 2023 03 15 10 45 13 597115631 0500 CDT m=+0 041086554 ```

Björn · Answer

Maybe it s better to continue the discussion here rather than in the PR ^ ^

Björn · Answer

I m using the same flytectl

Björn · Answer

You can start the container with bash as entrypoint by running `docker run it entrypoint bash flyte sandbox gpu latest` and then try to start the cluster with ` bin k3d entrypoint sh` and checking the output in the logs in ` var log k3d entrypoints $ date + y m d H M S log` However since 1 4 there is the new bootstrapping functionality and the k3d entrypoint script doesn t start the cluster on its own anymore

Björn · Answer

`flytectl demo start image flyte sandbox gpu latest` should work however You could also commit the failed container the one that exits with code 1 to a new image start a container based on that with bash as entrypoint and check the logs from there

Nan Qin · Answer

here are the logs ``` 2023 03 15T17 53 37+00 00 Running k3d entrypoints 2023 03 15T17 53 37+00 00 Running bin k3d entrypoint cgroupv2 sh 2023 03 15T17 53 37+00 00 Running bin k3d entrypoint flyte sandbox bootstrap sh 2023 03 15 17 53 37 failed to apply transformations lookup host docker internal on 8 8 8 8 53 no such host```

Björn · Answer

yeah I get that same error when running the entrypoint with v1 4 so it s something else check with the stopped container after a failed `flytectl demo start` or maybe < jeev> can help

Björn · Answer

Aha inspecting the container from the cluster it seems you need to pass an extra host to the container try `docker run it entrypoint bash add host host docker internal host gateway flyte sandbox gpu latest`

jeev · Answer

flytectl does that for you

Björn · Answer

then run the entrypoint with parameters like the Dockerfile ` bin k3d entrypoint sh server disable=traefik disable=servicelb`

Björn · Answer

< jeev> true slightly smiling face we re trying to debug why flytectl demo start can t start the gpu demo container

jeev · Answer

ah

jeev · Answer

try with this make target <https github com flyteorg flyte blob b82eaa5c507640a551f942de5c789198d076a491 docker sandbox bundled Makefile L42|https github com flyteorg flyte blob b82eaa5c507640a551f942de5c789198d076a491 docker sandbox bundled Makefile L42>

jeev · Answer

but instead of detach use it so you see logs

Nan Qin · Answer

here is what I did ```docker run it gpus all entrypoint bash add host host docker internal host gateway flyte sandbox gpu latest``` in the container ``` bin k3d entrypoint sh server disable=traefik disable=servicelb``` part of the logs ```INFO 0010 certificate CN=system node 293d72719bcd O=system nodes signed by CN=k3s client ca 1678906975 notBefore=2023 03 15 19 02 55 +0000 UTC notAfter=2024 03 14 19 03 05 +0000 UTC INFO 0010 Waiting to retrieve agent configuration server is not ready overlayfs snapshotter cannot be enabled for var lib rancher k3s agent containerd try using fuse overlayfs or native failed to mount overlay operation not permitted INFO 0011 Waiting for control plane node 293d72719bcd startup nodes 293d72719bcd not found INFO 0012 Waiting for control plane node 293d72719bcd startup nodes 293d72719bcd not found INFO 0013 Waiting for control plane node 293d72719bcd startup nodes 293d72719bcd not found INFO 0014 Waiting for control plane node 293d72719bcd startup nodes 293d72719bcd not found I0315 19 03 10 389468 14 range allocator go 83 Sending events to api server ```

Nan Qin · Answer

logs rolling like that forever

Nan Qin · Answer

Björn · Answer

Try starting the container in privileged mode I think that might be needed `docker run it gpus all privileged entrypoint bash add host host docker internal host gateway flyte sandbox gpu latest`

Nan Qin · Answer

I updated makefile ``` PHONY start start FLYTE SANDBOX IMAGE = flyte sandbox gpu latest start FLYTE DEV = False start n $ shell docker volume ls filter name=^flyte sandbox$$ format Name || docker volume create flyte sandbox n $ shell docker ps filter name=^flyte sandbox$$ format Names || docker run it rm privileged name flyte sandbox gpus all add host host docker internal host gateway env FLYTE DEV=$ FLYTE DEV env K3S KUBECONFIG OUTPUT= kube kubeconfig volume $ PWD kube kube volume $ HOME flyte sandbox var lib flyte config volume flyte sandbox var lib flyte storage publish 6443 6443 publish 30000 30000 publish 30001 30001 publish 30002 30002 publish 30080 30080 $ FLYTE SANDBOX IMAGE export KUBECONFIG= kube kubeconfig``` and it is working with `make start`

Björn · Answer

Surprising that `flytectl demo start` didn t work then thinking face

Nan Qin · Answer

I committed the failed container Here are log files in ` var log` ```alternatives log apt bootstrap log btmp dpkg log faillog lastlog wtmp``` do you want to take a look at any of these files

Nan Qin · Answer

< jeev> how to set the kubectl context after `make start` `make kubeconfig` doesn t do the magic as `flytectl demo start`

jeev · Answer

source lt make kubeconfig

Björn · Answer

< Nan Qin> weird there aren t any k3d entrypoint logs not sure to look for in the others

Björn · Answer

< jeev> Any theories why a container would start with the Makefile command but not `flytectl demo start image X`

Björn · Answer

Also for me `flytectl demo start image flyte sandbox gpu latest` works

jeev · Answer

if you use the dry run flag with flytectl it will print out commands

jeev · Answer

so maybe can repro that way

Björn · Answer

Aha thanks Please give it a go < Nan Qin> slightly smiling face

Nan Qin · Answer

``` Run the following command to create new sandbox container docker create privileged p 0 0 0 0 30000 30000 p 0 0 0 0 30001 30001 p 0 0 0 0 30002 30002 p 0 0 0 0 6443 6443 p 0 0 0 0 30080 30080 env SANDBOX=1 env KUBERNETES API PORT=30086 env FLYTE HOST=localhost 30081 env FLYTE AWS ENDPOINT=<http localhost 30084> env K3S KUBECONFIG OUTPUT= var lib flyte config kubeconfig mount type=bind source= home nan flyte target= etc rancher mount type=bind source= home nan flyte sandbox target= var lib flyte config mount type=volume source=flyte sandbox target= var lib flyte storage name flyte sandbox flyte sandbox gpu latest``` doesn t have ` add host host docker internal host gateway `

jeev · Answer

hmm that might just be a bug in rendering the line is everything else the same

jeev · Answer

it does work for < Björn> so it s likely an issue with your local setup < Nan Qin>

jeev · Answer

not sure what though

Nan Qin · Answer

gt hmm that might just be a bug in rendering the line is everything else the same the same as the make target

jeev · Answer

right you can try running the command directly with the add host arg

Nan Qin · Answer

yeah I added ` add host host docker internal host gateway gpus all` and it works

jeev · Answer

hmm

jeev · Answer

< Yee> cc

Björn · Answer

Just to verify if it s really missing < Nan Qin> you can `docker inspect` the failed container from flytectl and check if it has a section such as this one gt ExtraHosts gt host docker internal host gateway gt

Björn · Answer

I have that one even though add host is missing from the dryRun output

Nan Qin · Answer

yeah the failed one has it ``` ExtraHosts host docker internal host gateway ```

Björn · Answer

Focusing the the ` gpus all` I realise now that I have a non standard flag in my docker file needed for gpus to be passed to docker build

Björn · Answer

etc docker daemon json ``` default runtime nvidia runtimes nvidia path nvidia container runtime runtimeArgs ```

Björn · Answer

Without ` default runtime nvidia ` it doesn t start with the `flytectl demo start` for me either

Björn · Answer

Check your default runtime with `docker info|grep i runtime`

Björn · Answer

I have an error message that should notify the user but I guess it is hidden by the flytectl tool Would ve been better to put it in the logs confused

jeev · Answer

i was curious about gpus all as well

jeev · Answer

nice find < Björn>

Björn · Answer

Wouldn t have found it without your help < jeev> it was a group effort slightly smiling face Hope it works for < Nan Qin>

Nan Qin · Answer

yes after setting the default runtime to nvidia flytectl demo start works for me

Nan Qin · Answer

big thanks to < Björn> and < jeev>

Nan Qin · Answer

< Samhita Alla> I was trying to change the cpu memory storage limit as you suggested but there is no cm called `flyte admin base config` Below are all the cms in all namespaces ```kube system extension apiserver authentication 6 3h52m kube system cluster dns 2 3h52m flyte flyte sandbox cluster resource templates 1 3h52m flyte flyte sandbox config 5 3h52m flyte flyte sandbox docker registry config 1 3h52m flyte flyte sandbox extra cluster resource templates 0 3h52m flyte flyte sandbox extra config 0 3h52m flyte flyte sandbox proxy config 1 3h52m flyte kubernetes dashboard settings 0 3h52m kube system chart content nvidia device plugin 0 3h52m kube system chart values nvidia device plugin 0 3h52m kube system local path config 4 3h52m flyte kube root ca crt 1 3h52m kube system kube root ca crt 1 3h52m default kube root ca crt 1 3h52m kube public kube root ca crt 1 3h52m kube node lease kube root ca crt 1 3h52m kube system coredns 2 3h52m flytesnacks development kube root ca crt 1 3h51m flytesnacks staging kube root ca crt 1 3h51m flytesnacks production kube root ca crt 1 3h51m```

Nan Qin · Answer

I also searched through all the CMs and didn t find anything related to cpu memory storage limits Any idea to change those limits

jeev · Answer

we need better docs for this <https flyte org slack com archives CP2HDHKE1 p1678454867658579 thread ts=1677247518 480459 amp cid=CP2HDHKE1>

jeev · Answer

< Yee> cc

jeev · Answer

< Nan Qin> add ```task resources defaults cpu 1 memory 2Gi limits cpu 4 memory 8Gi``` or equivalent to `~ flyte sandbox config yaml` and run `flytectl demo reload`

jeev · Answer

wait a bit for pod to reconcile and restart on its own

Nan Qin · Answer

< jeev> is there a way to increase dev shm in the pod Pytorch dataloaders easily run out of shared memory as in this Below is output of df h from the pod ```overlay 1 8T 932G 808G 54 tmpfs 64M 0 64M 0 dev tmpfs 32G 0 32G 0 sys fs cgroup dev nvme0n1p2 1 8T 932G 808G 54 etc hosts shm 64M 8 0K 64M 1 dev shm tmpfs 32G 12K 32G 1 run secrets kubernetes io serviceaccount tmpfs 32G 12K 32G 1 proc driver nvidia tmpfs 32G 0 32G 0 proc acpi tmpfs 32G 0 32G 0 proc scsi tmpfs 32G 0 32G 0 sys firmware```

Nan Qin · Answer

<https flyte org slack com archives CP2HDHKE1 p1676661684505849 thread ts=1676648061 967599 amp cid=CP2HDHKE1> and update the cm to use podtemplate

jeev · Answer

yea using a pod template seems like the best bet there is no easier way to add volumes to task pods outside of using sidecar tasks but not sure how this works within docker though

Nan Qin · Answer

hmm it didn t pick up the podTemplate Here is what I did 1 kubectl n flyte edit cm flyte sandbox config add template name so it is now ``` 010 inline config yaml | plugins k8s default env vars FLYTE AWS ENDPOINT FLYTE AWS ACCESS KEY ID minio FLYTE AWS SECRET ACCESS KEY miniostorage default pod template name flyte template``` 2 kubectl apply f podTemplate yaml which is ```apiVersion v1 kind PodTemplate metadata name flyte template namespace flyte template spec volumes name dshm emptyDir medium Memory sizeLimit 64000Mi containers name default image volumeMounts mountPath dev shm name dshm env name FOO value BAR``` 3 wait for a few mins and start a workflow Neither the volume nor the env vars are in the task container

Nan Qin · Answer

Did I miss something

jeev · Answer

you don t have to modify the configmap directly

Nan Qin · Answer

also tried setting pod template name in task as in <https github com flyteorg flytekit pull 1425|this pr> but got error ```Workflow flytesnacks development workflows workflow baby training wf failed RuntimeExecutionError max number of system retry attempts 11 10 exhausted Last known status message failed at Node n0 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin container BadTaskSpecification PodTemplate flyte template does not exist``` however I have the podTemplate in both `flyte` and `flytesnacks development` ns

jeev · Answer

just add it to ~ flyte sandbox config yaml

jeev · Answer

did you already create that file < Nan Qin>

Nan Qin · Answer

`~ flyte sandbox config yaml`

jeev · Answer

yea

Nan Qin · Answer

should it look like this ```task resources defaults cpu 1 memory 2Gi storage 32Gi limits cpu 8 memory 128Gi storage 512Gi inline plugins k8s default pod template name flyte template```

jeev · Answer

yes

jeev · Answer

drop the inline and indent plugins left

Nan Qin · Answer

ok reloaded

Nan Qin · Answer

still getting this ```Workflow flytesnacks development workflows workflow baby training wf failed RuntimeExecutionError max number of system retry attempts 11 10 exhausted Last known status message failed at Node n0 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin container BadTaskSpecification PodTemplate flyte template does not exist```

Nan Qin · Answer

I have the podTemplates as ```NAMESPACE NAME CONTAINERS IMAGES POD LABELS flyte flyte template default lt none gt flytesnacks development flyte template default lt none gt ```

jeev · Answer

did the flyte sandbox deployment restart

jeev · Answer

did you flytectl demo reload

Nan Qin · Answer

yes reload

jeev · Answer

ok did the pod restart and pick up the changes

Nan Qin · Answer

still getting this ```Workflow flytesnacks development workflows workflow baby training wf failed RuntimeExecutionError max number of system retry attempts 11 10 exhausted Last known status message failed at Node n0 RuntimeExecutionError failed during plugin execution caused by failed to execute handle for plugin container BadTaskSpecification PodTemplate flyte template does not exist```

Nan Qin · Answer

let me try again

Nan Qin · Answer

it works now

Nan Qin · Answer

you are my hero < jeev>

Brian Tang · Answer

very helpful thread helped me get going on testing using a GPU on the demo sandbox thanks < Nan Qin> for raising this < Björn>< jeev>for the guidance looking forward to when we get all this as part of the official image and docs