When running `flytectl demo` is there a way to increase the

Question

When running `flytectl demo` is there a way to increase the allowed total ~G~CPU Memory per task My mem= 10Gi requests are received by the flyte server and silently truncated to 1Gi

Dan Farrell · Answer

ok I m pretty sure i need to change `projectQuotaMemory` somehow following I run `flytectl get cluster resource attribute p flytesnacks d development` but I get ```panic runtime error invalid memory address or nil pointer dereference signal SIGSEGV segmentation violation code=0x1 addr=0x70 pc=0x147f37c goroutine 1 running 0xc0000560a8 0x7ffc1eb6a9db 0xc00099f998 0x7ffc1eb6a9ea 0xc000ae5090 0x0 0xc000afc701 0x0 0x0 home runner work flytectl flytectl cmd get matchable attribute util go 32 +0xbc 0xc0000560a8 0xc000315d80 0x0 0x257b120 0x0 0x0 0x0 0x0 0x0 home runner work flytectl flytectl cmd get matchable cluster resource attribute go 78 +0x2a6 0xc000315d80 0x0 0x4 home runner work flytectl flytectl cmd core cmd go 70 +0x93d 0xc000315d40 0x4 0x4 home runner go pkg mod github com spf13 cobra v1 4 0 command go 856 +0x67c home runner go pkg mod github com spf13 cobra v1 4 0 command go 974 +0x3bd home runner go pkg mod github com spf13 cobra v1 4 0 command go 902 home runner work flytectl flytectl cmd root go 137 +0x1e main main home runner work flytectl flytectl main go 12 +0x1d```

Dan Farrell · Answer

trying to follow the instructions for updating the project quotas ```flytectl update cluster resource attribute attrFile cra yaml``` cra yaml ```attributes projectQuotaCpu 1000 projectQuotaMemory 5Ti domain development project flytesnacks``` yields ```panic runtime error invalid memory address or nil pointer dereference signal SIGSEGV segmentation violation code=0x1 addr=0x28 pc=0x1aeed82 goroutine 1 running 0xc0000560a8 0xc000a9cb00 0xb 0xc000a9caf0 0xb 0x0 0x0 0x0 0x0 home runner work flytectl flytectl cmd update matchable attribute util go 37 +0x2e2 0xc0000560a8 0xc000a1fb40 0x100000049c5e5 0x257b120 0x0 0x0 0x0 0x0 0x0 home runner work flytectl flytectl cmd update matchable cluster resource attribute go 75 +0x206 0xc000660760 0x0 0x2 home runner work flytectl flytectl cmd core cmd go 70 +0x93d 0xc000660740 0x2 0x2 home runner go pkg mod github com spf13 cobra v1 4 0 command go 856 +0x67c home runner go pkg mod github com spf13 cobra v1 4 0 command go 974 +0x3bd home runner go pkg mod github com spf13 cobra v1 4 0 command go 902 home runner work flytectl flytectl cmd root go 137 +0x1e main main home runner work flytectl flytectl main go 12 +0x1d```

Dan Farrell · Answer

ah getting closer ```$ flytectl update cluster resource attribute attrFile cra yaml config ~ flyte config sandbox yaml Updated attributes from flytesnacks project and domain development``` seems to work My jobs aren t OOMing Maybe stragely If i go grab the k8s definition from the k8s dashboard I see the pod has the instructions ``` resources limits cpu 2 memory 1Gi requests cpu 2 memory 1Gi``` while my Task has defined `requests=Resources mem= 6500Mi ` So I feel like I m not correctly doing this

Dan Farrell · Answer

edit my task was still eventually oomkilled

Dan Farrell · Answer

ah finally I think I got it In addition to the `cra yaml` above I needed to update the `task resource attribute` as well tra yaml ```defaults cpu 1 memory 1Gi limits cpu 1000 memory 5Ti project flytesnacks domain development ``` cmd ```flytectl update task resource attribute attrFile tra yaml config ~ flyte config sandbox yaml```

Dan Farrell · Answer

now my pod resource correctly reflects my settings ``` resources limits cpu 1 memory 6500Mi requests cpu 1 memory 6500Mi```

Dan Farrell · Answer

<https github com flyteorg flytesnacks pull 1183> added some of the above to the documentation

Kevin Su · Answer

flytectl demo can t use GPU for now cc < L godlike>

Dan Farrell · Answer

that was a typo < Kevin Su> sorry supposed to be CPU Although looking forward to GPU usage as well

Kevin Su · Answer

Yes we are working on enabling GPUs in the sandbox

L godlike · Answer

I am working on it I will mention you when I finished

Ryuu · Answer

ths for your help heart

L godlike · Answer

< Ryuu> Hi currently my ubuntu is using WSL kernel which might cause unexpected behavior when using GPU in the sandbox If you are using Ubuntu can you help me test this step by step Here is a guidance about how to execute it

L godlike · Answer

< Dan Farrell> If you can help I will be really appreciated too

Dan Farrell · Answer

< L godlike> the command ```flytectl demo start image futureoutlier flyte sandbox gpu v2 disable agent force``` does not seem to work on my ubuntu gpu workstation it just exits with no logs

L godlike · Answer

Thanks a lot maybe the dockerfile is not correct I will try to create my own and mention you to try it

Dan Farrell · Answer

```docker run rm it futureoutlier flyte sandbox gpu v2``` also gives no stdout and fails immediately

Dan Farrell · Answer

k3d entrypoint logs ``` 2023 10 30T15 31 24+00 00 Running k3d entrypoints 2023 10 30T15 31 24+00 00 Running bin k3d entrypoint cgroupv2 sh 2023 10 30T15 31 24+00 00 Running bin k3d entrypoint flyte sandbox bootstrap sh 2023 10 30 15 31 24 failed to apply transformations lookup host docker internal on 205 171 3 26 53 no such host```

Ryuu · Answer

Ok i will try it now

Dan Farrell · Answer

those logs are from running via docker run so maybe it is not expected to work when running via docker run Also < L godlike> I must need more coffee The command ```flytectl demo start image futureoutlier flyte sandbox gpu v2 disable agent force``` Does not exit with no logs it gives me the following output ``` status Status Downloaded newer image for futureoutlier flyte sandbox gpu v2 booting Flyte sandbox container Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up Waiting for cluster to come up ```

Dan Farrell · Answer

I was meaning to say that the `flyte sandbox` container that it spawns fails and has no logs ```73a7896f3379 futureoutlier flyte sandbox gpu v2 bin k3d entrypoint 17 minutes ago Exited 1 16 minutes ago flyte sandbox```

L godlike · Answer

Ok I will try to make a new dockerfile with GPU tomorrow and if there s any problem I face and I can t solve it I will come here for help

L godlike · Answer

Thanks you two really appreciated pray skin tone 2

Dan Farrell · Answer

Maybe this is my problem ``` $ docker run gpus=all rm it entrypoint bin bash futureoutlier flyte sandbox gpu v2 docker Error response from daemon failed to create shim task OCI runtime create failed runc create failed unable to start container process error during container init error running hook 0 error running hook exit status 1 stdout stderr Auto detected mode as legacy nvidia container cli requirement error unsatisfied condition cuda gt =12 1 please update your driver to a newer version or use an earlier cuda container unknown ``` My cuda is 11 8

L godlike · Answer

I will update how to specify the cuda version < Ryuu> Can I know your test result

L godlike · Answer

< Dan Farrell> Are you willing to start following this guide from scratch and build your own image I can t test it due to my WSL kernel if the guide doesn t work I will try to produce another Docker GPU image

Dan Farrell · Answer

ok it s building now

L godlike · Answer

Thanks a lot if there s any problem please let me know

Dan Farrell · Answer

< L godlike> with the locally built docker image I can run it with gpu access but I still get the k3s no such host error

Dan Farrell · Answer

is `flyte sandbox bootstrap` doing something wild here or is this a problem with my env

Dan Farrell · Answer

< L godlike> finally getting somewhere

Dan Farrell · Answer

```root 153de33f2937 cat var log k3d entrypoints 231031053514 log 2023 10 31T05 35 14+00 00 Running k3d entrypoints 2023 10 31T05 35 14+00 00 Running bin k3d entrypoint cgroupv2 sh 2023 10 31T05 35 14+00 00 Running bin k3d entrypoint flyte sandbox bootstrap sh 2023 10 31T05 35 14+00 00 Running bin k3d entrypoint gpu check sh bin k3d entrypoint sh 14 bin k3d entrypoint gpu check sh Permission denied root 153de33f2937 chmod +x bin k3d entrypoint gpu check sh``` this needs to be executable

L godlike · Answer

Hi

L godlike · Answer

Looking

L godlike · Answer

So there are 2 main questions to figure out now 1 how `flyte sandbox bootstrap` works 2 how to make `k3s` get the host

L godlike · Answer

I will study it today and reply you

Dan Farrell · Answer

well

Dan Farrell · Answer

All you need to do is ```chmod +x docker sandbox bundled bin k3d entrypoint gpu check sh```

L godlike · Answer

So the image is correct and it works

L godlike · Answer

Can you allocate GPU to the node Can you use the GPU in the task

L godlike · Answer

If possible can you also provide the screenshot in the github description s comment

L godlike · Answer

you can use `kubectl describe node | grep i gpu`

Ryuu · Answer

< L godlike> sr i have some own work right now I will response as soon as possible

L godlike · Answer

It s OK good luck with your work Appreciated to your help

L godlike · Answer

< Dan Farrell> If possible please add the screenshots under this PR description thanks really really much