https://flyte.org logo
Powered by
#ask-the-community
Title
# ask-the-community
j

James Evers

09/14/2022, 6:38 PM
hey everyone, i'm trying to run ray on the flyte sandbox deployment, and am running into the following error. it looks like its basically an OOM error, but i'm not sure how to increase the default size of 
/dev/shm
. what's the best approach to bumping this default in the sandbox deployment?
Copy code
/usr/local/lib/python3.8/dist-packages/paramiko/transport.py:236: CryptographyDeprecationWarning: Blowfish has been deprecated
  "class": algorithms.Blowfish,
2022-09-14 18:10:43,190	WARNING services.py:1882 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=0.10gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
2022-09-14 18:10:46,611	INFO worker.py:1509 -- Started a local Ray instance. View the dashboard at [1m[32m<http://127.0.0.1:8265> [39m[22m
[2022-09-14 18:10:57,864 E 1 1] <http://core_worker.cc:149|core_worker.cc:149>: Failed to register worker 01000000ffffffffffffffffffffffffffffffffffffffffffffffff to Raylet. IOError: [RayletClient] Unable to register worker with raylet. No such file or directory
k

Ketan (kumare3)

09/14/2022, 8:08 PM
Running ray on sandbox maybe too much of memory required
But you should definitely increase docker memory
j

James Evers

09/14/2022, 8:08 PM
oh okay, so this should be done via the base Dockerfile instead of via 
flytectl
?
oh or do you mean via Docker Desktop?
k

Ketan (kumare3)

09/14/2022, 8:10 PM
Ya
j

James Evers

09/14/2022, 8:13 PM
is there a way to do it that addresses the error here:
Copy code
Error: rpc error: code = InvalidArgument desc = Requested MEMORY default [3500Mi] is greater than current limit set in the platform configuration [1Gi]. Please contact Flyte Admins to change these limits or consult the configuration
i'm not sure how to interact with 
flyteadmin
, is this doable in the sandbox?
my guess is that i should be able to bump the memory in 
config-sandbox.yaml
, but i'm not seeing a full list of parameters for the config anywhere
e

Eduardo Apolinario (eapolinario)

09/14/2022, 8:35 PM
@James Evers,
Copy code
$ kubectl -n flyte edit cm flyte-admin-base-config
...
  task_resource_defaults.yaml: |
    task_resources:
      defaults:
        cpu: 100m
        memory: 500Mi
        storage: 500Mi
      limits:
        cpu: 2
        gpu: 1
        memory: 1Gi <- change this value
        storage: 20Mi
j

James Evers

09/14/2022, 8:35 PM
ahh! perfect, thank you!
as a general rule, is most of the cluster resource management done via 
kubectl
rather than 
flytectl
?
e

Eduardo Apolinario (eapolinario)

09/14/2022, 8:41 PM
correct, at least for these cluster-wide limits, that's the case. 
flytectl
is more about interacting with a flyte deployment, more specifically with its api (via flyteadmin)
j

James Evers

09/14/2022, 8:42 PM
okay, makes sense. one more super newbie question: is this the full list of parameters for the deployment config?
e

Eduardo Apolinario (eapolinario)

09/14/2022, 8:53 PM
No, the full list is the helm charts we publish (you can take a look at them here). We have documentation on the different types of deployment we support in https://docs.flyte.org/en/latest/deployment/index.html (notice the sections on helm charts in each deployment guide)
j

James Evers

09/14/2022, 9:03 PM
okay great, thanks for the help!
e

Eduardo Apolinario (eapolinario)

09/14/2022, 9:06 PM
sure thing and don't hesitate to reach out in case you encounter an issue.
k

Ketan (kumare3)

09/14/2022, 9:33 PM
@James Evers we are working on some deployment streamlining to help Configure Flyte faster - stay tuned and please provide feedback / file Issues
j

James Evers

09/14/2022, 9:34 PM
sounds good! looking forward to it
11 Views
Powered by