Hi was trying distributed training using ray in flyte I am g

Question

Hi was trying distributed training using ray in flyte I am getting this error while running ```from flytekitplugins ray import HeadNodeConfig RayJobConfig WorkerNodeConfig import ray from ray import tune ray init ray init auto ignore reinit error=True ray config = RayJobConfig head node config=HeadNodeConfig ray start params= log color True worker node config= WorkerNodeConfig group name= ray group replicas=2 num actors = 4 num cpus per actor = 1 ray params = RayParams num actors=num actors cpus per actor=num cpus per actor def train model config train x train y = load breast cancer return X y=True train set = RayDMatrix train x train y evals result = bst = train params=config dtrain=train set evals result=evals result evals= train set train verbose eval=False ray params=ray params bst save model model xgb task task config=ray config limits=Resources mem= 2000Mi cpu= 1 def train model task gt dict config = tree method approx objective binary logistic eval metric logloss error eta tune loguniform 1e 4 1e 1 subsample tune uniform 0 5 1 0 max depth tune randint 1 9 analysis = tune run train model config=config metric= train error mode= min num samples=4 resources per trial=ray params get tune resources return analysis best config workflow def train model wf gt dict return train model task ```

Ketan (kumare3) · Answer

Running out of disk

Ketan (kumare3) · Answer

Request more pleas

Samhita Alla · Answer

` task task config=ray config limits=Resources mem= 2000Mi cpu= 1 ephemeral storage= 500Mi `

Padma Priya M · Answer

```from sklearn datasets import load breast cancer from flytekit import Resources task workflow from flytekitplugins ray import HeadNodeConfig RayJobConfig WorkerNodeConfig import ray from ray import tune ray shutdown ray init ray init auto ignore reinit error=True ray config = RayJobConfig head node config=HeadNodeConfig ray start params= log color True worker node config= WorkerNodeConfig group name= ray group replicas=3 num actors = 2 num cpus per actor = 1 ray params = RayParams num actors=num actors cpus per actor=num cpus per actor def train model config train x train y = load breast cancer return X y=True train set = RayDMatrix train x train y evals result = bst = train params=config dtrain=train set evals result=evals result evals= train set train verbose eval=False ray params=ray params bst save model model xgb task limits=Resources mem= 2000Mi cpu= 1 task task config=ray config limits=Resources mem= 3000Mi cpu= 1 ephemeral storage= 3000Mi def train model task gt dict config = tree method approx objective binary logistic eval metric logloss error eta tune loguniform 1e 4 1e 1 subsample tune uniform 0 5 1 0 max depth tune randint 1 9 analysis = tune run train model config=config metric= train error mode= min num samples=4 max concurrent trials=1 resources per trial=ray params get tune resources return analysis best config workflow def train model wf gt dict return train model task ``` Still getting this error when we specify `ephemeral storage` value also do u have any suggested limit for cpu and memory

Samhita Alla · Answer

If you re using demo cluster I think 1Gi is the limit

Padma Priya M · Answer

i am trying it on EKS cluster

Samhita Alla · Answer

<https github com flyteorg flyte blob aae01aa33eadfb86f1c952eb415f21326ea5519b charts flyte core values eks yaml L216> section specifies the task resource defaults

Samhita Alla · Answer

Can you check yours Please increasing the mem I believe `kubectl n flyte edit cm flyte admin base config` is the command but I m not very sure Let me know if this doesn t work

Samhita Alla · Answer

Nice Please increase your mem and try again

Padma Priya M · Answer

I increased the memory in task the execution is getting queued but it is in pending state for long time Even in remote run the workflow is running for more than 4h for 4 trials but the execution is not happening

Samhita Alla · Answer

Have you seen the message saying you asked for 3 cpu and 0 gpu but the cluster has 2 cpu and 0 gpu

Padma Priya M · Answer

yes but i have requested for only 1 cpu should i change anywhere else ``` task task config=ray config limits=Resources mem= 5000Mi cpu= 1 ephemeral storage= 3000Mi ```

Samhita Alla · Answer

I think it s because of `get tune resources `

Samhita Alla · Answer

Have you seen <https docs ray io en releases 1 11 0 ray more libs xgboost ray html memory usage> section in the doc

Samhita Alla · Answer

I m assuming you re training an xgboost model

Padma Priya M · Answer

```ray config = RayJobConfig head node config=HeadNodeConfig ray start params= log color True worker node config= WorkerNodeConfig group name= ray group replicas=3 ``` do we have any ways to specify the number of cpus in the ray cluster config like this ```ray config = RayJobConfig head node config=HeadNodeConfig ray start params= log color True worker node config= WorkerNodeConfig group name= ray group replicas=3 num cpus=4 ```

Padma Priya M · Answer

bcause as mentioned above we have 64 cpus in eks cluster but it shows this warning that we have only 2 cpus in ray cluster how to increase the cpu limit in ray cluster config

Samhita Alla · Answer

I believe you you can set them in `RayParams`

Samhita Alla · Answer

Padma Priya M · Answer

yeah but in RayParams we could specify the number of cpus that has to be utilized for each trial `cpus per actor` Is there any config to be changed to increase the cpu of the ray cluster as a whole bcause when I increased the cpus per actor also the requested cpu is still 2 and shows the warning that it has only 2 cpu in the cluster

Samhita Alla · Answer

< Kevin Su> any idea how we can set the ray cluster resources As per the docs it should be possible with `init ` but in this case since Flyte initializes the cluster how can a user modify those values

Kevin Su · Answer

To set Ray cluster resource just update the `limit` and `request` in the task Like <https github com flyteorg flytesnacks blob a3b97943563cfc952b5683525763578685a93694 cookbook integrations kubernetes ray example ray example py L56|https github com flyteorg flytesnacks blob a3b97943563cfc952b5683525763578685a93 694 cookbook integrations kubernetes ray example ray example py>

Padma Priya M · Answer

``` task task config=ray config requests=Resources mem= 5000Mi cpu= 5 ephemeral storage= 1000Mi limits=Resources mem= 7000Mi cpu= 9 ephemeral storage= 2000Mi ``` I have requested for 5 cpus but when it executes it shows requested cpus as 2 only

Padma Priya M · Answer

and show same warning too that we have only 2 cpu in the cluster

Samhita Alla · Answer

I m wondering where it s picking you asked for 9 0 cpu from Is it from your `limits`

Padma Priya M · Answer

I think it is based on the resource requested per trial when i specified cpus per trial and num actors as 2 and 4 it showed requested cpus as 9 when i decreased the resource requested and num actors as 2 and 1 it showed 3

Padma Priya M · Answer

When the cpus per trial and num actors are 1 the actual requested cpu is 2 and the execution is happening fine since we have sufficient 2 cpus in the cluster when the num actors are increased it requests for more cpus so the execution is not happening

Samhita Alla · Answer

Um got it We need to find a way to increase the cluster resources Not sure why `requests` isn t assigning the requested resources to the cluster

Padma Priya M · Answer

yeah kindly notify if there is any way to do so

Samhita Alla · Answer

< Kevin Su> do you have any ideas

Kevin Su · Answer

< Padma Priya M> Could you describe the RayJob kubectl describe and check if the resource is same as you specify in the task I guess the head node doesn t use all the cpu in the pod In other words the cpu of head pod could be 10 but cpu of the head node process in the pod could be 2

Padma Priya M · Answer

I have attached the allocated memory when we describe the node ``` task task config=ray config requests=Resources mem= 5000Mi cpu= 5 limits=Resources mem= 7000Mi cpu= 9 ``` This is the requested resources

Kevin Su · Answer

sorry could you describe the rayJob you are running

Padma Priya M · Answer

is there any command for this

Padma Priya M · Answer

This is the shown when we describe the kuberay operator while running

Kevin Su · Answer

kubectl describe RayJobs lt name gt n lt namespace gt