Hi, while initiating ray cluster, the task is running in only one instance and pod. Generally if a ray cluster is initiated it is expected to run in different instance in distributed manner right? can we do horizontal scaling here to increase the pool of resources here?

cc: @Kevin Su

hmm, if ray task is started, propeller should create head node and workers nodes. did you enable the ray plugin in propeller?

tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
      - ray
    default-for-task-types:
      container: container
      sidecar: sidecar
      container_array: k8s-array
      ray: ray

Yeah ray plugin is enabled

is there any error in the kuberay operator?

not sure. how to check if it works fine?

kubectl logs <kuberay-operator> -n ray-system

message has been deleted

have you installed ingress controller? if not, it will cause an error in kuberay, kuberay use ingress controller to create a new ingress route for RayJob

yes ingress controller is installed in the setup

@Padma Priya M do you have couple mins to hop on a call?

sure ... pls let me know ur feasible timings

maybe 9~12 AM in your time

Sorry for the inconvenience @Kevin Su. We were having live demo so couldn't work on the setup. Will tomorrow same time work for u ?

No worries, yes, ping me tomorrow when you are available

Hi actually once the helm is upgraded I am able to see the worker pods getting created. But the issue now is that the task is getting queued for a long time it is not getting initiated. It gets

The node was low on resource: ephemeral-storage

and it is trying to initiate a new pod but we have enough ephemeral storage in the instance.

I’ll call you at 9am your time through google meet