https://flyte.org
Join Slack
Hi, while initiating ray cluster, the task is running in only one instance and pod. Generally if a r...
# ray-integration
f
Hi, while initiating ray cluster, the task is running in only one instance and pod. Generally if a ray cluster is initiated it is expected to run in different instance in distributed manner right? can we do horizontal scaling here to increase the pool of resources here?
t
cc: @glamorous-carpet-83516
g
hmm, if ray task is started, propeller should create head node and workers nodes. did you enable the ray plugin in propeller?
Copy code
tasks:
  task-plugins:
    enabled-plugins:
      - container
      - sidecar
      - k8s-array
      - ray
    default-for-task-types:
      container: container
      sidecar: sidecar
      container_array: k8s-array
      ray: ray
f
Yeah ray plugin is enabled
g
is there any error in the kuberay operator?
f
not sure. how to check if it works fine?
g
kubectl logs <kuberay-operator> -n ray-system
f
message has been deleted
g
have you installed ingress controller? if not, it will cause an error in kuberay, kuberay use ingress controller to create a new ingress route for RayJob
f
yes ingress controller is installed in the setup
g
@future-notebook-79388 do you have couple mins to hop on a call?
f
sure ... pls let me know ur feasible timings
g
maybe 9~12 AM in your time
f
Sorry for the inconvenience @glamorous-carpet-83516. We were having live demo so couldn't work on the setup. Will tomorrow same time work for u ?
g
No worries, yes, ping me tomorrow when you are available
f
Hi actually once the helm is upgraded I am able to see the worker pods getting created. But the issue now is that the task is getting queued for a long time it is not getting initiated. It gets 
The node was low on resource: ephemeral-storage
and it is trying to initiate a new pod but we have enough ephemeral storage in the instance.
The docker image that we are trying to pull is nearly 10gb. will that be an issue? shall we connect by tomorrow mrng 9 AM on my time? can u confirm on where to connect through slack or google meet?
g
I’ll call you at 9am your time through google meet
158 Views
Open in Slack
PreviousNext
Powered by

Flyte

Flyte enables production-grade orchestration for machine learning workflows and data processing created to accelerate local workflows to production.

Powered by