hey team when i try running my code with ray plugin enabled

Question

hey team when i try running my code with ray plugin enabled the pod seems to be running file no issues in the log also but the issue being the code keeps on running and i dont get the output and the code runs fine locally Can u please look into it Thanks Code ```import typing from flytekit import ImageSpec Resources task workflow custom image = ImageSpec name= ray flyte plugin registry= anirudh1905 packages= flytekitplugins ray if custom image is container import ray from flytekitplugins ray import HeadNodeConfig RayJobConfig WorkerNodeConfig ray remote def f1 x return x x ray remote def f2 x return x 2 ray config = RayJobConfig head node config=HeadNodeConfig ray start params= log color True worker node config= WorkerNodeConfig group name= ray group replicas=1 runtime env= pip numpy pandas or runtime env= requirements txt task cache=True cache version= 0 2 task config=ray config requests=Resources mem= 2Gi cpu= 1 container image=custom image def ray task n int gt int futures = f2 remote f1 remote i for i in range n return sum ray get futures workflow def ray workflow n int gt int return ray task n=n ``` project config yaml ```domain development project flytesnacks defaults cpu 1 memory 2Gi limits cpu 3 memory 8Gi ``` I also tried with kuberay version 0 3 and 0 5 2 in both its not working

Anirudh Sridhar · Answer

Steps that i followed `flytectl demo start` Installed kuberay ```export KUBERAY VERSION=v0 5 2 kubectl create k kubectl apply k ``` `flytectl update task resource attribute attrFile project config yaml` `pyflyte run remote example ray py ray workflow n 1`

Samhita Alla · Answer

With kuberay `v0 6 0` I m seeing the following error in the task pod ```RuntimeError Request failed with status code 500 Traceback most recent call last File usr local lib python3 11 site packages ray dashboard modules job job head py line 287 in submit job resp = await job agent client submit job internal submit request ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages ray dashboard modules job job head py line 73 in submit job internal async with File usr local lib python3 11 site packages aiohttp client py line 1141 in aenter self resp = await self coro ^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages aiohttp client py line 560 in request await resp start conn File usr local lib python3 11 site packages aiohttp client reqrep py line 899 in start message payload = await protocol read type ignore union attr ^^^^^^^^^^^^^^^^^^^^^ File usr local lib python3 11 site packages aiohttp streams py line 616 in read await self waiter aiohttp client exceptions ServerDisconnectedError Server disconnected``` < Kevin Su>

Samhita Alla · Answer

With kuberay `v0 5 2` that I installed via helm I m seeing the following error ```2023 09 12T13 40 30 840Z ERROR controllers RayJob failed to submit job error SubmitJob fail Traceback most recent call last n File usr local lib python3 10 site packages aiohttp connector py line 980 in wrap create connection n return await self loop create connection args kwargs type ignore return value noqa n File usr local lib python3 10 asyncio base events py line 1076 in create connection n raise exceptions 0 n File usr local lib python3 10 asyncio base events py line 1060 in create connection n sock = await self connect sock n File usr local lib python3 10 asyncio base events py line 969 in connect sock n await self sock connect sock address n File usr local lib python3 10 asyncio selector events py line 501 in sock connect n return await fut n File usr local lib python3 10 asyncio selector events py line 541 in sock connect cb n raise OSError err f Connect call failed address nConnectionRefusedError Errno 111 Connect call failed 10 42 0 17 52365 n nThe above exception was the direct cause of the following exception n nTraceback most recent call last n File usr local lib python3 10 site packages ray dashboard modules job job head py line 287 in submit job n resp = await job agent client submit job internal submit request n File usr local lib python3 10 site packages ray dashboard modules job job head py line 73 in submit job internal n async with n File usr local lib python3 10 site packages aiohttp client py line 1141 in aenter n self resp = await self coro n File usr local lib python3 10 site packages aiohttp client py line 536 in request n conn = await self connector connect n File usr local lib python3 10 site packages aiohttp connector py line 540 in connect n proto = await self create connection req traces timeout n File usr local lib python3 10 site packages aiohttp connector py line 901 in create connection n proto = await self create direct connection req traces timeout n File usr local lib python3 10 site packages aiohttp connector py line 1209 in create direct connection n raise last exc n File usr local lib python3 10 site packages aiohttp connector py line 1178 in create direct connection n transp proto = await self wrap create connection n File usr local lib python3 10 site packages aiohttp connector py line 988 in wrap create connection n raise client error req connection key exc from exc naiohttp client exceptions ClientConnectorError Cannot connect to host 10 42 0 17 52365 ssl default Connect call failed 10 42 0 17 52365 n ``` ImageSpec config ```custom image = ImageSpec name= ray flyte plugin registry= samhitaalla packages= flytekitplugins ray==1 9 1 ray==2 6 3 base image= ```

Kevin Su · Answer

I create an issue <https github com flyteorg flyte issues 4026|here> Ray seems to have some issues so it failed to run the task in the sandbox The same task can be run on the eks cluster

Samhita Alla · Answer

Should we be making any changes to the plugin to fix this

Anirudh Sridhar · Answer

Hey < Kevin Su> i need it for the POC

Kevin Su · Answer

Ok will deep dive into it today

Kevin Su · Answer

I find the bug are you using mac M1 or M2

Kevin Su · Answer

cc < Anirudh Sridhar>

Anirudh Sridhar · Answer

M1 Max chip

Kevin Su · Answer

kuberay doesn t build the image for arm64

Kevin Su · Answer

so kuberay operator keeps failing to submit the job

Anirudh Sridhar · Answer

So what should I do to test it locally

Kevin Su · Answer

I m building the arm image right now I can share it with you shortly

Kevin Su · Answer

then you can replace current kuberay image with mine

Kevin Su · Answer

pingsutw kuberay v2

Kevin Su · Answer

you could use `kubeclt edit n ray system deploy kuberay operator` to update the image

Kevin Su · Answer

```custom image = ImageSpec platform= linux arm64 ```

Kevin Su · Answer

pass platform config to image Spec for building the image for m1

Anirudh Sridhar · Answer

the pods are created but the flyte console giving this error `Failed to create Ray job fe6dfd3a66f074481942 n0 0`

Anirudh Sridhar · Answer

< Kevin Su>

Anirudh Sridhar · Answer

and i saw the logs of the head node everything looks fine

Kevin Su · Answer

are you using kuberay 0 5 2

Kevin Su · Answer

did you see any error in kuberay operator

Anirudh Sridhar · Answer

no

Kevin Su · Answer

do push your image to docker hub if so could you share the image uri I can test it on my side

Anirudh Sridhar · Answer

ig this is the one `anirudh1905 ray flyte plugin`

Kevin Su · Answer

what s its tag

Anirudh Sridhar · Answer

`anirudh1905 ray flyte plugin MyZdKEQHTVHkHlvw ZUxQ `

Kevin Su · Answer

there are some issues in flytekit 1 9 1 I just tested 1 9 0 and it works for me ```custom image = ImageSpec packages= flytekitplugins ray flytekit==1 9 0 platform= linux arm64 ```

Anirudh Sridhar · Answer

Hey < Kevin Su> the code works fine for n=1 and n=10 but fails for 20 and starts giving error insufficient cpu Also 1 more observation when i pass 3 as cpu it starts giving insufficient cpu for n=1 also given that i have given 5 cpu to docker desktop and this is my project config yaml ```domain development project flytesnacks defaults cpu 3 memory 2Gi limits cpu 4 memory 8Gi ```