Hello! I am running Ray on Flyte. I am getting a warning about Ray using /tmp instead of /dev/shm because /dev/shm has only 67108864 bytes available. To fix this, I would just specify --shm-size=3.55gb when running the container. But Flyte is running the containers for us, so I cannot figure out how to specify any run options. Is there a way for specifying run options for the containers that Flyte runs? Full text of warning attached.

cc @Dan Rammer (hamersaw) do you know if we can specify the pod spec for ray jobs?

@Ketan (kumare3) yes!

awesome @Ruksana Kabealo

Hey @Ruksana Kabealo, I'm looking into this a bit and not finding a definitive answer. So Flyte uses the kuberay project to launch Ray tasks, basically it creates an instance of the RayJob which is then executed. So basically, we need to figure out what configuration we need to change on the RayJob CR to support this. I have found this issue which seems to indicate that these are simple memory requests / limits. Does this sound correct?

Hey @Dan Rammer (hamersaw) ! Yes, it should be a simple memory limit change to expand the size of /dev/shm

Have you tried using the task resource requests / limits? IIUC Flytes Ray plugin uses those to set the container-level requests.