has anyone else encountered something similar? Is there a way to provision more shared memory to a specific task?

cc @high-accountant-32689 @freezing-boots-56761

Thanks for the quick reply. We added a

pod_template_name

that volumes it in. A feature request could be to control this via the resource requests 😛

yeah we’re seeing if there’s anything common between migs and shm

👍

well, and maybe how it’s specified by the user.

in flytekit? there might also validations we need to do where shm size limit has to be less than memory limit - shm usage counts towards container memory usage. and sizelimit is also optional - uses up to container memory limit.

Hey @thankful-minister-83577 @freezing-boots-56761 @loud-oil-9899 Did you guys solve it in any easy way or any example for pod_template for that? I’m encountering it when working with DataLoaders 😛

there is an example here: https://github.com/unionai-oss/llm-fine-tuning/blob/77c42f7b297aec8783720ce2dd14fe873cb55503/fine_tuning/llm_fine_tuning.py#L378

Thanks!