I am having trouble configuring shm (shared memory for the cluster). I have created a “Memory” volume, and also assigned sizeLimit of “20Gi”. The pod description accurately reflects the new tmpfs file system and size. However when I SSH into the container running on the Pod, I see only 4.3Gi allocated. The node has 64GB of RAM. Any pointers, how I can solve this?. Thank you !! Here is default Pod Template:

apiVersion: v1
kind: PodTemplate
metadata:
  name: path-flyte-default-pod-template
  namespace: flytesnacks-development
template:
  spec:
    containers:
      - name: default
        image: <<http://docker.io/rwgrim/docker-noop%7Cdocker.io/rwgrim/docker-noop>>
        terminationMessagePath: "/dev/foo"
        volumeMounts:
        - mountPath: /dev/shm
          name: dshm
    volumes:
      - emptyDir:
          medium: Memory
          sizeLimit: 20Gi
        name: dshm

This is the right way

Hi Ketan, when I ssh into container, and check for size , the shm mount shows only 4 GB. On the pod description I see the correct settings .

Did you try ‘df /dev/shm’

Yes,

@salmon-flower-36598, can you double-check which version of k8s you're running? Memory-backed volumes was set to true by default only in 1.22 (as per https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/, the feature gate is called

SizeMemoryBackedVolumes

. Also, keep in mind that the max size of a memory-backed volume is capped at the

memory

limits for the container, which in the case of flyte is controlled by default by the

task_resources

value in the flyte config. Are you deploying flyte using the

flyte-core

helm chart?

Hi Eduardo, I am running 1.30 version in EKS. and I believe SizeMemoryBackedVolumes is turned on by default. I have setup high limits in task_resources, and yes I am using flyte-core helm chart.

@salmon-flower-36598, thanks for double-checking. Yeah, can you confirm the memory resource limits in the pod, specifically in the container ?

Hi Eduardo, Yes I do have the resource limits on the pod. Eventually I abandoned the idea, and instead of medium: Memory , I am using filebacking