Hey guys I need some help with pod templates not being applied to map_task. Context: I'm running a map task where each individual task runs a python script which then runs subprocess and uses

multiprocessing.SharedMemory

to process CPU intensive work in parallel avoiding memory overhead. It works and runs super fast locally but inside the Flyte environment I'm seeing

135 SIGBUS

errors. The container gets hard killed so there's virtually no logs other than

[anvhqfqwz7x9c56675l9-fkge6mai-0-dn0-0-n3-0] terminated with exit code (135). Reason [Error]. Message: <random unrelated print logs>

I did some print debugging and found out that

/dev/shm

only has 65MB assigned to it so now I'm trying to raise the amount of tempfs memory that's available for this path. the only way i could conceive of doing this is using a pod template so i made one (see appendix 1. below). I then hooked it up to the map_task first at the map_task syntax layer

map_task(
    functools.partial(
        my_map_task,
    ),
    concurrency=...,
    metadata=TaskMetadata(...),
)(partition_idx=[...]).with_overrides(
    requests=Resources(...),
    limits=Resources(...),
    pod_template="large-shm-template", <----- HERE
)

but when that didn't work i did it on the task decorator layer

@task(
    secret_requests=[...],
    pod_template_name="large-shm-template", <---- HERE
)
def my_map_task(

This also didn't work. Both times, it would simply not mount the volume into the container. It would however mount a volume that's specified in the default pod template which is a mystery to me as well given it's being overridden. The only thought I have is PodTemplate has

placeholder

and

dummy/image

because naturally map_task containers have dynamic container names and images. But I don't know what else to try. If anyone could help that would be splendid thank you. Appendix 1.

apiVersion: v1
kind: PodTemplate
metadata:
  name: large-shm-template
  namespace: maps-flyte
template:
  spec:
    containers:
      - name: placeholder
        image: dummy/image  # won't be used 
        volumeMounts:
          - name: shm-volume
            mountPath: /dev/shm
    volumes:
      - name: shm-volume
        emptyDir:
          medium: Memory
          sizeLimit: 40Gi

@strong-soccer-41351 have you tried with the

shared_memory

argument of the task decorator? According to the docs:

shared_memory – If True, then shared memory will be attached to the container where the size is equal to the allocated memory. If int, then the shared memory is set to that size.

In the end the

map_task

is a node type that will wrap the task (and its config)