Hi everyone.... Hope you are all good. I am currently dealing with two issues which I am trying to implement through flyte. 1. I wanted to provide my flyte users a custom pod template with particular custom Image so that every time when they try to run their task on flyte . Flyte uses that image for execution. 2. I wanted to utilize my GPU's through flyte. Like how can I provide flyte to use GPU's instead of CPU's Note -: My flyte is deployed on the on Prem server having Redhat OS. and We are using flyte-core helm chart. Can you please suggest something for both of my problems? or some way to implement both of them

Hey @loud-keyboard-72751 1. I think you can use a

default-pod-template

for this (docs). Is it what you're looking for? If so, any particular struggle using it? 2. Even on-prem, your GPU device driver should be exposing a supported toleration (like the one shown in this section). Is it OpenShift? or vanilla K8s on RHEL?

Hey @average-finland-92144 I use default pod template ...but it can only overwrite the labels and selectors and everything other than container image

I just tried the following

from flytekit import task, workflow
from flytekitplugins.pod import Pod
from kubernetes.client.models import V1Container, V1PodSpec, V1ResourceRequirements, V1Volume, V1VolumeMount

# Define the shared volume
shared_volume = V1Volume(
    name="shared-data",
    empty_dir={}
)

# Define the volume mount
shared_volume_mount = V1VolumeMount(
    name="shared-data",
    mount_path="/data"
)

# Define the pod spec with two containers
pod_spec = V1PodSpec(
    containers=[
        V1Container(
            name="primary",
            image="primary-image:latest",
            resources=V1ResourceRequirements(
                requests={"cpu": "1", "memory": "100Mi"},
                limits={"cpu": "1", "memory": "100Mi"}
            ),
            volume_mounts=[shared_volume_mount],
            command=["sh", "-c", "echo 'Hello from primary container' && sleep 30"]
        ),
        V1Container(
            name="secondary",
            image="secondary-image:latest",
            command=["sh", "-c", "echo 'Hello from secondary container' > /data/message.txt"],
            volume_mounts=[shared_volume_mount]
        )
    ],
    volumes=[shared_volume]
)

# Define the task using the Pod plugin
@task(task_config=Pod(pod_spec=pod_spec))
def pod_task() -> str:
    with open("/data/message.txt", "r") as f:
        return f.read()

# Define the workflow
@workflow
def pod_workflow() -> str:
    return pod_task()

it's strange, it respects the image for the

secondary

container but overrides it for the

primary

And BTW, it respects the container names

Ok So Let me try this and let you know

I think you have to install the NVIDIA device plugin then you can use tolerations from the Task definition}

Btw we are having single node cluster...Does it still need taint and toleration concept to set the node