#3123 [Core feature] Add PodTemplate configuration to flytekit task definitions Issue created by hamersaw Motivation: Why do you think this is important? Many users require more complex k8s configuration within the Flyte ecosystem. This may include specifying task-specific tolerations, volume mounts, etc. Currently this is unsupported within flytekit, where the API favors ergonomics in abstracting away the complexities of k8s configuration from end-users. However, some use-cases require this complex configuration and striking a balance is very difficult. Goal: What should the final outcome look like, ideally? Users should be able to configure every aspect of the k8s Pod that Flyte creates without bloating the task definition. One solution is to add a PodTemplate configuration to every flytekit task definition. This would work similarly to the default PodTemplate scheme where this PodTemplate definition serves as the base in creating the k8s Pod. By default this will be empty and therefore not applied, but if it is specified it will serve as the base for Pod configuration. This API could look something like:

@task(
    request=Resources(),
    pod_template=V1PodTemplate{
        spec=V1PodSpec{
            tolerations=[
                V1Toleration{
                    key= "num-gpus"
                    operator= "Equal"
                    value= 1
                    effect= "NoSchedule"
                },
            ]
        }
    }
)

Describe alternatives you've considered Currently, proposals often include one-off configuration updates that solve a specific use-case but are not applicable in generalized terms. This approach was used in the k8s plugin and has proved unmaintainable in that every option requires code changes which may change between k8s versions. Propose: Link/Inline OR Additional context No response Are you sure this issue hasn't been raised already? ☑︎ Yes Have you read the Code of Conduct? ☑︎ Yes flyteorg/flyte