#4105 [Core feature] Add PodTemplate support for Spark driver and executor pods Issue created by andrewwdye ### Motivation: Why do you think this is important? Today Spark driver and executor pods are not configurable on a per task basis; they use the shared propeller k8s plugin config. This disallows, for instance, setting additional tolerations when running a Spark job. This extends work PodTemplate investments made in ### Goal: What should the final outcome look like, ideally? flytekit should accept optional driver and executor pod templates in Spark task config

@task(
    task_config=Spark(
        ...
        driver_pod_template=PodTemplate(...),
        executor_pod_template=PodTemplate(...),
    ),
    ...
)

The Spark plugin should use these optional templates and existing ToK8sPodSpec logic to populate the SparkApplicationSpec. ### Describe alternatives you've considered The Spark on K8s operator supports a bunch of k8s pod settings via

spark_conf

, which is already available in flytekit. While it doesn't support tolerations, for example, we could extend this concept with the addition of keys like

spark.kubernetes.driver.tolerations

and either get support added to Spark on K8s operator or parse out separately in flyteplugins and apply to the SparkApplicationSpec when building it. ### Propose: Link/Inline OR Additional context One solution here is to add driver and executor pod template parameters to the Spark dataclass and underlying SparkJob proto. Then in flyteplugins we can populate the driver and executor specs. ### Are you sure this issue hasn't been raised already? • Yes ### Have you read the Code of Conduct? • Yes flyteorg/flyte