<#4179 Add driver and executor pod template fields...
# flyte-github
a
#4179 Add driver and executor pod template fields to SparkJob Pull request opened by andrewwdye Tracking issue #4105 Describe your changes This change defines a common pattern for per-role pod configuration and adds RoleSpec fields for driver and executor pods, accordingly. • Adds a
RoleSpec
to common.proto with initial fields for
pod
and
pod_template_name
. This mirrors the interface we have for container tasks spread across TaskTemplate and TaskMetadata. Fields are optional. • Adds
driverSpec
and
executorSpec
to SparkJob This PR was manually migrated from flyteorg/flyteidl#450. See initial comment discussions there. How should a plugin handle TaskTemplate/TaskMetadata pod templates vs. these RoleSpecs? For Spark, the task-level pod template is not consumed today and can be treated as an error if set (not supported). The plugin can inspect optional per-role specs for configuration. For existing plugins that consume the task-level pod template (Dask, MPI, PyTorch, Ray), they can continue doing so. If/when we are ready to introduce per-role pod configuration, we can add
RoleSpec
fields to the plugin's config, deprecate the task-level pod template, and rely on a runtime check to validate that only task-level pod template OR per-role pod templates are specified. Check all the applicable boxes ☑︎ I updated the documentation accordingly. ☐ All new and existing tests passed. ☑︎ All commits are signed-off. flyteorg/flyte ✅ All checks have passed 16/16 successful checks