acoustic-carpenter-78188
10/06/2023, 12:18 AMRoleSpec to common.proto with initial fields for pod and pod_template_name. This mirrors the interface we have for container tasks spread across TaskTemplate and TaskMetadata. Fields are optional.
• Adds driverSpec and executorSpec to SparkJob
This PR was manually migrated from flyteorg/flyteidl#450. See initial comment discussions there.
How should a plugin handle TaskTemplate/TaskMetadata pod templates vs. these RoleSpecs?
For Spark, the task-level pod template is not consumed today and can be treated as an error if set (not supported). The plugin can inspect optional per-role specs for configuration.
For existing plugins that consume the task-level pod template (Dask, MPI, PyTorch, Ray), they can continue doing so. If/when we are ready to introduce per-role pod configuration, we can add RoleSpec fields to the plugin's config, deprecate the task-level pod template, and rely on a runtime check to validate that only task-level pod template OR per-role pod templates are specified.
Check all the applicable boxes
☑︎ I updated the documentation accordingly.
☐ All new and existing tests passed.
☑︎ All commits are signed-off.
flyteorg/flyte
✅ All checks have passed
16/16 successful checksacoustic-carpenter-78188
10/09/2023, 7:28 PM