<#7383 feat: add ClusteredTaskSpec proto definitio...
# flyte-github
a
#7383 feat: add ClusteredTaskSpec proto definition for JobSet-based distributed training Pull request opened by AdilFayyaz Why are the changes needed? The existing PyTorch plugin depends on the Kubeflow Training Operator, which upstream Kubeflow is moving away from. A new clustered-task type needs a well-typed proto contract between the Python SDK and the Go plugin before either can be built. What changes were proposed in this pull request? • Adds
flyteidl2/plugins/clustered.proto
with ClusteredTaskSpec, Runtime/TorchRunSpec, ClusterFailurePolicy, and Interconnect/RdzvBackend enums • Runtime is a tagged union (oneof) — Phase 1 ships TorchRun only; DeepSpeed/JAX/Accelerate are additive future fields with no plugin changes required • ClusterFailurePolicy.restart_on_host_maintenance maps to a JobSet RestartJobSetAndIgnoreMaxRestarts rule, enabling free restarts on node eviction • Interconnect enum replaces a hypothetical enable_rdma bool — makes invalid states (RDMA on TCP) unrepresentable How was this patch tested? • Run cd flyte2 && make buf — must exit 0 and produce gen/go/flyteidl2/plugins/clustered.pb.go • Run make buf-lint — must pass with no violations Setup process Screenshots Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. Related PRs Stack If you do use
git town
to manage PR Stacks, the stack relevant to this PR will show below. Otherwise, you can ignore this section. Docs link flyteorg/flyte