<#7417 feat: add clustered-task plugin for JobSet-...
# flyte-github
a
#7417 feat: add clustered-task plugin for JobSet-based distributed training Pull request opened by AdilFayyaz Why are the changes needed? The existing PyTorch/Kubeflow Training Operator plugin is brittle and upstream Kubeflow is moving toward JobSet. This plugin provides a cleaner, Kubernetes-native approach to distributed training using sigs.k8s.io/jobset. What changes were proposed in this pull request? • Adds
flyteplugins/go/tasks/plugins/k8s/clustered/
— a new k8s plugin that reads a ClusteredTaskSpec proto and emits a jobset.x-k8s.io/v1alpha2 JobSet • Splices
python -m flyte.distributed._entrypoint
as the container command and injects torchrun env vars (
NNODES, NPROC_PER_NODE, MASTER_PORT, RDZV_BACKEND
, plus downward API vars for JobSet
name/restart-attempt/namespace
) • Maps JobSet status conditions to Flyte PhaseInfo (initializing → running → success/retryable failure) • Registers the plugin via a blank import in `executor/setup.go`; adds sigs.k8s.io/jobset v0.5.2 to go.mod How was this patch tested? • Unit tests cover BuildResource golden output, entrypoint splicing, env var injection, failure policy construction, and all GetTaskPhase condition mappings • Run:
go test ./flyteplugins/go/tasks/plugins/k8s/clustered/... -v
• All 13 tests pass Setup process Screenshots Check all the applicable boxes • I updated the documentation accordingly. • All new and existing tests passed. • All commits are signed-off. Related PRs Stack If you do use
git town
to manage PR Stacks, the stack relevant to this PR will show below. Otherwise, you can ignore this section. Docs link flyteorg/flyte