#1697 Feat: Warn when doing local torch elastic training with nnodes > 1 Pull request opened by fg91 TL;DR With

@task(task_config=Elastic(...))

one can perform training with torch elastic launch (

torchrun

). This works both locally as well as in a cluster with a kubeflow

PyTorchJob

. When executing a workflow locally, i.e.

python workflow.py

, but setting e.g.

Elastic(nnodes=2)

, the rendezvous of the workers will timeout because the workers wait for the non-existing workers from the non-existing 2nd node to join. One would have to set the log level to debug in order to see that torch is waiting for the rendezvous to complete. By default, the workflow appears to not do anything. I thins PR I add a warning log message that informs the user about this. Type ☐ Bug Fix ☑︎ Feature ☐ Plugin Are all requirements met? ☑︎ Code completed ☑︎ Smoke tested ☐ Unit tests added ☐ Code documentation added ☐ Any pending items have an associated Issue Complete description I check for an environment variable that is set by the kubeflow training operator. If this is not set but the user set

nnodes>1

, the warning is emitted. One could discuss whether we should just automatically switch to

nnodes=1

if the environment variables for distributed training have not been set by the training operator but I found this too intrusive. Warning the user, however, should be done. Tracking Issue NA Follow-up issue NA flyteorg/flytekit ✅ All checks have passed 30/30 successful checks