Hi, I am wondering if there is an example/tutorial on multi-node multi-gpu training. I only see the single-node multi-gpu training example.

for multi-node multi-gpu you will need to use mpi plugin

when using a the mpi plugin to create a mpijob, how would you create a job like this:

mpirun -np 4 \
-H 104.171.200.62:2,104.171.200.182:2 \
-x MASTER_ADDR=104.171.200.62 \
-x MASTER_PORT=1234 \
-x PATH \
-bind-to none -map-by slot \
-mca pml ob1 -mca btl ^openib \
python3 main.py --backend=nccl --use_syn --batch_size=8192 --arch=resnet152

The ip addresses of the nodes aren't know until they are spun up correct?

tf-plugin and pytorch-plugin also support multi-node multi-gpu training. basically, we use training-operator to manage the pods (tasks) life cycle, and it will set a env

TF_CONFIG

in every pods.

TF_CONFIG

contains each node ip address, and master’s address. Therefore, the tasks can communicate with each others when training. same as horovod and mpi

@delightful-computer-49028 the mpi job is automatically created