For distributed pytorch (or tf, …) tasks, the return value of which worker is passed along to subsequent tasks? Is this random/a race condition? When creating a Pytorch task, the

args

of both pods specify the same values for:

- --output-prefix
    - gs://.../metadata/propeller/sandbox-development-f6695ca08aa47490c859/n0/data/0
    - --raw-output-data-prefix
    - gs://.../xq/f6695ca08aa47490c859-n0-0



    - --output-prefix
    - gs://...metadata/propeller/sandbox-development-f6695ca08aa47490c859/n0/data/0
    - --raw-output-data-prefix
    - gs://.../xq/f6695ca08aa47490c859-n0-0

In case both return the same value (as ist assumably often the case), it shouldn’t matter if both write. But in case I want to return a metric which might be slightly different for each worker, is it random which one I get?

No you can control this. For all worker processes you can raise an ignoreoutputs exception and let rank0 return output

I’ll just add an

if os.environ.get("RANK") != 0: raise IgnoreOutputs

then? But otherwise it is random? (Just to make sure I fully understand)

this is a docs issue

I can document it, where should it go?

where would you put it from a user point

I first looked here: https://docs.flyte.org/projects/cookbook/en/stable/auto/integrations/kubernetes/kfpytorch/pytorch_mnist.html

sure