Do built-in trainers support distributed training ?

[fabito]

Hi,

First of all, thanks for this awesome library.
I noticed that the DistributedTripletMarginLossMNIST example does not use any of the built-in trainers.
Is it correct to assume that all trainers extending BaseTrainer don't support distributed training ?
Do you think we could have a BaseDistributedTrainer class. Does that make sense ?

[KevinMusgrave]

That's correct, the trainers do not support distributed training. I'd like to add this functionality, but it isn't the top priority right now, so I won't be able to add it anytime soon.

[fabito]

Thanks @KevinMusgrave .
If I'll create a PR if I manage to sort it out.
BTW, do you have any thoughts on how you would design/implement it ? A separate base class ? Additional hooks in BaseTrainer ?

[KevinMusgrave]

I think there's 2 approaches:

1. Add the necessary code to BaseTrainer, and then add a flag to __init__ to run in distributed mode.
2. Make BaseTrainer extend one of the pytorch frameworks that takes care of distributed training, like Ignite, Catalyst, or Lightning.

Probably the 2nd approach is better because those frameworks are already focused on solving this type of problem.

If you'd like to discuss more, please post in issue #72