batch size issue in a multi-GPU environment

[edisonguo]

Training with horovod in a multi-GPU environment will invoke setup_datagen on each replica. But the batch_size of setup_datagen is set to self.global_batch_size, which will result in an effective batch size of self.global_batch_size * num_replicas. In both single and multi-GPUs, setup_datagen's batch_size should set to self.batch_size because the "global batch size" will be implicitly handled by the distributed data parallelism of horovod.

Please also see a related discussion here.