Topic classification model in keras with MXNet backend errors out - Too many slices such that some splits are empty

[sandeep-krishnamurthy]

Running Reuters Topic Classification Model under keras/example for MXNet backend with 8 GPU errors out with following error:
Error: Too many slices such that some splits are empty

Code - https://github.com/fchollet/keras/blob/master/examples/reuters_mlp.py

Setting:

1. MXNet latest commit till (March 8, 2017)
2. dmlc/keras latest commit till (March 8, 2017)
3. Number of GPU - 8

However, same example works with Tensorflow backend.

Also, with MXNet backend for 1, 2, 4 GPUs it works fine.

[sandeep-krishnamurthy]

Stack trace for reference:

Loading data...
8982 train sequences
2246 test sequences
46 classes
Vectorizing sequence data...
X_train shape: (8982, 1000)
X_test shape: (2246, 1000)
Convert class vector to binary class matrix (for use with categorical_crossentropy)
Y_train shape: (8982, 46)
Y_test shape: (2246, 46)
Building model...
Train on 8083 samples, validate on 899 samples
Epoch 1/5
/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/bucketing_module.py:348: UserWarning: Optimizer created manually outside Module but rescale_grad is not normalized to 1.0/batch_size/num_workers (1.0 vs. 0.03125). Is this intended?
force_init=force_init)
8000/8083 [============================>.] - ETA: 0s - loss: 3.6955 - acc: 0.3495Traceback (most recent call last):
File "reuters_mlp.py", line 64, in
verbose=1, validation_split=0.1)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/models.py", line 672, in fit
initial_epoch=initial_epoch)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 1198, in fit
initial_epoch=initial_epoch)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 907, in _fit_loop
verbose=0)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 992, in _test_loop
batch_outs = f(ins_batch)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 1968, in test_function
data, label, _, data_shapes, label_shapes = self._adjust_module(inputs, 'test')
File "/usr/local/lib/python2.7/dist-packages/Keras-1.2.2-py2.7.egg/keras/engine/training.py", line 1914, in _adjust_module
self._mod._curr_module.reshape(data_shapes, label_shapes)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/module.py", line 427, in reshape
self._exec_group.reshape(self._data_shapes, self._label_shapes)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/executor_group.py", line 311, in reshape
self.bind_exec(data_shapes, label_shapes, reshape=True)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/executor_group.py", line 276, in bind_exec
self.data_layouts = self.decide_slices(data_shapes)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/module/executor_group.py", line 221, in decide_slices
self.slices = _split_input_slice(self.batch_size, self.workload)
File "/usr/local/lib/python2.7/dist-packages/mxnet-0.9.4-py2.7.egg/mxnet/executor_manager.py", line 48, in _split_input_slice
raise ValueError('Too many slices such that some splits are empty')
ValueError: Too many slices such that some splits are empty

[imranshaikmuma]

increase batch_size