Error with Elmo during training

[seangtkelley]

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py", line 768, in _GradientsHelper
    grad_fn = ops.get_gradient_function(op)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py", line 2462, in get_gradient_function
    return _gradient_registry.lookup(op_type)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/registry.py", line 93, in lookup
    "%s registry has no entry for: %s" % (self._name, name))
LookupError: gradient registry has no entry for: AddV2

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "train.py", line 191, in <module>
    run_and_debug(lambda: run(args), args['--debug'])
  File "/usr/local/lib/python3.6/dist-packages/dpu_utils/utils/debughelper.py", line 21, in run_and_debug
    func()
  File "train.py", line 191, in <lambda>
    run_and_debug(lambda: run(args), args['--debug'])
  File "train.py", line 177, in run
    parallelize=not(arguments['--sequential']))
  File "train.py", line 72, in run_train
    model.make_model(is_train=True)
  File "/home/dev/src/models/model.py", line 234, in make_model
    self._make_training_step()
  File "/home/dev/src/models/model.py", line 378, in _make_training_step
    gradients = tf.gradients(self.ops['loss'], trainable_vars)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py", line 630, in gradients
    gate_gradients, aggregation_method, stop_gradients)
  File "/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/gradients_impl.py", line 772, in _GradientsHelper
    (op.name, op.type))
LookupError: No gradient defined for operation 'query_encoder/elmo_encoder/module_apply_tokens/aggregation/add_2' (op type: AddV2)

[seangtkelley]

possible fix: https://github.com/google-research/google-research/issues/91