Skip to content
This repository has been archived by the owner on Nov 29, 2023. It is now read-only.

Try Teacher Forcing for RNNs #45

Open
jacobbieker opened this issue Jul 7, 2021 · 2 comments
Open

Try Teacher Forcing for RNNs #45

jacobbieker opened this issue Jul 7, 2021 · 2 comments
Assignees
Labels
enhancement New feature or request

Comments

@jacobbieker
Copy link
Member

This is where, with some random chance, we give the RNN like ConvLSTM the ground truth label when its generating sequences in training. This can help with convergenc especially in the beginning, since if the model messes up in the second timestep, all later ones will be useless, but if we give it the GT image to continue the sequence it might still learn better representations.

@jacobbieker jacobbieker added the enhancement New feature or request label Jul 7, 2021
@jacobbieker jacobbieker self-assigned this Jul 7, 2021
@jacobbieker
Copy link
Member Author

The chance that a single output of the RNN is swapped with a GT label decreases somewhat rapidly, like going from nearly 1 near the beginning of training, and dropping down to 0 after 5 or so epochs.

@tcapelle
Copy link

tcapelle commented Oct 7, 2021

A good approach to do this is putting a param on the model like:

teacher_forcing_prob = 0
...
def forward(self, x, targets):
  if self.teacher_forcing_prob:
    replace the model input with targets

and then in the training:

model.teacher_forcing_prob = 1 - epoch/total_epochs

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants