Continuing training saving new best checkpoint

[JRMeyer]

>>> abdullah.tayyab
[January 10, 2021, 9:15pm]

Hello,

I am training a dataset for Urdu in the native text and successfully
used transfer learning from the English pretrained model to achieve a
loss of 36.299953 after 80 epochs on this data. I want to further
improve on this by adjusting the parameters and applying some
augmentation through Deep Speech.

The one big question I have is that if we are 'continuing' training, why
is the new best validating model saved if it is not better than the one
used in the previous run?

The other question I have is what techniques can we use to decrease this
loss rate? This is the command I am using to 'continue' training.

python3 DeepSpeech.py slash
/ slash $HOME/DeepSpeech/dataset/trained_load_checkpoint slash
/ slash $HOME/DeepSpeech/dataset/trained_load_checkpoint slash

I now want to adjust the parameters to continue and try to improve the
loss.

How much difference would one form of augmentation alone make to our
data? Or would it be more useful to use multiple augmentations together
in the same run?

I know you can't 'think' for me but I am looking for a pointer to try
and improve this. Will running the same data set (around 60 hours)
produce better loss with different augmentation combinations?

The WER at 80 epochs is around 58% with a loss of 36.3. The training
loss is at 32. Both continue to decrease so I know it is not overfitting
and continuing training will reduce this a bit.

On other data sets, the training loss continues to decrease but
validation loss starts increasing - based on other forum questions, that
is overfitting, is my understanding correct?

[This is an archived TTS discussion thread from discourse.mozilla.org/t/continuing-training-saving-new-best-checkpoint]

[JRMeyer]

>>> othiele
[January 11, 2021, 8:20am]

> The one big question I have is that if we are 'continuing' training,
> why is the new best validating model saved if it is not better than
> the one used in the previous run?

Check your checkpoint dir. It has a text file that states what the last
saved and best model are.

> The other question I have is what techniques can we use to decrease
> this loss rate?

More material, maybe augmentation.

> How much difference would one form of augmentation alone make to our
> data? Or would it be more useful to use multiple augmentations
> together in the same run?

Augmentation is quite new, search the forum and experiment.

> On other data sets, the training loss continues to decrease but
> validation loss starts increasing - based on other forum questions,
> that is overfitting, is my understanding correct?

Yes, this is typical for fresh trainings. Look at the es_epochs flag.

And maybe use reduce_lr_on_plateau . In general, 60 hours might not be
enough to transfer to a WER of 15. dan.bmh, for example, uses 2000 hours
to transfer.

[Archived Post]

[JRMeyer]

>>> othiele
[January 11, 2021, 9:54am]

ah, and why do you drop 2 layers?

[Archived Post]

[JRMeyer]

>>> abdullah.tayyab
[January 11, 2021, 10:14am]

Because many posts referencing transfer learning have done the same. The
Urdu text is with a completely new alphabet so I am presuming that is
the way to go?

[Archived Post]

[JRMeyer]

>>> othiele
[January 11, 2021, 10:48am]

Please read and understand [the
#transfer-learning-new-alphabet).
By scraping 2 layers off the model you loose quite a lot of trained
weights. The more you leave out, the more you have to retrain. You might
be better off with dropping just 1 layer. I usually have the same
alphabet, but with such few data as you have, it might yield better
results.

But in the end you'll have to use more data. Think of 600 instead of 60
hours.

[Archived Post]