How are multiple datasets combined during training?

[DomhnallBoyle]

Hi, thanks for this great work.

I have a question about the section "3.8 Using Additional Training Data" from your paper "Visual Speech Recognition for Multiple Languages in the Wild"

For example, for LRS3 the best WER of 32.1 is achieved by combining the datasets LRW + LRS2 + AVSpeech + LRS3. I was just wondering what way they're combined during training, which of the scenarios would be correct?

Scenario A:

1. Pretrain using LRW + LRS2 + AVSpeech datasets
2. Initialise from 1 above, then train on the LRS3 dataset only

Scenario B:

1. Pretrain using LRW + LRS2 + AVSpeech datasets
2. Initialise from 1 above, then train on LRW + LRS2 + AVSpeech + LRS3 datasets

Would there be a performance difference between these 2 scenarios?

Thanks