2022-ACL 2022-Understanding and Improving Sequence-to-Sequence Pretraining for Neural Machine Translation

[thangk]

Link: arXiv

Main problem

Previous encoder-only pertaining approaches produce low translation quality and induce over-estimation issues, and model robustness is not there.

Proposed method

This paper proposes a simple strategy to overcome the limitations of the main problem, via two key components: in-domain pertaining and input adaptation.

My Summary

The author took the approach to jointly pretrain the decoder which produced a more diverse translation in the experiments and also reduced adequacy-related translation errors compared to the encoder-only pretaining approach. After applying the proposed method, the author observed up to 19% improvement in performance in some cases (W19 EN->DE). The end result is improved translation performance and model robustness. The future works include validating the findings with more Seq2Seq pertaining models and language pairs.

Datasets

(1) WMT19 English-German
(2) WMT16 English-Romanian (low resource)
(3) IWSLT17 English-French
(4) a subset from WMT19 English-German (for ablation studies)