Denoising seq2seq pre-training

How this idea evolved

Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that BART cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.

• Permutation language modellingXLNet · 2019cited directly

  Autoregressive training over all factorisation orders, capturing bidirectional context without masks.

• Masked language modellingBERT · 2018cited directly

  Hide some tokens and predict them from context on both sides.

• Robustly optimised BERT recipeRoBERTa · 2019cited directly

  Train BERT longer, on more data, with bigger batches and dynamic masking.