Robustly optimised BERT recipe

How this idea evolved

Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that RoBERTa cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.

• Masked language modellingBERT · 2018cited directly

  Hide some tokens and predict them from context on both sides.

• Permutation language modellingXLNet · 2019cited directly

  Autoregressive training over all factorisation orders, capturing bidirectional context without masks.

• Deep bidirectional LM featuresELMo · 2018cited directly

  Use all internal layers of a pretrained bidirectional LM as features for downstream tasks.