Unified LM via attention masks

How this idea evolved

Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that UniLM cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.

• Masked language modellingBERT · 2018cited directly

  Hide some tokens and predict them from context on both sides.

• Deep bidirectional LM featuresELMo · 2018cited directly

  Use all internal layers of a pretrained bidirectional LM as features for downstream tasks.

• Contextual word vectorsLearned in Translation · 2017two citations away

  Word vectors that depend on the sentence, taken from a pretrained encoder.