Masked language modelling

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

1. 2014
   Bahdanau attentionAttention mechanism

   Let a decoder look back at the most relevant input positions instead of one fixed vector.

   Cites · not yet reviewedcited 5× · §Model Architecture

   “Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].”

2. 2017
   TransformerTransformer

   A sequence model built only from attention and feed-forward layers, with no recurrence.

   Also draws on: Encoder–decoder seq2seq (Seq2Seq)

   Cites · not yet reviewedcited 4× · §BERT

   “BERT’s model architecture is a multi-layer bidirectional Transformer encoder based on the original implementation described in Vaswani et al. 2017 and released in the tensor2tensor library.11 1 https://github.com/tensorflow/tensor2tensor Because the use of Transformers has become common and our implementation is almost identical to the original, we will omit an exhaustive background description of the model architecture and refer readers to Vaswani et al. 2017 as well as excellent guides such as ‘‘The Annotated Transformer.’’22 2 http://nlp.seas.harvard.edu/2018/04/03/attention.html”

3. 2018
   BERTMasked language modelling

   Hide some tokens and predict them from context on both sides.