PrefixLM vision-language pre-training

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

1. 2019
   UniLMUnified LM via attention masks

   One Transformer does bidirectional, left-to-right and seq2seq modelling by switching attention masks.

   No direct link between these papers in this dataset
2. 2021
   SimVLMPrefixLM vision-language pre-training

   One generative prefix-LM objective on weakly aligned pairs replaces many task-specific losses.