Two-stage vision-to-language bootstrapping

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

1. 2021
   FrozenFrozen language model

   Keep a pretrained LM fixed and teach only a small module to feed it new information.

   Cites · not yet reviewedcited 3× · §Introduction

   “Frozen (Tsimpoukelli et al. 2021), Flamingo (Alayrac et al. 2022)) resort to an image-to-text generation loss, which we show is insufficient to bridge the modality gap.”

2. 2023
   BLIP-2Two-stage vision-to-language bootstrapping

   First align the bridge with vision and text, then teach it to talk to a frozen LLM.

   Also draws on: Locked image tuning (LiT)