Zero-shot transfer via text prompts

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

1. 2018
   CPCContrastive predictive coding / InfoNCE

   Predict future latents and score them against negatives with a contrastive loss.

   Cites · not yet reviewedcited 1× · §Methods

   “This loss takes the same form as the InfoNCE loss Oord et al. 2018, and minimizing it leads to encoders that maximally preserve the mutual information between the true pairs under the representation functions.”

2. 2020
   ConVIRTContrastive image–text learning

   Pull matching image and text embeddings together and push mismatched pairs apart.

   Also draws on: Joint image–text embedding (Deep Fragment Embeddings)

   Cites · not yet reviewedcited 5× · §Approach

   “Zhang et al. 2020, Gomez et al. 2017, Joulin et al. 2016, and Desai & Johnson 2020 all introduce methods which learn visual representations from text paired with images but describe their approaches as unsupervised, self-supervised, weakly supervised, and supervised respectively.”

3. 2021
   CLIPZero-shot transfer via text prompts

   Classify into unseen classes by comparing an image with text descriptions of them.