Contrastive + captioning in one model

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

1. 2018
   CPCContrastive predictive coding / InfoNCE

   Predict future latents and score them against negatives with a contrastive loss.

   Cites · not yet reviewedcited 1× · §Methods

   “This loss takes the same form as the InfoNCE loss Oord et al. 2018, and minimizing it leads to encoders that maximally preserve the mutual information between the true pairs under the representation functions.”

2. 2020
   ConVIRTContrastive image–text learning

   Pull matching image and text embeddings together and push mismatched pairs apart.

   Also draws on: Joint image–text embedding (Deep Fragment Embeddings)

   No direct link between these papers in this dataset
3. 2022
   CoCaContrastive + captioning in one model

   One image–text encoder–decoder trained with both a contrastive and a captioning loss.

   Also draws on: PrefixLM vision-language pre-training (SimVLM)