Align before fuse

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

1. 2018
   CPCContrastive predictive coding / InfoNCE

   Predict future latents and score them against negatives with a contrastive loss.

   Cites · not yet reviewedcited 1× · §Methods

   “This loss takes the same form as the InfoNCE loss Oord et al. 2018, and minimizing it leads to encoders that maximally preserve the mutual information between the true pairs under the representation functions.”

2. 2020
   ConVIRTContrastive image–text learning

   Pull matching image and text embeddings together and push mismatched pairs apart.

   Also draws on: Joint image–text embedding (Deep Fragment Embeddings)

   No direct link between these papers in this dataset
3. 2021
   ALBEFAlign before fuse

   Align unimodal embeddings contrastively before fusing them with cross-attention.

   Also draws on: Two-stream co-attention (ViLBERT)