Momentum distillation

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

1. 2019
   MoCoMomentum contrast

   A queue of negatives encoded by a slowly-updated momentum encoder.

   Cites · not yet reviewedcited 3× · §ALBEF Pre-training

   “Inspired by MoCo [24], we maintain two queues to store the most recent MM image-text representations from the momentum unimodal encoders.”

2. 2021
   ALBEFMomentum distillation

   Learn from soft targets produced by a moving-average model, tolerating noisy web pairs.

   Also draws on: Knowledge distillation (Knowledge Distillation)