objectiveIntroduced by BYOL · 2020
Self-distillation without negatives
An online network predicts a slowly-moving target network's view, with no negatives needed.
Drafted by AI · not yet reviewed
How this idea evolved
Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that BYOL cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.
Strong augmentation plus a nonlinear projection head is what makes contrastive learning work.
A queue of negatives encoded by a slowly-updated momentum encoder.
Predict future latents and score them against negatives with a contrastive loss.
Papers using this
- 2021ViT-VQGAN