training techniqueIntroduced by DeiT · 2020

Distillation token (data-efficient ViT)

Train ViTs on ImageNet alone by adding a token that learns from a CNN teacher.

Drafted by AI · not yet reviewed

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

2014
Bahdanau attention Attention mechanism
Let a decoder look back at the most relevant input positions instead of one fixed vector.
Cites · not yet reviewedcited 5× · §Model Architecture
“Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].”
From Transformer · §Model Architecture
2017
Transformer Transformer
A sequence model built only from attention and feed-forward layers, with no recurrence.
Also draws on: Encoder–decoder seq2seq (Seq2Seq)
Cites · not yet reviewedcited 4× · §Method
“In model design we follow the original Transformer (Vaswani et al. 2017) as closely as possible.”
From ViT · §Method
2020
ViT Vision Transformer (ViT)
Split an image into patches and feed them to a plain Transformer as tokens.
Cites · not yet reviewedcited 12× · §Introduction
“We build upon the visual transformer architecture from Dosovitskiy et al. [15] and improvements included in the timm library [55].”
From DeiT · §Introduction
2020
DeiT Distillation token (data-efficient ViT)
Train ViTs on ImageNet alone by adding a token that learns from a CNN teacher.
Also draws on: Knowledge distillation (Knowledge Distillation)