Perceiver resampler

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

1. 2014
   Bahdanau attentionAttention mechanism

   Let a decoder look back at the most relevant input positions instead of one fixed vector.

   Cites · not yet reviewedcited 5× · §Model Architecture

   “Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].”

2. 2017
   TransformerSelf-attention

   Every token attends to every other token, weighted by learned query–key similarity.

   Cites · not yet reviewedcited 3× · §Introduction

   “Transformer, (Vaswani et al. 2017)).”

3. 2018
   Set TransformerInducing-point attention (learned latents)

   A few learned vectors attend to a large set and summarise it: the idea behind later resamplers and query transformers.

   Cites · not yet reviewedcited 2× · §Methods

   “Attention is a permutation-invariant operation, and this property is preserved by the Perceiver and related models (Lee et al. 2019).”

4. 2021
   PerceiverLatent-array cross-attention (Perceiver)

   A small latent array cross-attends to huge inputs, decoupling depth from input size.

   Also draws on: Self-attention (Transformer)

   Cites · not yet reviewedcited 2× · §Approach

   “Similar to Perceiver [48] and DETR [13], we learn a predefined number of latent input queries which are fed to a Transformer and cross-attend to the visual features.”

5. 2022
   FlamingoPerceiver resampler

   Learned queries cross-attend to many visual features and compress them to a fixed number of tokens.