architectureIntroduced by Perceiver · 2021

Latent-array cross-attention (Perceiver)

A small latent array cross-attends to huge inputs, decoupling depth from input size.

Drafted by AI · not yet reviewed

How this idea evolved

Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.

2014
Bahdanau attention Attention mechanism
Let a decoder look back at the most relevant input positions instead of one fixed vector.
Cites · not yet reviewedcited 5× · §Model Architecture
“Most competitive neural sequence transduction models have an encoder-decoder structure [5, 2, 35].”
From Transformer · §Model Architecture
2017
Transformer Self-attention
Every token attends to every other token, weighted by learned query–key similarity.
Cites · not yet reviewedcited 3× · §Introduction
“Transformer, (Vaswani et al. 2017)).”
From Set Transformer · §Introduction
2018
Set Transformer Inducing-point attention (learned latents)
A few learned vectors attend to a large set and summarise it: the idea behind later resamplers and query transformers.
Cites · not yet reviewedcited 2× · §Methods
“Attention is a permutation-invariant operation, and this property is preserved by the Perceiver and related models (Lee et al. 2019).”
From Perceiver · §Methods
2021
Perceiver Latent-array cross-attention (Perceiver)
A small latent array cross-attends to huge inputs, decoupling depth from input size.
Also draws on: Self-attention (Transformer)