architectureIntroduced by Transformer-XL · 2019
Segment-level recurrence & relative positions
Reuse hidden states from earlier segments so attention can reach beyond a fixed context.
Drafted by AI · not yet reviewed
How this idea evolved
Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that Transformer-XL cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.
Every token attends to every other token, weighted by learned query–key similarity.
Add fixed sine/cosine signals so an order-blind Transformer knows token positions.
A sequence model built only from attention and feed-forward layers, with no recurrence.