architectureIntroduced by Transformer · 2017
Sinusoidal positional encoding
Add fixed sine/cosine signals so an order-blind Transformer knows token positions.
Drafted by AI · not yet reviewed
How this idea evolved
Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that Transformer cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.
Let a decoder look back at the most relevant input positions instead of one fixed vector.
Papers using this
- 2018Set Transformer
- 2019Transformer-XL
- 2019T5
- 2020DeiT
- 2021RoFormer (RoPE)
- 2021Gopher