Gated feed-forward variants (SwiGLU)

How this idea evolved

Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that GLU variants cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.

• Self-attentionTransformer · 2017cited directly

  Every token attends to every other token, weighted by learned query–key similarity.

• Sinusoidal positional encodingTransformer · 2017cited directly

  Add fixed sine/cosine signals so an order-blind Transformer knows token positions.

• TransformerTransformer · 2017cited directly

  A sequence model built only from attention and feed-forward layers, with no recurrence.