architectureIntroduced by Switch Transformer · 2021
Top-1 (switch) routing
Send each token to a single expert, simplifying MoE and scaling to trillions of parameters.
Drafted by AI · not yet reviewed
How this idea evolved
Suggested from citations. No curator has recorded what this idea builds on yet. These ideas from the same theme come from papers that Switch Transformer cites, directly or one step removed. Citation is a fact; the connection between the ideas is not verified.
Route each token to a few of many expert feed-forward layers: huge capacity, constant compute.
Papers using this
- 2022Chinchilla