optimizationIntroduced by GPipe · 2018
Pipeline parallelism
Split layers across devices and stream micro-batches through them.
Drafted by AI · not yet reviewed
How this idea evolved
No earlier ideas recorded for this concept yet.
Papers using this
- 2019Megatron-LM
- 2022Megatron-Turing NLG
- 2022PaLM