training techniqueIntroduced by FLAN · 2021
Instruction tuning
Fine-tune on many tasks phrased as instructions, so the model follows new instructions zero-shot.
Drafted by AI · not yet reviewed
How this idea evolved
Each step is the idea’s introducing paper. The line between steps is the citation link between those papers.
- 2019
Cast every task as text in, text out: one model, one loss.
Cites · not yet reviewedcited 3× · §FLAN: Instruction Tuning Improves Zero-Shot Learning“To balance the different sizes of datasets, we limit the number of training examples per dataset to 30k and follow the examples-proportional mixing scheme (Raffel et al. 2020) with a mixing rate maximum of 3k.22 2 In this mixing scheme, a mixing rate maximum of 3,000 means that a dataset does not receive additional sampling weight for examples in excess of 3,000.”
From FLAN · §FLAN: Instruction Tuning Improves Zero-Shot Learning - 2021
Fine-tune on many tasks phrased as instructions, so the model follows new instructions zero-shot.
Papers using this
- 2022PaLM
- 2022Flan-T5 / Flan-PaLM