True Few-Shot Learning with Language Models
Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning.
Also cited · not yet reviewed (3)
- GPT-32020 · cited 11×, 2 in Method“Recent work does not assume access to data from other distributions, performing few-shot learning using only a few examples from a single distribution to update a pretrained LM [2, 12].”From this paper · §Can We Do Model Selection in Few-Shot Learning?
- LPAQA (What LMs know)2019 · cited 4דHowever, the few-shot performance of LMs is very sensitive to the textual task description [3, 4, 5, 6, “prompt”;], order of training examples [6, 7, 8], decoding strategy [9, 10], and other hyperparameters [3, 5, 9, 11,…”From this paper · §Introduction
- CLIP2021 · cited 2דPrior work uses large train or held-out sets with many examples to choose prompts [2, 12, 13] and hyperparameters [12].”From this paper · §Introduction
Led to
- T02021 · cited 3×, 2 in Method“However, this ability requires a sufficiently large model and is sensitive to the wording of its prompts (Perez et al. 2021; Zhao et al. 2021; Reynolds and McDonell 2021).”From T0 · §Introduction
Abstract
Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly-selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.