Cross-Task Generalization via Natural Language Crowdsourcing Instructions
, crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. , a question-answering system cannot solve classification tasks).
Also cited · not yet reviewed (2)
- BART2019 · cited 3×, 2 in Method“We use BART (base) Lewis et al. 2019 which allows us to fine-tune its model parameters.”From this paper · §Problem Setup and Models
- GPT-32020 · cited 3×, 2 in Method“We build models using pre-trained LMs with encoder-decoder architectures BART Lewis et al. 2019 for fine-tuning and GPT3 Brown et al. 2020 for few-shot experiments.”From this paper · §Problem Setup and Models
Led to
- Super-NaturalInstructions2022 · cited 6דRecent literature has been motivated by building models that are generalizable across a variety of NLP tasks, when prompted with either a few examples Ye and Ren 2021; Bragg et al. 2021 or language definitions Efrat and…”From Super-NaturalInstructions · §Related Work
- BIG-bench2022 · cited 2דThis project is an expansion of NATURAL INSTRUCTIONS (Mishra et al. 2021), which contained a collection of 61 tasks.”From BIG-bench · §Additional related work
Abstract
Humans (e.g., crowdworkers) have a remarkable ability in solving different tasks, by simply reading textual instructions that define them and looking at a few examples. Despite the success of the conventional supervised learning on individual datasets, such models often struggle with generalization across tasks (e.g., a question-answering system cannot solve classification tasks). A long-standing challenge in AI is to build a model that learns a new task by understanding the human-readable instructions that define it. To study this, we introduce NATURAL INSTRUCTIONS, a dataset of 61 distinct tasks, their human-authored instructions, and 193k task instances (input-output pairs). The instructions are obtained from crowdsourcing instructions used to create existing NLP datasets and mapped to a unified schema. Using this meta-dataset, we measure cross-task generalization by training models on seen tasks and measuring generalization to the remaining unseen ones. We adopt generative pre-trained language models to encode task-specific instructions along with input and generate task output. Our results indicate that models benefit from instructions when evaluated in terms of generalization to unseen tasks (19% better for models utilizing instructions). These models, however, are far behind an estimated performance upperbound indicating significant room for more progress in this direction.