Measuring Massive Multitask Language Understanding
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more.
Also cited · not yet reviewed (2)
- GPT-32020 · cited 4דHowever, larger pretrained models like GPT-3 (Brown et al. 2020) have made it possible to achieve competitive performance without fine-tuning by using few-shot learning, which removes the need for a large fine-tuning set…”From this paper · §Related Work
- T52019 · cited 2דUnifiedQA uses the T5 (Raffel et al. 2019) text-to-text backbone and is fine-tuned on previously proposed question answering datasets (Lai et al. 2017), where the prediction is the class with the highest token overlap wi…”From this paper · §Experiments
Led to
- Chinchilla2022 · cited 2דWe thus place more emphasis on other tasks for which leakage is less of a concern, such as MMLU (Hendrycks et al. 2020) and BIG-bench (BIG-bench collaboration 2021) along with various closed-book question answering and c…”From Chinchilla · §Chinchilla
- GPT-NeoX-20B2022 · cited 3דTo do this, we use a dataset of multiple choice questions in a variety of diverse domains developed by Hendrycks et al. 2021a.”From GPT-NeoX-20B · §Performance Evaluations
- BIG-bench2022 · cited 5דNew evaluations can be contributed as test suits on their project website.55 5 https://syntaxgym.org/ The Massive Multitask Language Understanding (MMLU) benchmark (Hendrycks et al. 2021b) is a collection of 57 diverse t…”From BIG-bench · §Additional related work
- Flan-T5 / Flan-PaLM2022 · cited 4דInstead, we use the following challenging benchmarks, for which current language models still perform well below expert human raters. (1) MMLU (Hendrycks et al. 2020) includes exam questions from 57 tasks such as mathema…”From Flan-T5 / Flan-PaLM · §unknown section
Abstract
We propose a new test to measure a text model's multitask accuracy. The test covers 57 tasks including elementary mathematics, US history, computer science, law, and more. To attain high accuracy on this test, models must possess extensive world knowledge and problem solving ability. We find that while most recent models have near random-chance accuracy, the very largest GPT-3 model improves over random chance by almost 20 percentage points on average. However, on every one of the 57 tasks, the best models still need substantial improvements before they can reach expert-level accuracy. Models also have lopsided performance and frequently do not know when they are wrong. Worse, they still have near-random accuracy on some socially important subjects such as morality and law. By comprehensively evaluating the breadth and depth of a model's academic and professional understanding, our test can be used to analyze models across many tasks and to identify important shortcomings.