Fine-Tuning Language Models from Human Preferences
Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks.
Also cited · not yet reviewed (3)
- BookCorpus (books & movies)2015 · cited 2×, 1 in Method“For stylistic continuation tasks we perform supervised fine-tuning of the language model to the BookCorpus dataset of Zhu et al. 2015 prior to RL fine-tuning; we train from scratch on WebText, supervised fine-tune on Boo…”From this paper · §Methods
- Transformer2017 · cited 1×, 1 in Method“The model is a Transformer with 36 layers, 20 heads, and embedding size 1280 [Vaswani et al. 2017].”From this paper · §Methods
- PPO2017 · cited 1×, 1 in Method“Train π\pi via Proximal Policy Optimization (PPO, Schulman et al. 2017) with reward RR from eq. 2 on x∼𝒟x\simD.”From this paper · §Methods
Led to
- GPT-32020 · cited 2דOne direction for future work might be attempting to generate a broader set of explicit tasks for multi-task learning, for example through procedural generation [128], human interaction [144], or active learning [80].”From GPT-3 · §Related Work
- Learning to summarize from human feedbac2020 · cited 7דWe follow the works of [3, 73], who fine-tune language models from human feedback using reward learning [35].”From Learning to summarize from human feedbac · §unknown section
- Recursive book summarization2021 · cited 3דThis has been applied in many domains including summarization (Böhm et al., 2019; Ziegler et al., 2019; Stiennon et al., 2020), dialogue (Jaques et al., 2019; Yi et al., 2019; Hancock et al., 2019), translation (Kreutzer…”From Recursive book summarization · §unknown section
- InstructGPT2022 · cited 5×, 2 in Method“Compared to earlier work that collects human preference data on the task of summarization (Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021), our inputs span a much broader range of tasks, and can occasionall…”From InstructGPT · §Methods and experimental details
Abstract
Reward learning enables the application of reinforcement learning (RL) to tasks where reward is defined by human judgment, building a model of reward by asking humans questions. Most work on reward learning has used simulated environments, but complex information about values is often expressed in natural language, and we believe reward learning for language is a key to making RL practical and safe for real-world tasks. In this paper, we build on advances in generative pretraining of language models to apply reward learning to four natural language tasks: continuing text with positive sentiment or physically descriptive language, and summarization tasks on the TL;DR and CNN/Daily Mail datasets. For stylistic continuation we achieve good results with only 5,000 comparisons evaluated by humans. For summarization, models trained with 60,000 comparisons copy whole sentences from the input but skip irrelevant preamble; this leads to reasonable ROUGE scores and very good performance according to our human labelers, but may be exploiting the fact that labelers rely on simple heuristics.