Learning to summarize from human feedback
As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality.
Also cited · not yet reviewed (5)
- RLHF for LMs (Ziegler)2019 · cited 7דWe follow the works of [3, 73], who fine-tune language models from human feedback using reward learning [35].”From this paper · §unknown section
- PPO2017 · cited 3דFinally, we train a policy via reinforcement learning (RL) to maximize the score given by the RM; the policy generates a token of text at each ‘time step’, and is updated using the PPO algorithm [58] based on the RM ‘rew…”From this paper · §unknown section
- GPT-32020 · cited 3דWe follow the works of [48, 4] and use large pretrained GPT-3 models with as many as 6.7 billion parameters.”From this paper · §unknown section
- Transformer2017 · cited 2דOur work is most similar to [73], who also train Transformer models [62] to optimize human feedback across a range of tasks, including summarization on the Reddit TL;DR and CNN/DM datasets.”From this paper · §unknown section
Show 1 more
- T52019 · cited 2דFinally, there has been extensive research on modifying architectures [22, 59] and pre-training procedures [70, 36, 49, 60, 53, 14] for improving summarization performance.”From this paper · §unknown section
Led to
- Recursive book summarization2021 · cited 8דFor training the model, we most closely follow the procedure of Stiennon et al., 2020.”From Recursive book summarization · §unknown section
- InstructGPT2022 · cited 10×, 6 in Method“Compared to earlier work that collects human preference data on the task of summarization (Ziegler et al., 2019; Stiennon et al., 2020; Wu et al., 2021), our inputs span a much broader range of tasks, and can occasionall…”From InstructGPT · §Methods and experimental details
Abstract
As language models become more powerful, training and evaluation are increasingly bottlenecked by the data and metrics used for a particular task. For example, summarization models are often trained to predict human reference summaries and evaluated using ROUGE, but both of these metrics are rough proxies for what we really care about -- summary quality. In this work, we show that it is possible to significantly improve summary quality by training a model to optimize for human preferences. We collect a large, high-quality dataset of human comparisons between summaries, train a model to predict the human-preferred summary, and use that model as a reward function to fine-tune a summarization policy using reinforcement learning. We apply our method to a version of the TL;DR dataset of Reddit posts and find that our models significantly outperform both human reference summaries and much larger models fine-tuned with supervised learning alone. Our models also transfer to CNN/DM news articles, producing summaries nearly as good as the human reference without any news-specific fine-tuning. We conduct extensive analyses to understand our human feedback dataset and fine-tuned models We establish that our reward model generalizes to new datasets, and that optimizing our reward model results in better summaries than optimizing ROUGE according to humans. We hope the evidence from our paper motivates machine learning researchers to pay closer attention to how their training loss affects the model behavior they actually want.