模型微调与人类反馈
We’ve fine-tuned the 774M parameter GPT-2 language model using human feedback for various tasks, successfully matching the preferences of the external human labelers, though those preferences did not always match our own.
摘要任务中的行为偏差
Specifically, for summarization tasks the labelers preferred sentences copied wholesale from the input (we’d only asked them to ensure accuracy), so our models learned to copy.
数据规模与任务复杂度
Summarization required 60k human labels; simpler tasks which continue text in various styles required only 5k.
研究动机与安全目标
Our motivation is to move safety techniques closer to the general task of “machines talking to humans,” which we believe is key to extracting information about human values.