What is Direct Preference Optimization (DPO)?

Large language models learn from vast amounts of text, but making them behave in ways that match human preferences is hard, because it is difficult to write down a precise rule for what a good answer is. A common approach, reinforcement learning from human feedback (RLHF), first trains a separate reward model (a second model that learns to score answers the way people would) and then uses reinforcement learning to tune the main model to earn high scores from it, a complex and often unstable process. Direct Preference Optimization (DPO) is a training method that instead tunes a language model directly on human judgments of which of two answers to the same request is better, without needing an explicit reward model. It was introduced in 2023 in a paper by Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn.

Written from primary sources and reference pages, and checked against them by an independent AI fact-checker. Drafted with AI assistance.