Direct Preference OptimisationDPO

A simpler alternative to RLHF that trains models on human preferences without a reward model.

DPO (Direct Preference Optimisation, Rafailov et al., 2023) is a training algorithm that fine-tunes LLMs on human preference data — chosen vs. rejected response pairs — without training a separate reward model or running reinforcement learning.

Why it improves on RLHF: Standard RLHF has three stages (SFT → reward model → PPO optimisation), each with its own instabilities. DPO collapses this into a single supervised fine-tuning step by directly optimising the implicit reward defined by the optimal policy.

In practice: Given a prompt with a preferred response (chosen) and a less preferred response (rejected), DPO increases the likelihood of the chosen response relative to the rejected one — the math works out to be equivalent to RLHF under certain assumptions.

Why practitioners care:

Simpler to implement and tune than PPO-based RLHF
Requires only paired preference data, not a running reward model
More training stable — fewer hyperparameters to manage
Has become the dominant preference optimisation method for open-source fine-tuning (used heavily with Llama, Mistral fine-tunes)

Variants: IPO, KTO, ORPO — each with slightly different objectives addressing edge cases in DPO's derivation.

In plain terms

Instead of hiring a separate judge to score essays and then training a writer to please that judge, you just directly show the writer: here are two essays; write more like the first one.

Learn more

Related concepts

🎓

Reinforcement Learning from Human Feedback

Training AI to be helpful and safe by learning from human preference ratings.

🎯

Fine-tuning

Specialising a general AI model on your specific data and style.

📜

Constitutional AI

Teaching AI to critique and revise its own outputs against a written set of principles.

⬡

Large Language Model

AI trained on vast text to understand and generate language.