Reinforcement Learning from Human FeedbackRLHF

Training AI to be helpful and safe by learning from human preference ratings.

RLHF is the training technique that transforms a raw pre-trained LLM into a helpful, harmless assistant. It has three stages:

Supervised Fine-Tuning (SFT): Fine-tune the base model on high-quality human-written demonstrations of good responses.
Reward Model Training: Show human raters pairs of model outputs for the same prompt; they choose which is better. Train a separate "reward model" to predict these preferences.
RL Optimisation: Use the reward model as a feedback signal to further train the LLM via Proximal Policy Optimisation (PPO) — pushing the model toward responses humans prefer.

Modern variants:

DPO (Direct Preference Optimisation) — skips the explicit reward model, more stable
Constitutional AI (Anthropic) — uses AI-generated feedback against written principles, scales better than human raters
RLAIF — replaces human raters with AI raters entirely

ChatGPT, Claude, and Gemini all use variants of RLHF. Without it, LLMs are surprisingly bad at following instructions and often generate harmful content.

In plain terms

Teaching a new employee by having experienced colleagues rate their work, then using those ratings to guide further training — rather than writing out every rule explicitly.

Learn more

Related concepts

🎯

Fine-tuning

Specialising a general AI model on your specific data and style.

📜

Constitutional AI

Teaching AI to critique and revise its own outputs against a written set of principles.

⬡

Large Language Model

AI trained on vast text to understand and generate language.

🛡️

AI Safety

The field working to ensure AI systems do what humans actually want — now and as they become more capable.