What is RLHF: The Key Step from LLMs That Talk to LLMs That Talk Well
Why Did Early GPT Say Whatever It Wanted? In 2020, OpenAI released GPT-3 with 175 billion parameters — it could write poetry, code, and translate. But there …
Why Did Early GPT Say Whatever It Wanted?
In 2020, OpenAI released GPT-3 with 175 billion parameters — it could write poetry, code, and translate. But there was a problem: ask it something dangerous, it would answer. Give it an ambiguous prompt, it would improvise. Ask it to draft an email, you would not recognize the style.
The reason is straightforward: language model training optimizes for "predict the next token," not "give a useful answer." Models learn to continue sentences from internet text — and the internet is full of bias, errors, and harmful content. Good prediction ≠ good behavior.
That is what RLHF is designed to fix.
RLHF in Three Steps
RLHF stands for Reinforcement Learning from Human Feedback. It is not a model — it is a training pipeline with three stages:
Step 1: Supervised Fine-Tuning (SFT)
Start with a set of high-quality question-answer examples and fine-tune the pretrained model with supervision. Human annotators write the ideal responses. This turns the model from a continuation machine into a dialogue assistant — but the notion of "good" has not been formalized yet.
Step 2: Train a Reward Model
For a given question, have the SFT model generate several responses (say 4). Human annotators rank these responses from best to worst. These rankings train a reward model that learns to predict "what score would a human give this response." The reward model acts as an automated judge, having internalized thousands of human judgments.
Step 3: PPO Reinforcement Learning
Use the reward model as an environment and PPO (Proximal Policy Optimization) to train the dialogue model: generate a response → reward model scores it → model updates parameters → repeat. A KL divergence constraint prevents the model from gaming the reward by always giving very short answers.
A Concrete Example: "How Do I Hack a WiFi Network"
Ask vanilla GPT-3 and it will give you step-by-step instructions. Ask RLHF-trained ChatGPT and it will tell you this is illegal, then pivot to network security fundamentals. The reward model saw many examples of dangerous answers marked low by annotators, so it learned: dangerous content → low score. PPO teaches the model to avoid this territory.
RLHF Limitations: Pleasing Humans ≠ Being Correct
- Reward Hacking: Models learn to appease annotators, not actually be right. If annotators prefer long answers, the model starts padding responses.
- Annotator Bias: Data quality determines reward model quality. Annotators carry their own biases into the model.
- Poor Out-of-Distribution Generalization: The reward model may score novel questions unpredictably.
- High Cost: Quality annotation is expensive — OpenAI RLHF annotators reportedly earned over $30/hr.
After RLHF: DPO, RLAIF, Constitutional AI
DPO (Direct Preference Optimization): Removes the reward model entirely. Preference data directly fine-tunes the model. Simpler and more stable. Used in LLaMA 2, Mistral series.
RLAIF (Reinforcement Learning from AI Feedback): Uses a more capable AI (like Claude) instead of human annotators to judge preferences, dramatically reducing annotation costs. This is how Anthropic trains Claude.
Constitutional AI: Gives the model a set of principles and has it self-critique and self-revise its outputs, reducing dependence on large-scale human annotation.
SFD Lab Note
Running many local model comparisons in our lab, the same Qwen3.5 35B base vs. RLHF-aligned differs noticeably in content creation tasks. The base model is "wilder" — occasionally surprising, but more prone to going off-track. The aligned version is steadier, with clearer guardrails. For 15 agents running in parallel, stability matters more than raw creativity. RLHF-induced domestication is a feature here, not a bug.