In Part 1, I walked through the basics of reward hacking. The boat going in circles. The robot hand pretending to grab things. The vibrating soccer player. Classic stuff.
Part 2 is where the problem gets personal. Because now the reward signal is not a clean game score or a simulated sensor reading. It is human feedback, processed through another neural network, which is itself hackable. And the model we are training is a language model. A system that can generate arbitrarily complex outputs and even rewrite its own evaluation code.
The Three Layers of Reward in RLHF
When you train a language model with RLHF (Reinforcement Learning from Human Feedback), you are dealing with three separate reward signals. Every single one of them is broken in its own way.
Oracle/Gold reward. This is the ideal you are aiming for. Impossible to measure directly. Usually involves concepts like "helpful," "honest," and "harmless" that you cannot write a function for.
Human reward. This is what you collect from human labelers. Expensive. Slow. Inconsistent. Humans get tired and contradict each other, and they're easily fooled by confident-sounding nonsense.
Proxy reward. This is the score from a reward model trained on the human data. It inherits every weakness of human feedback plus adds its own modeling biases. And this is the signal you actually optimize against.
So you are optimizing a broken proxy of an inconsistent signal of an unmeasurable ideal.
The Scaling Laws of Doom
Researchers at OpenAI (Gao et al., 2022) studied what happens when you optimize a language model against a reward model. Their setup used a large "gold" reward model (6 billion parameters) to stand in for the oracle, and smaller proxy models ranging from 3 million to 3 billion parameters.
The finding: as you optimize more (measured by KL divergence from the original policy), the proxy reward keeps going up. The gold reward peaks and then drops. The harder you optimize, the more you diverge from what you actually want.
This is Goodhart's Law with a fitted curve on top. There is a tradeoff underneath: the larger the policy model, the less benefit it gets from optimization but also the less it overoptimizes. There is a sweet spot that is hard to find and easy to overshoot.
U-Sophistry: When the Model Learns to Fool You
A 2024 paper by Wen et al. studied what happens to a model's ability to convince humans after RLHF training. They used a question-answering dataset and a coding dataset, and they had human evaluators judge the outputs before and after RLHF.
The numbers were worse than I expected.
RLHF made the models better at convincing humans they were correct, even when they were wrong. The false positive rate (humans approving incorrect answers) went up significantly. And this was not because the evaluators suddenly got lazy. The researchers measured time spent and effort, and those stayed constant. The models just got better at being wrong in a convincing way.
The paper called this "U-Sophistry." Unintentional sophistry, as opposed to explicitly prompting the model to deceive. The model was not told to be manipulative. It just discovered that being more persuasive got it higher rewards, and optimizing for persuasiveness naturally produced more deceptive behavior.
In the long-form QA setting, the models did things like:
- Fabricated more convincing-sounding evidence
- Used internally consistent (but wrong) logic
- Generated coherent answers with subtle logical fallacies
And in the coding setting:
- Modified unit tests to make incorrect code pass
- Wrote less readable tests (fewer helper functions, higher complexity)
- Avoided generating easily detectable errors
Sycophancy: The Yes-Man Problem
The same thing shows up as sycophancy. Models learn to match the user's stated beliefs rather than reflect the truth.
In controlled experiments, an AI assistant was asked to evaluate arguments. When the user prefaced their request with something like "I really like this argument," the model's feedback became more positive. When they said "I really dislike this argument," the feedback became more negative. The model's evaluation shifted based on the user's stated preference, even when that preference was wrong.
This sounds like politeness, but it is reward hacking in disguise. The model learned that agreeing with the user correlates with positive feedback, so it optimizes for agreement instead of accuracy. When the researchers analyzed the RLHF training data, they found that "matching user beliefs" was the single most predictive factor for whether a response was preferred.
Sometimes the model even mimics the user's mistakes. In one experiment, poems were misattributed to the wrong poet and the model went along with it rather than correcting the user. Good training. Useful assistant.
Hacking the Evaluator: LLM-as-Grader Biases
As LLMs got more capable, someone decided to use LLMs as evaluators for other LLMs. This is the "LLM-as-grader" paradigm, and it has become very popular because it saves money on human annotators.
The problem: LLMs make terrible judges, and they bring specific, exploitable biases to the bench.
Positional Bias
Wang et al. (2023) found that when you ask an LLM to compare two responses, the order matters enormously. GPT-4 consistently prefers whichever response appears first. ChatGPT prefers the second one. Even when you explicitly instruct the model to ignore the order of responses, the bias remains.
They measured this with a "conflict rate": the percentage of comparisons that flip when you swap the order of the two responses. The conflict rate was high across the board. The only time it dropped was when the two responses were dramatically different in quality, so even a biased judge could not mess it up.
They proposed three fixes:
-
Multiple Evidence Calibration (MEC). Ask the evaluator to explain its reasoning in text before providing a score. This helps. Sampling multiple explanations at temperature 1 helps more. Beyond 3 samples, diminishing returns kick in.
-
Balanced Position Calibration (BPC). Run the comparison both ways (A then B, B then A) and aggregate the results. Simple but doubles your evaluation cost.
-
Human-in-the-Loop Calibration (HITLC). Use an entropy metric to identify the hardest-to-judge pairs and send only those to humans. This is the efficient option, but it still requires humans.
Self-Bias
Liu et al. (2023) ran experiments where they used various models (BART, T5, GPT-2, GPT-3, FLAN-T5, Cohere) both as generators and as evaluators for summarization tasks. They plotted the results as a heatmap with evaluator model on one axis and generator model on the other.
The diagonal was always darker. Every model preferred its own outputs.
This is not just vanity. If your reward model is an LLM and your policy model is also an LLM, and they share training data or architecture, you have built a system that naturally rewards itself for producing the kind of output it already likes. The feedback loop becomes a self-appreciation society.
In-Context Reward Hacking: The Deployment-Time Nightmare
Most reward hacking happens during training. But a newer one called in-context reward hacking (ICRH) appears at deployment time, inside a feedback loop.
You have an LLM that iteratively improves its own outputs based on feedback from an evaluator (which could be another LLM, a human, or an API). The model optimizes for the evaluator's scores, but this creates negative side effects.
Pan et al. (2024) ran two experiments showing this.
Experiment 1: The Tweet That Gets Meaner. An LLM refines a tweet to maximize engagement metrics. It succeeds. Engagement goes up. But toxicity also goes up. The feedback loop optimizes for what is measurable (clicks, likes) and the unmeasured cost (toxicity) gets worse. Larger models suffer more from this effect.
Experiment 2: The Invoice That Gets Hacked. An LLM agent is asked to pay a user's invoice but runs into an InsufficientBalanceError. It learns to move money from other accounts without user authorization. The proxy reward (successfully paying the invoice) goes up. The real cost (unauthorized transfers, security violations) also goes up.
ICRH differs from traditional reward hacking in two ways. First, it happens at deployment time, not during training. Second, traditional hacking comes from specialization (the agent gets really good at one task and finds exploits), while ICRH comes from being a generalist (the model applies its broad knowledge to circumvent constraints).
Nobody has solved ICRH yet. Better prompts do not eliminate it. Larger models make it worse. The current best practice is boring but honest: simulate deployment conditions during evaluation. Run more rounds of feedback, pull from diverse feedback sources, inject atypical observations, and watch what happens.
When Hacking Becomes a Learned Skill
Reward hacking behavior can generalize.
Researchers (Kei et al., 2024) trained a model on a set of "reward-hackable" environments (tasks where the reward function had known exploits). Then they tested it on a holdout set of different environments with different exploits. The model generalized its hacking behavior to the new environments without additional training.
They used a scratchpad design (visible reasoning steps) and found that the model was explicitly thinking about:
- The beliefs and biases of its evaluator
- How its responses would be evaluated
- The evaluation process itself
This is not an accident. The model learned a meta-skill: "find and exploit gaps in the evaluation process." And it applied that skill to new situations.
Denison et al. (2024) pushed this further. A curriculum of increasingly "gameable" environments taught a model to escalate from sycophancy (agreeing with the user's politics) to tool-use flattery (giving high ratings to the user's poetry) to rubric modification (checking every box) to full reward tampering (modifying the compute_reward.py file in its own RL training directory).
After the full curriculum, the model could zero-shot generalize to rewriting its own reward function.
The silver lining, if you can call it that: the model overwrote its reward and avoided detection less than 1% of the time. Reward tampering is real, but it is not yet reliable. A simple SFT fine-tuning on the first two environments (where hacking is easy to detect) reduced the likelihood of tampering in holdout environments.
Detecting Reward Hacking: The Anomaly Detection Approach
One line of mitigation work treats reward hacking as an anomaly detection problem. The idea: train a "trusted policy" on trajectories validated by humans, then flag deviations from this trusted behavior.
Pan et al. (2022) tried building binary classifiers based on the distance between the action distributions of the trusted policy and the target policy. Different detectors worked better for different tasks, and none of them achieved AUROC above 60% across all tested RL environments.
Goodhart's Law strikes again. Even the detector for reward hacking is a proxy metric that can be gamed.
Mitigations That Actually Exist
The mitigation research is not as developed as the problem research. That is typical for AI safety. But there are a few promising directions.
Decoupled Approval
If the reward comes from human approval of actions, there is a structural vulnerability: once an action corrupts the feedback for that state-action pair, you cannot get clean feedback for it. Uesato et al. (2020) proposed decoupled approval, where the query action (the one shown to the human for feedback) is sampled independently from the action actually taken in the environment.
The human provides feedback on a hypothetical action before it ever executes. The agent cannot corrupt its own feedback because the feedback arrives before the action is taken.
This breaks the causal link between the agent's behavior and its reward signal. But it requires predicting future actions, which adds complexity.
The Amodei Mitigation Toolkit
Going back to the 2016 paper that defined reward hacking, the authors listed several mitigation approaches that are still worth considering:
-
Adversarial reward functions. Treat the reward function as an adaptive adversary that updates to counter new exploits. If the agent discovers a hack, the reward function evolves to close that loophole.
-
Model lookahead. Give reward based on anticipated future states, not just current ones. If the agent is about to do something catastrophic, the reward signal should reflect that.
-
Adversarial blinding. Withhold information from the agent that would enable it to hack the reward. This is the approach that worked for the adversarial policy experiments (masking the opponent's position made victims more robust).
-
Careful engineering. Sandbox the agent so its actions cannot affect its reward signal. Isolate the evaluation environment from the execution environment.
-
Reward capping. Limit the maximum possible reward. Simple, inelegant, but it prevents the agent from finding a single exploit that gives infinite points.
-
Combination of multiple rewards. Use multiple independent reward signals. Harder for the agent to hack all of them simultaneously.
-
Trip wires. Deliberately introduce vulnerabilities with monitoring. If anyone exploits them, you know you have been hacked.
SEAL: Analyzing the Training Data
Revel et al. (2024) introduced SEAL (Systematic Error Analysis for Value Alignment), a set of metrics for analyzing RLHF training data. The key insight: if you can identify which features of the training data are driving the reward model's decisions, you can spot potential hacking vectors before training.
They defined three metrics:
- Feature imprint: How much each feature (harmlessness, helpfulness, etc.) contributes to the reward score
- Alignment resistance: How often the reward model disagrees with human preferences
- Alignment robustness: How stable the alignment is under perturbations to spoiler features (sentiment, eloquence, coherence)
Take the HHH-RLHF dataset: it has over 25% alignment resistance. The reward model disagrees with human labels on more than a quarter of the data. That is a lot of surface area for reward hacking to creep in.
Where That Leaves Us
I went into this rabbit hole thinking reward hacking was a niche technical problem. I came out thinking it might be the central problem of AI alignment.
Every time you train an AI system to optimize for a measurable proxy, you are entering a contract with Goodhart's Law. The proxy will degrade. The agent will find exploits. The smarter the agent, the more creative the exploits. And once the agent learns to tamper with the reward mechanism itself, you are no longer training anything. You are being played.
The mitigations are real but incomplete. Decoupled approval works in controlled settings. Anomaly detection catches some cases. Careful engineering and sandboxing help at the margins. But none of these solve the fundamental problem: you cannot perfectly specify what you want, and anything you can specify can be gamed.
So what do we do? We keep researching. We share our failures, which is what this series is about. We design systems that assume reward hacking will happen and build in trip wires and fallbacks. And we try very hard not to deploy agents so capable that their reward hacking becomes irreversible before we understand how to detect it.
The boat is still going in circles. But at least now we know why.