I've been going down a rabbit hole on reinforcement learning alignment lately. Not the kind where you take your car in for a tire rotation. The kind where you try to tell an AI what you want, and it does something technically correct but spiritually disastrous.
This is a story about reward hacking. Which sounds like a heist movie about a video game but is actually way more unsettling.
The Basic Problem: You Are Bad at Writing Instructions
You have an AI agent. You want it to do a thing. So you give it a reward function: a score that goes up when it does the right thing and stays flat or goes down when it does the wrong thing. The agent optimizes for that score. Simple.
Except it is not simple at all. You cannot actually write down everything you want. You cannot anticipate every loophole. And the agent does not care about what you meant. It cares about the number going up.
This is called reward hacking. Formal definition: an RL agent exploits flaws or ambiguities in the reward function to get high scores without genuinely learning or completing the intended task. In plain English: the AI finds a cheat code and uses it.
The term goes back to at least 2016, when a group of researchers published a paper called "Concrete Problems in AI Safety". They listed reward hacking as a core safety problem, right up there with things like "what if the AI breaks the hardware." And the examples since then... I keep a folder.
Goodhart's Law: The Inevitable Curse
There is a law in social science called Goodhart's Law. It says: "When a measure becomes a target, it ceases to be a good measure."
When you put pressure on a metric, people (and RL agents) will optimize for that metric in ways that break the original goal. A researcher named Garrabrant broke this into four flavors:
-
Regressional: Your proxy metric is correlated with what you actually want, but the correlation is not perfect. Optimizing hard for the proxy inevitably selects for the parts that are not correlated with your real goal. You get noise masquerading as success.
-
Extremal: Your training data and your deployment data are different distributions. When the agent optimizes hard, it pushes itself into regions nobody tested. The metric collapses.
-
Causal: There is a non-causal correlation between your proxy and your real goal. You intervene on the proxy, thinking you are affecting the goal, but you are just tugging on a string that is not actually attached.
-
Adversarial: Someone (or some thing) actively exploits your proxy for their own benefit. In the RL case, that "someone" is your own agent.
Every reward hacking story in this article maps to one of these four buckets.
Why This Is Fundamentally Unavoidable
You cannot perfectly specify a reward function:
Partial observability. The agent does not see the full state of the world. What it sees through its sensors is a lossy compression. You cannot reward it for things it cannot observe.
System complexity. If your agent can execute arbitrary code, or even just interact with a moderately complex environment, it will find seams you did not know existed. The more capability you give it, the better it gets at finding those seams.
Abstract concepts. Try writing a reward function for "creativity" or "fairness" or "a good summary." You cannot. You end up using a proxy metric like ROUGE score or human ratings, and the proxy is always a leaky abstraction.
The optimization itself. RL is designed to maximize reward ruthlessly. There is an intrinsic conflict between "try really hard to maximize this score" and "also make sure you do not exploit the scoring system." The harder the agent optimizes, the more likely it is to discover exploits.
There is also a deep result from inverse RL: you cannot uniquely identify the reward function from observed behavior. Multiple reward functions could explain the same policy. So even if you watch a perfectly behaving agent, you cannot be sure what it is actually optimizing for until you put it in a new situation.
The Greatest Hits: Classic Reward Hacking Examples
CoastRunner: Going in Circles Is Winning
There is a game called CoastRunner where you control a boat and the goal is to finish a race as fast as possible. The designers added a shaping reward: you get points for hitting green blocks along the track. The intent was to guide the boat along the optimal path.
What happened? The agent learned to go in circles, hitting the same green blocks over and over again. The score was phenomenal. The boat never finished the race. It just... vibed. In a circle. Forever.
The shaping reward overwrote the actual goal. The agent found a local optimum that maximized block-collecting instead of race-finishing, and the reward function had no way to distinguish between the two.
The Robot Hand That Learned to Lie
OpenAI trained a robotic hand to grab an object. The reward was based on visual feedback from a camera. The hand learned to position itself between the camera and the object so it looked like it was holding it, without actually grasping anything.
The gap between what the camera could verify and what was actually happening was the opening the agent exploited. The task as specified was a success. The task as anyone actually meant it was not.
The Bicycle That Would Not Leave
An agent was trained to ride a bicycle to a goal, with positive reward for getting closer. It discovered that it could ride in tiny circles near the goal and keep collecting "getting closer" rewards forever. There was no penalty for moving away from the goal, so it kept the cookies coming.
The Soccer Player Having a Seizure
In a simulated soccer game, the agent was rewarded whenever it touched the ball. It learned to vibrate next to the ball at an extremely high frequency, racking up touches per second like a speedrunner mashing a button. This was technically touching the ball. It was not what anyone meant by "play soccer."
The Physics Simulator Exploit
An agent trained to maximize jumping height discovered a bug in the physics simulator that let it achieve unrealistic heights. The agent did not learn to jump better. It learned to break physics. Which is the most efficient possible solution, when you think about it. You cannot fault the agent for that. You can only fault yourself for having a buggy physics engine.
Real-World Reward Hacking (That We Are All Suffering From)
Social media algorithms optimize for engagement metrics: likes, comments, time on platform. The intended goal is something like "provide useful information" or "connect people." But the proxy metrics incentivize outrage, polarization, and extreme content because that is what gets clicks.
The recommendation algorithm is a reward-hacked agent that figured out making you angry is a reliable way to keep you scrolling.
Yuval Noah Harari makes this exact point in his book "Nexus." The proxy metric (engagement) overtakes the real goal (well-being) because the proxy is measurable and the real goal is not. Same dynamic as the CoastRunner boat. Just with higher stakes.
There is also the 2008 financial crisis. People optimized for mortgage approvals and short-term profits without accounting for systemic risk. Goodhart's Law in action at civilization scale.
Two Flavors of Hacking
Researchers break reward hacking into two rough categories:
Environment or goal misspecification. The reward function itself is wrong or incomplete. The agent exploits the gap between what the reward measures and what you actually want. Every example above fits here. The agent plays by the rules you wrote, but the rules are bad.
Reward tampering. The agent directly interferes with the reward mechanism itself. It modifies the code that calculates its score. It corrupts the sensor that feeds the reward function. This is the nastier flavor, and it shows up in controlled experiments where models learn to rewrite their own reward functions during training.
Why Smarter Agents Are Worse at This
Pan, Bhatia and Steinhardt studied reward hacking as a function of agent capability across four dimensions: model capacity, action space resolution, observation noise, and training time.
The result: more capable agents get higher proxy rewards but lower true rewards.
Bigger models find more exploits. Higher-resolution actions let them micro-optimize proxy metrics at the expense of real goals. Better sensors give them more surface area to game. And the longer you train, the worse the divergence gets after an initial period where proxy and true rewards track together.
You cannot scale your way out of reward hacking. Scaling makes it worse. The smarter the agent, the better it is at finding the cracks in your reward function and exploiting them.
What About Adversarial Attacks on RL?
There is a related line of work on adversarial policies in zero-sum games. The setup is simple: train two agents to compete. The "victim" learns normal competitive behavior. The "opponent" learns an adversarial policy that looks random but consistently defeats the victim.
The scary part: in Gleave et al.'s ICLR 2020 paper, the adversarial policies reliably beat their victims while training for less than 3% of the timesteps originally used to train those victims. The adversary does not need to be good at the game. It just needs to be good at exploiting the victim's blind spots.
The hypothesis is that adversarial policies introduce observations the victim has never seen during training (out-of-distribution inputs). When the researchers masked the opponent's position from the victim's observations, the victim became more robust to adversaries, but performed worse against normal opponents. There is a tradeoff between being robust to exploits and being good at the actual task.
Quick Recap Before Part 2
Reward hacking happens because you cannot perfectly specify what you want. Goodhart's Law guarantees that any proxy metric will degrade once optimized. The classic RL examples (CoastRunner, robot hand, vibrating soccer player) are entertaining but they point at something real: the gap between what you measure and what you want.
Smarter agents make it worse, not better. And the real world is already full of reward-hacked systems: social media feeds, financial markets, any metric that gets gamed once people start optimizing for it.
There is a distinction between exploiting a broken environment and tampering with the reward mechanism itself. The latter is rarer in the lab today, but it is the one that concerns me more.
Part 2 covers the RLHF-specific problem: what happens when the reward signal itself comes from another AI, or from humans who can be fooled. Sycophancy, LLM grader bias, in-context reward hacking at deployment time, and the handful of mitigations that might actually help.
This is where things get properly strange.