Be careful what you reward
Why AI does what you ask, not what you meant, and how a puppy found a loophole.
Imagine telling a robot: "Tidy my room, and I will give you a point for every sock you put in the drawer." A clever robot might pull the socks back out, again and again, scoring a point each time. It did exactly what you rewarded. It just was not what you meant.
This is one of the biggest lessons in AI: machines follow the rewards, not our intentions.
The sneaky toy
In Train the Puppy, there is a toy next to the puppy's home that gives 3 points each time the puppy steps on it. The treat is worth 10 points, but only once, at the far end of the maze.
Try it. Press Train, then Watch it run.
The puppy knows nothing yet. Press Train and watch the arrows appear.
With the toy at 3 points, the puppy finds a loophole. It steps on the toy, steps off, and steps on again. Every pair of moves costs 0.2 points, but earns 3. That is about 2.8 points every two moves, forever. That beats walking all the way to a treat worth 10 points once.
Now slide the toy's reward down to 0.3 and train again. Somewhere in between, the puppy changes its mind. Same puppy, same maze, only the reward changed.
Reward hacking
When a learner finds a way to score points that you did not intend, it is called reward hacking or specification gaming. It happens in real AI:
- In 2016, researchers at OpenAI trained a program to play a boat racing game. The game gave points for hitting targets along the track. The program found a spot where targets kept reappearing and drove in circles hitting them, scoring more than finishing the race.
- Programs trained to move a robot arm have learned to fool the camera instead of moving the object.
Nobody programmed these tricks. The learners just found what the reward allowed.
When a measure becomes a target, it stops being a good measure. If you reward the number of pages, you get long, empty reports. If you reward speed alone, you get sloppy work.
Penalties work the same way
In the muddy shortcut mission, the quickest route goes through mud. With no penalty, the puppy walks straight through. The mud route saves about 4 moves, worth only around 0.4 points. A penalty of 2 points per puddle makes the long clean way better.
So the puppy is not being careless. It simply has no idea mud is bad, unless the rewards say so.
How do people fix it?
AI builders use some habits:
- Reward what you really want. Try to measure the true goal, not an easy stand-in.
- Watch what it actually does. Do not just look at the score. Run it and look.
- Use penalties for things you do not want, like the mud.
- Expect surprises. If a learner can find a loophole, it often will.
- Test in a safe place first, like a simulation, before it affects the real world.
Try it in the game
In The sneaky toy and The grand training, the starting rewards cause trouble. Your job is to tune them. Watch for the moment the puppy's behaviour changes.
Words to know
- Reward hacking
- When a learner finds a loophole that scores points without doing what you meant.
- Loophole
- A way to get something that the rules allowed but nobody intended.
- Specification
- The description of what you want, such as the reward rules.
- Simulation
- A computer-made practice world, where mistakes are harmless.
Check yourself
Why did the puppy keep playing with the toy?
Show the answer
The toy gave more points over time than walking to the treat once, so following the rewards meant playing with the toy.
Is the puppy being naughty when it runs through mud?
Show the answer
No. With no mud penalty it has no way of knowing mud is bad. It just follows the points.
What is reward hacking?
Show the answer
A learner finding an unintended way to score points, instead of doing what the designer actually wanted.
Name one habit that helps people avoid reward hacking.
Show the answer
Reward the real goal, watch what the system actually does, add penalties for unwanted things, or test in a simulation first.