How a puppy learns: reinforcement learning

Imagine a puppy dropped into a maze. It has no map. Somewhere there is a treat. Nobody tells it the way. How could it ever learn a route?

The answer is the same way real puppies learn: by trying things, and noticing what pays off. Computers can learn like this too. It is called reinforcement learning.

The learning loop

Every move the puppy makes goes round the same loop:

1. Pick a move: up, down, left or right.
2. The world responds: the puppy ends up on a new square (or bumps into a fence and stays put).
3. Reward or penalty: a treat is worth a big reward. Mud costs points. Every move costs a tiny bit, so wasting time is a little bad.
4. Update the notes: the puppy adjusts what it believes about that move.

Then it starts again. After many loops, the puppy's notes point toward the treat.

The puppy's notes

What exactly does the puppy remember? For every square in the maze, it keeps four notes, one for each move. Each note says: "how good do I think it is to go this way from here?"

At the start, every note is 0. The puppy knows nothing. After each move, it improves one note, like this:

new note = old note + a little step toward (the reward I just got + the best note from the square I landed on)

Look at that second part. The puppy does not only count the reward it just got. It also counts how good the next square looks. That way, a good score slowly spreads backward from the treat:

• The square next to the treat learns "going to the treat is great".
• Then the square before it learns "going to the square next to the treat is good".
• And so on, all the way back to the start.

It is like following a trail of breadcrumbs that the puppy lays down in reverse.

Try it

Watch a tiny puppy learn a tiny maze. Press Train a few times, and watch the arrows appear and settle.

The arrow in each square is the move the puppy now thinks is best. Notice how the arrows nearest the treat appear first, and the ones near home appear last.

Curiosity: explore or play it safe?

If the puppy always picked its best-known move from the start, it could get stuck. Its first guess might lead to a long route, and it would never find a shortcut because it never tried anything else.

So early on, the puppy is curious. Often it picks a random move, just to see what happens. As it learns, the curiosity fades, and it mostly follows its best notes. You can see the curiosity percentage in the game.

Scientists call this the balance between exploring (trying new things) and exploiting (using what you know). Too much exploring wastes time, too little and you miss better options. People do this too, when we choose between our favourite restaurant and a new one.

Why does the puppy find the shortest way?

Two small things push it toward short routes:

• Each move costs a little. A long route piles up more costs.
• Later rewards count a little less. A treat ten moves away is worth slightly less than one two moves away.

You never told the puppy "find the shortest route". The rewards did that for you.

How big is this in real life?

The puppy keeps 4 notes for each of about 35 squares: roughly 140 numbers. That is tiny. Real systems use the same idea with far bigger "notes", stored in neural networks:

• Programs that play board games and video games at expert level.
• Robots that learn to walk or grab things, often practising in a simulation first.
• Chat assistants, which are fine-tuned using human ratings as rewards.