Be the algorithm: train a puppy with paper and a friend

You can learn how reinforcement learning works without a computer. In this activity, people become the algorithm. One person is the puppy, one is the trainer, and one keeps the notes. It takes about 30–40 minutes and works for a small group, a classroom, or self-study with a friend.

The big ideas

• Learning by trial and error, from rewards and penalties.
• Curiosity: trying random things at first, then using what you know.
• Notes that improve with experience.
• What happens when you change the rewards.

What you need

• A grid of squares on the floor (masking tape) or on paper: 4 columns × 3 rows is a good size
• A paper "treat" for one corner, and a "home" square in the opposite corner
• Optional: a "mud" square, and later a "toy" square
• A six-sided die
• Paper, pencil and an eraser, and a printed or drawn notes table

Roles

• The puppy. Moves a counter (or walks the grid) one square at a time.
• The trainer. Knows the rewards, and calls them out after each move.
• The scribe. Keeps the notes table: for every square, four numbers (up, right, down, left), all starting at 0.

The rewards

The trainer calls these out after each move:

• Reaching the treat: +10
• Stepping onto mud: −5
• Every move: −1 (wasting time costs a little)
• Walking into the edge of the grid: the puppy stays put, and still gets −1

Playing a round

1. The puppy starts at home.

2. Choose a move with the die. Roll once:

   • 1–4: curious. Move in a random direction: 1 = up, 2 = right, 3 = down, 4 = left.
   • 5–6: confident. Move in the direction with the highest note for this square. If there is a tie, pick any.

3. The puppy moves. The trainer calls out the reward.

4. The scribe updates the note for the move that was just made:

   new note = old note + ½ × (reward + best note on the new square − old note)

   Reaching the treat ends the round, so on that last move use only the reward.

5. Repeat until the puppy reaches the treat, or 20 moves have passed.

6. Count the moves in the round, and write it down.

A worked example

The puppy is at home, and all notes are 0. It rolls a 2 and moves right. The trainer says −1. The best note on the new square is 0.

new note = 0 + ½ × (−1 + 0 − 0) = −0.5

The scribe writes −0.5 next to "right" on the home square. Next time the puppy is at home and feels confident, "right" looks a bit worse than the other directions, which are still 0, so it will try another way.

After the first rounds

Play 10 or more rounds. Keep the notes table. As you play:

• The first rounds are slow and messy. The puppy wanders.
• A big positive number shows up next to the treat first.
• Squares further from the treat slowly get better numbers.
• Rounds get shorter. Plot the number of moves per round on a chart.

If you want to be more confident later, roll fewer 1–4s. A real learner's curiosity fades over time, and you can copy that: after 5 rounds, count only 1–2 as "curious".

Talk about it

• What did the puppy know after round 1? After round 10?
• Why did the numbers near the treat change first?
• Was the puppy ever stuck doing something silly? Why?
• How is the die like the computer's curiosity?

Change the rewards

Now be the designer. Change one rule and play 10 more rounds with a fresh notes table:

1. Mud. Put a mud square on the shortest route. First play with mud at 0, then at −5. Which route does the puppy learn each time?
2. A bigger treat. Make the treat +20. Does the puppy learn faster? Why might it?
3. A sneaky toy. Add a toy square worth +3 every time the puppy steps on it. What does the puppy learn to do? How could you fix the rules? (This is reward hacking, as in the game.)

Questions that match the game

Differences from the computer version

• The game uses a patience factor of 0.95, which makes distant rewards count slightly less. The paper version leaves it out to keep the sums easy.
• The computer practises hundreds of rounds in seconds, so its notes settle much more cleanly.
• The idea is exactly the same.

Common misconceptions

• "The computer knows the answer and is just showing it." It does not. Its notes start at zero and only improve through experience.
• "The puppy understands mud is bad." It only follows the points. Change the points and it changes its behaviour.
• "Learning means remembering every try." The puppy only keeps the notes. It forgets the individual tries.