A trained network.

The digits

MNIST is a collection of 70,000 handwritten digits, 0 through 9, each a 28×28 grayscale image with a label saying which digit it is. Every digit was scaled to fit a 20×20 box and then placed in the 28×28 frame so that its ink is centered. It has been a standard first task for CNNs since the 1990s.

The 70,000 are split in two. 60,000 are for training. The other 10,000 are the test set: digits held back from training, used only to measure how well the trained network does. A network can memorize the images it trained on, so a score on those says little. A score on images it never saw says how well it learned digits in general.

The network

The network has five layers. Every one of them comes from an earlier chapter:

1. A 5×5 convolution with 8 kernels, no padding, then ReLU (chapters 2, 4 and 7). The 28×28 image becomes 8 maps of 24×24, since 28 − 5 + 1 = 24.
2. A 2×2 max pooling at stride 2 (chapter 5): 8 maps of 12×12.
3. A 3×3 convolution with 16 kernels, each 3×3×8 because it reads all 8 maps (chapter 4), then ReLU: 16 maps of 10×10, since 12 − 3 + 1 = 10.
4. Another 2×2 max pooling: 16 maps of 5×5.
5. A dense layer (chapter 1) that reads all 16 × 5 × 5 = 400 values and gives 10 outputs, one score per digit. The highest score is the network’s guess.

Counting parameters as in chapter 4, each layer is (inputs × outputs) + biases:

• Convolution 1: (5 × 5 × 1) × 8 + 8 = 208.
• Convolution 2: (3 × 3 × 8) × 16 + 16 = 1,168.
• Dense: (400 × 10) + 10 = 4,010.

That’s 5,386 parameters, and three quarters of them are in the dense layer. Pooling has none.

The training

It was trained exactly the way chapter 7 described. Each digit’s target is 1 for its own score and 0 for the other nine. The loss is the mean squared error between the ten scores and those targets. Each step computes the gradients on a random batch of 64 training digits and moves every weight against its gradient, with step size 1. The network went through all 60,000 training digits 15 times, about 14,000 steps.

On the 10,000 test digits it gets 9,784 right, 97.84%. It gets 216 wrong.

Draw a digit in figure 9.1, or load one from the test set. Your drawing is prepared the way MNIST’s digits were: shrunk so it fits a 20×20 box and centered in 28×28. The network has only ever seen digits prepared that way.

Follow the digit down the layers. The first layer’s 8 maps still look like the digit, each picking out some of its strokes more brightly than others. One of them, the second, is the other way around: lit everywhere except on the ink. After pooling they’re half the size. The second layer’s 16 maps look less and less like the digit: each lights up in only a few places. By the last pooling, each map is a 5×5 grid of numbers, and the dense layer turns those 400 numbers into ten scores.

The scores show how training aimed them: the right digit’s score near 1, the others near 0. Some digits come out with a clear winner, others with a closer race. The last module comes back to what a close race means.

The rest of the chapter asks what the two convolution layers are responding to.