Learning a kernel.

Now all the pieces are in place. Training a convolution is a loop of four steps:

1. Forward. Slide the kernel over the input and compute the output map.
2. Loss. Compare the output map with the target, and compute each output’s gradient $G$.
3. Backward. Turn $G$ into a gradient for each of the nine weights, as in the Gradients for the weights module.
4. Step. Move every weight against its gradient: weight − step size × gradient.

Then repeat.

Learning the edge kernel

Figure 7.5 runs that loop on the 32×32 duck. The target is the edge kernel’s output from chapter 2, and the kernel being trained starts with small, arbitrary weights. Can gradient descent find the edge kernel, knowing only the duck and the map it should make?

The loss here is the mean squared error: the squared error added over all 900 outputs and then divided by 900, so it’s the average squared miss per output. Dividing keeps the gradients the same size however big the map is, so the same step size works on any image.

Press Train. The loss starts at 0.35. With a step size of 0.2 it falls below 0.001 after 319 steps, and after 806 steps every weight is within 0.05 of the edge kernel’s. Left running, the kernel becomes the edge kernel, to as many decimal places as the figure shows. The output map, which starts out nearly blank, takes on the red and blue edges of the target.

Watch the gradient grid while it trains. Early on, it’s large and the weights move fast. As the output gets closer to the target, every $G$ gets smaller, so every weight gradient does too, and the weights settle instead of overshooting, just like the single weight in figure 7.1.

Try the other step sizes. At 0.1, everything takes twice as many steps: 638 to get the loss under 0.001, and 1,613 to bring every weight within 0.05. At 0.05, it’s twice as slow again. At 0.3, the loss grows instead of shrinking, and after a few dozen steps it’s in the billions. For this image the cutoff is just under 0.25; above it, each step overshoots further than the last.

Three layers

One kernel learning to copy another kernel is the simplest case. Here’s a harder job: mark the duck. Given the gray duck, output 1 on every duck pixel, beak and eye included, and 0 on the sky and water. That’s 289 duck pixels out of 1,024.

A single 3×3 kernel can’t do this, for two reasons. First, the answer has to switch from 0 to 1 at some brightness: the duck’s body is 1.00, the sky 0.56. A kernel only multiplies and adds, so its output rises smoothly with brightness instead of switching. Second, the duck’s eye is black, darker than anything else in the image, yet it’s part of the duck. Getting it right means seeing the white feathers around it, which takes more than a 3×3 view.

So figure 7.6 stacks three convolution layers. Layer 1 reads the duck and makes 4 maps; layer 2 reads those 4 and makes 4 more; layer 3 reads those and makes the 1 output map. Each layer uses same padding from chapter 3, so every map stays 32×32, and as chapter 6 showed, three stacked 3×3 layers give each output a 7×7 receptive field. Counting with chapter 4’s formula, the layers have $(3 \times 3 \times 1 \times 4) + 4 = 40$, $(3 \times 3 \times 4 \times 4) + 4 = 148$ and $(3 \times 3 \times 4 \times 1) + 1 = 37$ parameters: 225 in all, every one of them learned.

ReLU

There’s one more piece between the layers. Stacking convolutions alone doesn’t help: a convolution of a convolution only multiplies and adds, so three of them in a row compute nothing that one bigger kernel couldn’t. To get the switch from 0 to 1 the job needs, something has to bend.

ReLU, short for rectified linear unit, is the simplest thing that bends: it keeps every positive number as it is and replaces every negative number with 0. Applied to a map, it’s applied to each value separately. Layers 1 and 2 put their outputs through a ReLU before the next layer reads them. That small cut is enough: with it, stacked layers can build up sharp yes-or-no answers out of smooth sums. Almost every CNN puts a ReLU, or something like it, after its convolutions.

Backpropagation goes through a ReLU easily. Where the ReLU’s input was positive, it passed the value through unchanged, so it passes the gradient back unchanged. Where the input was negative, the output was stuck at 0, and nudging the input changed nothing, so the gradient there is 0.

Press Train. The network starts with random weights. Its output is below 0.5 everywhere, so it calls every pixel “not duck”, which is right for the 735 pixels of sky and water and wrong for all 289 duck pixels. The loss starts at 0.43. As it trains, the duck appears in the output. After 500 steps it gets 1,004 pixels right and its loss is 0.028; after 3,000 steps, 1,008, or 98%. Each step is the same four parts as before, only longer: forward through three layers, the loss, gradients back through three layers and two ReLUs, and a step for all 225 parameters.

Watch the hidden maps while it trains. No one told them what to show; each one becomes whatever helps layer 3 get the answer right, and it doesn’t have to be anything a person would put a name to. Try the other starting points. The output ends up about as good each time, but the hidden maps come out different. There are many sets of weights that do the job, and where training ends up depends on where it started.

Now try start 2 with a step size of 0.2. The network never gets past 735. A few large early steps pushed every value in layer 2’s maps below zero, so the ReLU turns all four maps to 0. Then the gradient that reaches layer 2 is cut to 0 everywhere, layers 1 and 2 stop learning, and layer 3 is left reading blank maps. This is called a dead ReLU, and it’s a real hazard in training: too large a step doesn’t always blow up, as it did in figure 7.5. Sometimes it switches part of the network off for good.

How real training differs

These figures are small, honest versions of what happens inside a CNN, with three differences.

• The target. Here, the targets are whole maps: one made by a kernel we already knew, one drawn by hand. A real network is given labels, such as “this image is a duck”, and the loss is computed only at the network’s last layer. No one knows what any kernel should be, so there’s nothing to compare a kernel’s output map with directly.
• The depth. Figure 7.6 had three layers and 225 parameters. Real CNNs have dozens of layers, hundreds of channels, and millions of parameters, but the gradient for every kernel arrives the same way, through every layer above it. Each kernel ends up as whatever helps the final answer, which is why chapter 9 has to look inside a trained network to find out what its kernels found.
• The data. A real network trains on many images, not one. Each step usually computes the gradient on a small random batch of them, so the gradient is a little different every step. That version is called stochastic gradient descent, and it’s how nearly all CNNs are trained.

The core is the same: forward, loss, backward, step. Every weight in every kernel of every layer gets its own gradient, and moves a little downhill.