Loss and gradients.

To improve a network, you first need a way to say how wrong it is.

Loss

A loss is a single number that measures how far a network’s outputs are from the outputs you wanted, the targets. Zero means every output is exactly right; bigger means worse. Training is the search for weights that make the loss small.

This chapter uses the simplest loss, the squared error: subtract the target from the output and square the difference, $(y - t)^2$. Squaring makes every error count as positive, so a miss above the target and a miss below it both add to the loss, and it makes big misses count much more than small ones. For a whole map of outputs, add up the squared error of each.

One weight

Take the smallest network there is: one input $x$, one weight $w$, and one output $y = w \times x$. Let the input be 2 and the target 6. The right weight is obviously 3, but suppose the network starts at $w = 1$:

$$y = 1 \times 2 = 2, \qquad \text{loss} = (2 - 6)^2 = 16$$

Which way should $w$ move? Try nudging it. At $w = 1.01$, the output is 2.02 and the loss is $(2.02 - 6)^2 = 15.8404$. A nudge of 0.01 lowered the loss by 0.1596, about 16 times the nudge. So near $w = 1$, the loss changes by about −16 for every unit $w$ moves.

That rate is the gradient of the loss with respect to $w$: how much the loss changes per unit change in the weight, for a very small change. It’s the slope of the loss, drawn as a curve against the weight. A negative gradient means increasing the weight lowers the loss; a positive gradient means increasing it raises the loss. The bigger the gradient, the steeper the slope.

Computing it without nudging

Nudging works, but a real network has millions of weights, and nudging each one separately would take millions of extra runs. Instead, the gradient can be computed directly, by splitting the path from weight to loss into steps and multiplying their rates:

• How fast the loss changes with the output. For the squared error that’s $2 \times (y - t)$. Here, $2 \times (2 - 6) = -8$.
• How fast the output changes with the weight. Since $y = w \times x$, every unit of $w$ adds $x$ to the output. Here, 2.

Multiply them: $-8 \times 2 = -16$, the same answer the nudge gave. This step-by-step multiplying is called the chain rule: when one quantity drives a second, and the second drives a third, the rate from first to third is the product of the rates along the way. So

$$\text{gradient} = 2 \times (y - t) \times x$$

Gradient descent

Knowing the slope, move the weight downhill: against the gradient. Gradient descent is the method that does this over and over:

$$w \leftarrow w - \text{step size} \times \text{gradient}$$

The step size, also called the learning rate, is a small number you choose that sets how far each step goes. With a step size of 0.1, the first step is $w = 1 - 0.1 \times (-16) = 2.6$. Then $y = 5.2$, and the loss drops from 16 to $(5.2 - 6)^2 = 0.64$.

Press Step a few times in figure 7.1. As $w$ gets close to 3, the slope flattens, the gradient shrinks, and so do the steps, so gradient descent slows down as it arrives instead of running past.

Now set the step size to 0.3 and reset. The first step jumps from $w = 1$ to $w = 5.8$, past the bottom and higher up the other side: the loss goes from 16 to 31.36. The next step overshoots even further, and the loss keeps climbing: 61.47, then 120.47. For this curve, any step size above 0.25 makes every step land further out than the last. Too small a step size is slow; too large and training falls apart. Picking it well is one of the practical arts of training networks.

Backpropagation

A network has many layers, and the loss sits at the very end. The gradient of a weight deep inside has to account for every layer between that weight and the loss. Backpropagation is the method for computing all of a network’s gradients at once: start from the loss and work backward, one layer at a time, using the chain rule at each layer to turn the gradient of the layer’s output into gradients for its weights and its input. Every layer only needs to know how to do its own step. The next two modules work out that step for a convolution.