Gradients for the weights.

Backpropagation reaches a convolution layer with one number for each of its outputs: the gradient of the loss with respect to that output. Call that map of gradients $G$. The layer’s job is to turn $G$ into a gradient for each of its nine weights.

The example

Take chapter 2’s 4×4 example: the image, the edge kernel, and the 2×2 output map it made.

$$x = \begin{bmatrix} 1 & 4 & 8 & 8 \\ 2 & 5 & 9 & 9 \\ 1 & 4 & 8 & 8 \\ 1 & 4 & 8 & 8 \end{bmatrix} \qquad w = \begin{bmatrix} -1 & 0 & 1 \\ -2 & 0 & 2 \\ -1 & 0 & 1 \end{bmatrix} \qquad y = \begin{bmatrix} 28 & 16 \\ 28 & 16 \end{bmatrix}$$

Suppose the targets are 27, 16, 28 and 17. Two outputs are right, one is 1 too high, and one is 1 too low. The squared error is $1 + 0 + 0 + 1 = 2$, and each output’s gradient is $2 \times (y - t)$:

$$G = \begin{bmatrix} 2 \times (28 - 27) & 2 \times (16 - 16) \\ 2 \times (28 - 28) & 2 \times (16 - 17) \end{bmatrix} = \begin{bmatrix} 2 & 0 \\ 0 & -2 \end{bmatrix}$$

In a real network, $G$ would arrive from the layers above. Here it comes straight from the loss.

One weight, every position

Take the kernel’s middle weight. Because of weight sharing, it was used at all four positions, and each time it sat on a different pixel: 5 for the top-left output, 9 for the top-right, 4 for the bottom-left and 8 for the bottom-right.

Nudge that weight up by a small amount $h$. Each output changes by $h$ times the pixel the weight sat on, because that’s what the weight multiplied. And by the chain rule, each output’s change moves the loss by its gradient times that change. So each output contributes (pixel × $G$) to the weight’s gradient, and the four contributions add up:

$$(5 \times 2) + (9 \times 0) + (4 \times 0) + (8 \times (-2)) = 10 + 0 + 0 - 16 = -6$$

The gradient is −6, so increasing the middle weight would lower the loss. That makes sense: the bottom-right output is too low, and raising this weight raises it by 8 for every unit, more than it raises the too-high top-left output, by 5.

All nine

Every weight works the same way: add up (pixel × $G$) over all positions, using the pixel that weight sat on at each one.

A 3×3 kernel fits on the 4×4 image in four positions, so figure 7.2 draws the image four times, once per position, with the kernel outlined on it and each weight written under the pixel it sits on. Pick any weight. In each copy, the pixel that weight sat on is filled in, and underneath is what that position adds to the weight’s gradient. The line at the bottom adds up the four. The full gradient, one number per weight, is

$$\begin{bmatrix} -8 & -10 & -2 \\ -4 & -6 & 2 \\ -6 & -8 & 0 \end{bmatrix}$$

It has the same shape as the kernel, 3×3, because there’s one gradient for each weight.

Why sharing matters

Every weight of a convolution is used at every position, so its gradient collects a contribution from every position. On the 32×32 duck that’s 900 contributions per weight. Each position says how the weight should change to fix that position’s output, and the gradient adds up all those votes. That’s why a learned kernel ends up finding a pattern that’s useful everywhere in the image, not at one place: it was trained by all of them at once.