Gradients for the input.

The weight gradients are what training uses to update this layer. But backpropagation has to keep going, back to the layers below. This layer’s input is the output of the layer before it, so that layer needs a gradient for each of its outputs: the gradient of the loss with respect to each of this layer’s input pixels.

One pixel

Nudge one input pixel up by a small amount $h$. Every output whose window covers that pixel changes by $h$ times the weight that sat on it. So each of those outputs contributes ($G$ × weight) to the pixel’s gradient, and the contributions add up.

Take pixel (1, 1) of the 4×4 example, the 5. All four kernel positions cover it, each through a different weight:

• The top-left output saw it through the kernel’s middle weight, 0.
• The top-right output saw it through the middle-left weight, −2.
• The bottom-left output saw it through the top-middle weight, 0.
• The bottom-right output saw it through the top-left weight, −1.

With $G = \begin{bmatrix} 2 & 0 \\ 0 & -2 \end{bmatrix}$ from the last module:

$$(2 \times 0) + (0 \times (-2)) + (0 \times 0) + ((-2) \times (-1)) = 2$$

A corner pixel is covered by only one position. Pixel (0, 0) was read only by the top-left output, through the top-left weight, so its gradient is $2 \times (-1) = -2$.

Figure 7.3 draws the image once per kernel position, like figure 7.2. Pick a pixel in any copy. Each position whose kernel covers it shows the weight that sat on it and what that adds to the pixel’s gradient; positions that don’t cover it add nothing. In a 4×4 image the four corners are read once, the other edge pixels twice, and the middle four by every output. On a bigger image, a pixel far from the edges is read by nine outputs, one for each tap of the kernel, so its gradient adds nine pairs.

Spreading gradients back

There’s another way to see the same computation. Each output read a 3×3 window of the input. Going backward, it spreads its gradient back over that same window, multiplied by the kernel: it adds $G$ times the kernel’s weights onto the nine pixels it read. Where windows overlap, the spread-out gradients add up.

In the example, the top-left output stamps $2 \times$ the kernel onto the top-left 3×3 window, the bottom-right output stamps $-2 \times$ the kernel onto the bottom-right window, and the other two stamp zeros. The full input gradient is the sum of the stamps:

$$\begin{bmatrix} -2 & 0 & 2 & 0 \\ -4 & 2 & 4 & -2 \\ -2 & 4 & 2 & -4 \\ 0 & 2 & 0 & -2 \end{bmatrix}$$

Figure 7.4 runs both directions on the same example, one kernel position at a time, in the same order. Forward, each position gathers its 3×3 window into one output. Backward, each output’s $G$ is multiplied by the whole kernel, and that 3×3 stamp is added onto the window the output read. Step through the backward direction and watch the input gradient build up: the first stamp, 2 × the kernel, fills the top-left window; the two middle stamps add nothing, because their $G$ is 0; the last stamp, −2 × the kernel, lands on the bottom-right window, and where it overlaps the first, the two add.

So the backward step turns a 2×2 map into a 4×4 map. It runs the convolution in reverse: instead of gathering a window into one number, each number is spread out into a window. Chapter 8 uses exactly this operation, the transposed convolution, as a layer in its own right, to make maps bigger.

Chapter 2 mentioned that a mathematician’s convolution flips the kernel. That flip shows up here. Read the stamps from a pixel’s point of view: the bottom-right output saw pixel (1, 1) through the top-left weight. As you go from output to pixel in one direction, you go through the kernel in the other. Written as a formula, the input gradient slides the kernel, flipped left to right and top to bottom, over $G$ padded with two rings of zeros. Sliding a flipped kernel is exactly what a mathematician’s convolution does.