Sliding the kernel.

A single output reads one patch. A convolution computes an output at every position where the kernel fits: it slides the kernel across the image one pixel at a time, and at each stop it computes a dot product, exactly as in the last module. The outputs, laid out in the same arrangement as the positions they came from, form a new grid called the output map.

The kernel visits positions in reading order: left to right along the top, then one row down and left to right again, the same order flattening used in chapter 1. The output at row $r$ and column $c$ of the map comes from the kernel with its top-left tap on pixel $(r, c)$.

By hand

Here is a 4×4 image. Its top-left 3×3 patch is the patch from the last module:

A 3×3 kernel fits on a 4×4 image in four places: its top-left tap can sit on rows 0 or 1 and on columns 0 or 1. Using the same kernel as before:

• Top-left, on pixel (0, 0): the worked example, 28.
• Top-right, on pixel (0, 1): the patch is 4, 8, 8 / 5, 9, 9 / 4, 8, 8. The middle column is ignored, so this is $(4 \times (-1)) + (8 \times 1) + (5 \times (-2)) + (9 \times 2) + (4 \times (-1)) + (8 \times 1) = -4 + 8 - 10 + 18 - 4 + 8 = 16$.
• Bottom-left, on pixel (1, 0): the patch is 2, 5, 9 / 1, 4, 8 / 1, 4, 8, which gives $-2 + 9 - 2 + 16 - 1 + 8 = 28$.
• Bottom-right, on pixel (1, 1): the patch is 5, 9, 9 / 4, 8, 8 / 4, 8, 8, which gives $-5 + 9 - 8 + 16 - 4 + 8 = 16$.

So the output map is

Four outputs, all computed with the same nine weights. That’s the weight sharing from chapter 1: the layer has nine weights however many positions it visits.

How big the output map is

The kernel has to fit entirely inside the image. Along a row of $n$ pixels, a kernel 3 pixels wide can start at pixel 0, 1, 2 and so on, up to pixel $n - 3$; any later and it would hang off the end. That’s $n - 3 + 1 = n - 2$ positions. Columns work the same way, so

The 4×4 image gives a 2×2 map, and the 32×32 duck gives a 30×30 map. Every 3×3 convolution loses one pixel on each side, because the kernel’s middle tap can never sit on the outermost ring of pixels. Chapter 3 shows how to change that.

Figure 2.2 runs the whole slide on the duck with the blur kernel. The highlighted squares on the left are the nine pixels being read, and the map on the right fills in one output at a time. Pause it and press → to step one position at a time; the listing below shows the patch, the weights, and the sum. While paused, point at any output to see which nine pixels it came from.

The output map is a softer copy of the duck, two pixels smaller. Each output is the average of a 3×3 patch, so a sharp edge between sky and duck turns into a ramp across neighboring outputs.

A note on the name

In mathematics, a convolution flips the kernel, left to right and top to bottom, before sliding it. The operation without the flip is called cross-correlation: slide the kernel as it is, and take a dot product at each position. Deep learning libraries, PyTorch’s nn.Conv2d among them, compute cross-correlation but call it convolution. The flip makes no difference to a network, because its weights are learned: a network that needed a flipped kernel would simply learn the flipped weights. This book follows the libraries. Every kernel is used exactly as written, with no flip.