One output.

A kernel is a small grid of weights that a convolution places over the image, one patch at a time. This book uses 3×3 kernels, the most common size, so a kernel holds nine weights: numbers the layer learns, exactly like the weights of a dense layer in chapter 1.

Each square of the kernel is called a tap. When the kernel sits on the image, each tap lies on top of one pixel, so a 3×3 kernel reads nine pixels at a time: a 3×3 patch of neighbors.

One output

To compute one output, place the kernel on the image, multiply each pixel by the weight on top of it, and add up the nine products. That’s a dot product, the same multiply-then-add a dense layer uses, but over nine neighboring pixels instead of the whole image.

Here’s a worked example. The values are small whole numbers to keep the arithmetic easy; the method is the same for pixels between 0 and 1. The patch on the left is what the kernel covers, and the kernel is on the right:

$$\text{patch} = \begin{bmatrix} 1 & 4 & 8 \\ 2 & 5 & 9 \\ 1 & 4 & 8 \end{bmatrix} \qquad \text{kernel} = \begin{bmatrix} -1 & 0 & 1 \\ -2 & 0 & 2 \\ -1 & 0 & 1 \end{bmatrix}$$

Pair each pixel with the weight in the same spot, row by row, and multiply:

$$(1 \times (-1)) + (4 \times 0) + (8 \times 1) + (2 \times (-2)) + (5 \times 0) + (9 \times 2) + (1 \times (-1)) + (4 \times 0) + (8 \times 1)$$ $$= -1 + 0 + 8 - 4 + 0 + 18 - 1 + 0 + 8 = 28$$

So this output is 28. The kernel here has a name, and module 3 shows what it’s for. For now, notice only that the middle column of weights is all zeros, so the middle column of pixels, 4, 5 and 4, doesn’t affect the output at all.

Try it on the duck

In figure 2.1 the kernel sits on the duck. Drag it anywhere, or use the arrow keys. The three grids show the pixels under the kernel, the weights, and their products, and the line below writes out all nine (pixel × weight) pairs and their sum.

The figure starts with the blur kernel, a kernel whose nine weights are all 1/9. Multiplying nine pixels by 1/9 and adding is the same as adding them and dividing by 9, so the blur kernel’s output is the average of the nine pixels it covers. Put it over plain sky, where every pixel is 0.56, and the output is 0.56. Put it across the edge of the duck’s head, and the output lands between the sky and the duck.

Switch to sobel_x, the kernel from the worked example, and to sharpen. Same pixels, different weights, different output. The weights decide what the kernel computes.

The bias

A convolution layer also has a bias, one number added to every output, exactly like the bias of a dense-layer output in chapter 1. With a bias of −3, the worked example’s output would be 28 − 3 = 25. A single-channel 3×3 convolution therefore has 9 weights and 1 bias, the 10 parameters counted in chapter 1.

The bias shifts every output by the same amount, so it never changes the pattern of the outputs, only their level. To keep the figures readable, every figure in this book uses a bias of 0, and the output is just the dot product.