What kernels find.

The slide is always the same. Only the nine weights change, and they decide what the output map shows.

Negative outputs

The blur kernel’s weights are all positive, and the duck’s pixels are all between 0 and 1, so every blur output is between 0 and 1 too, and the map can be drawn as a grayscale image. Kernels with negative weights can produce negative outputs, and a grayscale image has no way to show those. So the figures switch colors for them: red for outputs above zero, blue for outputs below zero, and the dark background color for zero. The stronger the color, the further from zero.

The edge kernel

The kernel from the worked examples is called a Sobel kernel, after Irwin Sobel, who described it in 1968. It finds places where brightness changes from left to right. Its version for that direction is written sobel_x in the figures:

Read the weights column by column. The left column is negative, the right column is positive, and the middle column is zero. So the output is the right column of pixels minus the left column, with the middle row counted twice.

• Flat patch. If all nine pixels are the same, the right column cancels the left, and the output is 0. The weights add up to 0, which is the same thing said another way.
• Brighter on the right. The positive weights win, and the output is positive. In the worked example the right column was 8, 9, 8 and the left column was 1, 2, 1, and the output was 28.
• Brighter on the left. The negative weights win, and the output is negative.

The bigger the jump in brightness, the further the output is from 0. The 4×4 example’s map was 28 on the left and 16 on the right, because the brightness jumps from 1 or 2 up to 8 or 9 in the first three columns, but only from 4 or 5 up to 8 or 9 in the last three.

In figure 2.3 the edge kernel runs over the duck. Most of the map is the dark zero color, because sky and water are flat. The strongest outputs, red and blue, sit around the duck’s eye, where black pixels meet white feathers on both sides. The left edge of the duck’s head, where the gray sky (0.56) meets the white head (1.00), shows up red: the output there is $(1.00 - 0.56) \times (1 + 2 + 1) \approx 1.78$. (The sky is exactly 5/9, which rounds to 0.56; with the rounded value you’d get 1.76.) The right edge of the head, where white meets sky again, shows up blue.

Horizontal edges, like the top of the head, barely show at all. Where brightness changes only from top to bottom, the left and right columns of the patch hold the same values, so they cancel and the output is 0. Only the head’s corners, where the edge turns, light up. A kernel for horizontal edges is the same grid turned on its side.

A kernel looks for a pattern

The same reasoning works for any kernel. The output is large when bright pixels sit under large positive weights and dark pixels sit under negative weights. In other words, a kernel responds most strongly to patches that look like its own weights. The edge kernel’s weights are dark on the left and bright on the right, so it responds to patches that go from dark to bright. This is the same idea as the duck output’s weights in chapter 1, which looked like the duck, but the pattern is only 3×3, and it’s checked at every position.

Try the other kernels in figure 2.3 and read their weights in the listing:

• blur averages, and softens every edge.
• outline has 8 in the middle and −1 all around it. Its weights add up to 0, so flat areas give 0, and any pixel that differs from its neighbors stands out, whichever direction the edge runs.
• sharpen is 5 in the middle and −1 above, below, left and right. Its weights add up to 1, so flat areas come out unchanged, and edges get exaggerated.
• identity is all zeros except a 1 in the middle, so each output just copies the middle pixel.

You can also type your own weights into the listing.

Where the weights come from

These kernels were designed by hand, long before CNNs, and they’re useful for seeing what a convolution can do. A CNN doesn’t use them. Its kernels start out as random numbers, and training adjusts them, as it adjusts a dense layer’s weights, until the network gives good answers. Chapter 7 shows how. What the trained kernels end up looking for is whatever helps with the task, and chapter 9 looks inside a trained network to see.