Kernels with depth.

A 3×3 kernel on a grayscale image reads nine pixels. On a color image each of those nine pixels holds three numbers, so the kernel has 27 numbers to read. It gets a weight for each of them.

A kernel for an input with several channels has one 3×3 grid of weights per input channel. Each of those grids is called a slice of the kernel. For a color image the kernel has three slices, red, green and blue, so it’s 3×3×3 with 27 weights. The depth of a kernel always matches the number of channels in its input.

One output

To compute one output, place the kernel on the image as before. Each slice makes a dot product with its own channel’s 3×3 patch: the red slice with the red patch, green with green, blue with blue. That gives three sums. Add them, and that’s the output. Add the bias, if there is one.

That’s the same as one dot product over all 27 (pixel × weight) pairs. Grouping it by channel just makes it easier to read.

Here’s a worked example on a 0 to 9 scale, so the arithmetic stays in whole numbers. The patch is two rows of yellow duck, (9, 8, 2), over one row of blue water, (2, 4, 8):

$$\text{R} = \begin{bmatrix} 9 & 9 & 9 \\ 9 & 9 & 9 \\ 2 & 2 & 2 \end{bmatrix} \quad \text{G} = \begin{bmatrix} 8 & 8 & 8 \\ 8 & 8 & 8 \\ 4 & 4 & 4 \end{bmatrix} \quad \text{B} = \begin{bmatrix} 2 & 2 & 2 \\ 2 & 2 & 2 \\ 8 & 8 & 8 \end{bmatrix}$$

The kernel’s red slice is all 1s, its green slice is all 0s, and its blue slice is all −1s. Each slice’s dot product:

• Red: every weight is 1, so it adds the red patch: $9 + 9 + 9 + 9 + 9 + 9 + 2 + 2 + 2 = 60$.
• Green: every weight is 0, so it’s 0.
• Blue: every weight is −1, so it subtracts the blue patch: $-(2 + 2 + 2 + 2 + 2 + 2 + 8 + 8 + 8) = -36$.

The output is $60 + 0 + (-36) = 24$.

This kernel computes “red minus blue” over the patch. On a patch of pure duck yellow, it gives $9 \times 9 - 9 \times 2 = 63$. On pure water it gives $9 \times 2 - 9 \times 8 = -54$. Positive means more red than blue, which is exactly the combination the last module said picks out the duck.

On the duck

Figure 4.3 runs a red-minus-blue kernel over the color duck. Its weights are 1/9 and −1/9 rather than 1 and −1, so each output is the patch’s average red minus its average blue. Drag the kernel, and each row shows one slice at work: its channel’s patch, its nine weights, and its partial sum.

The output map on the right is red over the whole duck, beak and wing included, blue over the water, and faintly blue over the sky. Inside the duck’s head the output is 1.00 − 0.20 = 0.80. In open water it’s 0.19 − 0.84 = −0.65. By combining two channels, one kernel separates the duck from both the sky and the water by a wider margin than any single channel does.

Try the other kernels:

• blue − red is the same kernel with every sign flipped, so its map is the same with red and blue swapped. It finds water.
• brightness has all 27 weights equal to 1/27. It averages every number under it, so it ignores color and responds to how light the patch is.
• edge, all colors puts chapter 2’s edge kernel in every slice. Because the three slices are the same, it finds left-to-right changes in the sum of the three channels.

Still one map

However many slices a kernel has, the slices’ sums are added into one number. So one kernel on a three-channel input still makes one output map, with one channel. The channels went in, got mixed, and came out as one. The next module shows how a layer ends up with more than one channel in its output.