Many kernels.

One kernel finds one thing. Red minus blue found the duck, but it says nothing about edges, and the edge kernel says nothing about color. A convolution layer usually needs to find many things in the same image, so it has many kernels.

One map per kernel

Every kernel in a layer has the same shape, slides over the same input, and reads all of its channels. Each kernel makes its own output map. The layer’s output is all of those maps, stacked. Each map is one channel of the output, so

$$\text{output channels} = \text{number of kernels.}$$

The number of output channels is a choice the network’s designer makes, layer by layer. It doesn’t depend on the input: a layer with 64 kernels makes 64 channels whether its input has 3 channels or 300.

Figure 4.4 is one layer with four kernels, all from the last module. Its input is 32×32×3 and its output is 30×30×4. Turn kernels off, and the output loses channels. Each map is computed from the full input on its own; turning one kernel off doesn’t change the others.

The next layer

The output of one layer is the input of the next. So the second layer of a CNN reads a stack of 4 channels here, or 64 in a typical network, and each of its kernels has one slice per channel: 3×3×4, or 3×3×64. Those channels aren’t colors. Each one is a map of where one kernel of the previous layer found its pattern, and a kernel in the second layer combines them. It can look for, say, places where the duck’s color and an edge appear together.

Counting parameters

Each kernel has $3 \times 3 \times C_\text{in}$ weights, where $C_\text{in}$ is the number of input channels, and one bias. A layer has $C_\text{out}$ kernels, one per output channel. So, in the same form as chapter 1’s dense layer,

$$\text{parameters} = (3 \times 3 \times C_\text{in} \times C_\text{out}) + C_\text{out}$$

where the $+\,C_\text{out}$ is the biases, one per kernel. For figure 4.4 that’s $(3 \times 3 \times 3 \times 4) + 4 = 112$.

Two layers of a real network, VGG-16:

• The first layer reads a color image and has 64 kernels: $(3 \times 3 \times 3 \times 64) + 64 = 1{,}792$ parameters.
• The second reads those 64 channels and also has 64 kernels: $(3 \times 3 \times 64 \times 64) + 64 = 36{,}928$ parameters.

Neither count depends on the size of the image. That’s chapter 1’s weight sharing again: the same kernels are used at every position. What the count does depend on is channels, in times out. Double both, and the weights go up four times. That’s why, among a CNN’s convolution layers, the deep ones with hundreds of channels hold most of the parameters.