1×1 convolutions.

Nothing says a kernel has to be 3×3. Make it as small as it can be, one pixel wide, and something interesting is left.

A 1×1 convolution is a convolution whose kernel covers a single pixel. On an input with $C_\text{in}$ channels, each kernel is 1×1×$C_\text{in}$: one weight per channel, and nothing else. At every pixel, it multiplies that pixel’s channel values by the weights and adds them up.

On the duck

Figure 4.5 is a 1×1 convolution with one kernel on the color duck. The kernel is just three weights, one each for R, G and B. Point at any pixel to see its computation.

It starts with each weight at 1/3, which averages the three channels and makes a gray image. In that gray image the yellow duck, at (1.00 + 0.84 + 0.20) / 3 = 0.68, comes out darker than the pale sky, at 0.87. Set the weights to red − blue, and the output is red minus blue at every pixel: 0.80 inside the head, −0.65 on the water. That’s the same pattern figure 4.3 found, but sharper, because there’s no 3×3 average blurring the edges.

Move the sliders and the output changes everywhere at once. Every pixel uses the same three weights.

What it does, and doesn’t, do

A 1×1 convolution reads no neighbors at all. It can’t find edges or shapes, because those are about how neighboring pixels relate. What it can do is mix channels: at each pixel, it turns $C_\text{in}$ numbers into new combinations of them.

That’s a dense layer from chapter 1, applied to each pixel’s list of channel values, with the same weights at every pixel. A 1×1 layer with $C_\text{out}$ kernels is a dense layer with $C_\text{in}$ inputs and $C_\text{out}$ outputs, run once per pixel. Its parameter count is the dense layer’s:

$$\text{parameters} = (1 \times 1 \times C_\text{in} \times C_\text{out}) + C_\text{out} = (C_\text{in} \times C_\text{out}) + C_\text{out}$$

A 1×1 kernel also never shrinks the map. It fits at every pixel, so with stride 1 the output has the same width and height as the input. Only the number of channels changes.

Why networks use them

Changing the number of channels is exactly what makes 1×1 convolutions useful. Deep in a network, a map might have 256 channels. A 3×3 layer that reads it and makes 64 channels costs $(3 \times 3 \times 256 \times 64) + 64 = 147{,}520$ parameters. A 1×1 layer that does the same, 256 channels in and 64 out, costs $(256 \times 64) + 64 = 16{,}448$, about a ninth as many.

So networks use 1×1 layers to shrink the channel count before an expensive 3×3 layer, and sometimes to grow it back afterward. GoogLeNet (2014) uses them this way in every one of its main blocks, and ResNet’s deeper versions (2015) build their blocks as 1×1, then 3×3, then 1×1. The idea of a 1×1 convolution as a small dense layer at every pixel comes from a 2013 paper by Min Lin, Qiang Chen and Shuicheng Yan, Network in Network.