Making maps bigger.

Chapter 6 showed why networks shrink their maps: every stride or pooling layer makes each later value see a bigger patch of the image. A network that classifies the whole picture can keep shrinking until only a few values are left.

Some jobs need an answer for every pixel instead. Chapter 7’s three-layer network marked each pixel as duck or not duck. Doing that for every pixel of an image is called segmentation: the output is a map the same size as the input, with a label for each pixel. Medical images are a common use, for example marking which pixels of a scan show a tumor. Chapter 7’s network kept its map the same size the whole way, so every layer saw only a small patch. To see more, a segmentation network shrinks its maps like any other CNN, and then has to grow them back to the input’s size.

Making a map bigger is called upsampling: turning an n×n map into a bigger one, usually 2 or 4 times as wide and tall. The simplest ways use a fixed rule, with no weights.

Nearest neighbor

Nearest neighbor upsampling copies each value into a block. Upsampling ×2, every value becomes a 2×2 block of itself:

$$\begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix} \rightarrow \begin{bmatrix} 1 & 1 & 2 & 2 \\ 1 & 1 & 2 & 2 \\ 3 & 3 & 4 & 4 \\ 3 & 3 & 4 & 4 \end{bmatrix}$$

The name comes from where each output looks: lay the bigger grid over the smaller one, and every output pixel takes the value of the input pixel it’s nearest to. It’s fast, and the result looks blocky.

Bilinear

Bilinear upsampling blends instead of copying. Along a row, each output takes its two nearest inputs and mixes them, giving more weight to the closer one. It does the same along each column, which is where the “bi” comes from.

Take the row [0, 4, 8] and upsample it ×2. Lay the six outputs over the three inputs so the row’s two ends line up. Each input then covers two outputs, and output $i$, counting from 0, sits at input position $(i + 0.5) / 2 - 0.5$:

• Output 1 sits at position 0.25, a quarter of the way from input 0 to input 1. So it’s (0 × 0.75) + (4 × 0.25) = 1.
• Output 2 sits at 0.75, closer to input 1: (0 × 0.25) + (4 × 0.75) = 3.
• Output 3 sits at 1.25: (4 × 0.75) + (8 × 0.25) = 5.
• Output 0 sits at −0.25, before the first input, so it copies the end value, 0. Output 5 likewise copies 8.

The result is [0, 1, 3, 5, 7, 8]. The weights of each blend always add up to 1, so a flat map stays flat. In 2D, bilinear blends along the rows first, then along the columns, so each output mixes up to four inputs.

Figure 8.1 starts from the duck at 8×8 and makes it 4 times as big both ways. Point at an output pixel to see which inputs it came from and with what weights. Nearest neighbor gives back big flat blocks. Bilinear gives smooth ramps between them, which looks less blocky but blurs every edge. Compare both with the duck drawn directly at 32×32: the eye, the beak’s shape, the line between the body and the wing. Neither rule gets them back, because every output is computed from the 64 values of the 8×8 duck alone. Upsampling can rearrange and blend what it’s given, but it can’t recover detail that was lost when the map shrank.

That’s why the next module gives upsampling weights. A rule with weights can be trained, and a trained network can learn what to fill in, especially when it also gets help from earlier, bigger maps. U-Net, a network built to segment medical images, does both: on the way back up, it upsamples with learned weights and then combines each bigger map with the matching map from the way down, cropped to fit, which still holds the detail that shrinking lost.