Checkerboard artifacts.

Go back to the worked example from the Transposed convolution module, with a 3×3 kernel of ones at stride 2. Feed it an input of all ones instead, so any pattern in the output can’t have come from the input.

Along one row, the stamps cover the columns like this: the first stamp covers columns 0 to 2, the next covers 2 to 4, the next 4 to 6, and so on. Count how many stamps cover each column:

$$1, 1, 2, 1, 2, 1, 2, \ldots$$

Every even column after the first is covered twice, every odd column once. The same holds down the columns, and an output’s count is its row’s count times its column’s count. So inside the map, outputs get 1, 2 or 4 stamps, in a repeating grid: a flat input comes out as a pattern of bright and dark squares.

This is a checkerboard artifact: a grid-shaped pattern in an upsampled map that comes from uneven overlap between stamps, not from anything in the input. It shows up whenever the kernel size isn’t a multiple of the stride. A 3-wide stamp every 2 pixels can’t cover the row evenly.

In figure 8.4, every transposed kernel has all its weights equal, scaled so that a flat input averages out unchanged. Start with flat gray and transposed 3×3: the inside of the map is a checkerboard, and the stamps-per-column readout shows why. Try 5×5, where the columns alternate 2 and 3 stamps. Then try 4×4: each column inside is covered by exactly 2 stamps, and the inside comes out flat. Switch the input to the duck, and the uneven kernels draw the same grid over the duck.

Even overlap isn’t enough

A kernel size that’s a multiple of the stride removes the uneven overlap, but not every repeating pattern. Take a 2×2 kernel at stride 2. There’s no overlap at all, so each 2×2 block of the output is one input value times the same four weights. If training makes those four weights different, every block gets the same uneven pattern, repeating across the whole map. A network can learn weights that avoid patterns like this, but in practice they often survive training, even with kernel sizes that overlap evenly (Odena, Dumoulin and Olah, 2016).

Resize, then convolve

The fix that Odena and coauthors recommend is to split upsampling into two steps: first resize with a fixed rule, nearest neighbor or bilinear, then run an ordinary convolution with same padding. Select “resize, then conv” in figure 8.4. Every output of a convolution adds up a full window, 9 (pixel × weight) pairs for a 3×3 kernel, so away from the edges no output gets more contributions than its neighbor. A flat input comes out flat. The convolution still has learned weights, so the layer can still learn how to fill in.

Both kinds of layer are in common use. The original U-Net (Ronneberger, Fischer and Brox, 2015) upsampled with 2×2 transposed convolutions at stride 2.