Transposed convolution.

Chapter 7 ended with an operation that turned a small map into a bigger one. Going backward through a convolution, each output’s gradient was multiplied by the whole kernel, and that 3×3 stamp was added onto the window the output had read. Four gradients became sixteen values.

Use that operation as a layer and you get a transposed convolution: each input value is multiplied by the whole kernel, and the resulting k×k stamp is added onto the output. Input $(r, c)$ puts its stamp with its top-left corner at $(r \times s, c \times s)$, where $s$ is the stride. Where stamps overlap, they add. Like a convolution’s, its kernel weights are learned in training, so unlike nearest neighbor and bilinear it can learn how each value should be spread out.

By hand

Take a 2×2 input and a 3×3 kernel of ones, at stride 2:

$$x = \begin{bmatrix} 1 & 2 \\ 3 & 4 \end{bmatrix}$$

The top-left value, 1, stamps 1 × the kernel, a 3×3 block of 1s, at corner (0, 0). The top-right value, 2, stamps a block of 2s at corner (0, 2), two columns over. The bottom row’s stamps land two rows down, at (2, 0) and (2, 2). The stamps are 3 wide but only 2 apart, so neighbors overlap by one row or column, and there the values add:

$$\begin{bmatrix} 1 & 1 & 3 & 2 & 2 \\ 1 & 1 & 3 & 2 & 2 \\ 4 & 4 & 10 & 6 & 6 \\ 3 & 3 & 7 & 4 & 4 \\ 3 & 3 & 7 & 4 & 4 \end{bmatrix}$$

Column 2 is covered by both a left and a right stamp, and row 2 by both a top and a bottom stamp. The middle, (2, 2), is covered by all four: 1 + 2 + 3 + 4 = 10.

Step through figure 8.2 to watch the four stamps land. Click a weight to change it, and every stamp changes the same way, because every input value uses the same kernel. That’s weight sharing again, as in a convolution. Then try other settings:

• At stride 1, the stamps overlap heavily, and the output grows by only $k - 1$.
• With a 2×2 kernel at stride 2, the stamps tile the output exactly. Each output is one input value times one weight.
• With a 2×2 kernel at stride 3, the stride is bigger than the kernel. The stamps leave gaps, and the outputs in the gaps get nothing at all.

Output size

The last stamp’s corner lands at $(n - 1) \times s$, and the stamp reaches $k$ further. So without padding, the output size is

$$o = (n - 1)\,s + k$$

In the example, $(2 - 1) \times 2 + 3 = 5$.

Padding in a transposed convolution works in reverse too. Instead of adding $p$ rings of zeros around the input, it crops $p$ rows and columns off every side of the output. With padding:

$$o = (n - 1)\,s - 2p + k$$

Stride works in reverse as well. In a convolution, stride 2 roughly halves the size; here it roughly doubles it. A common choice for exact doubling is a 4×4 kernel at stride 2 with padding 1: a 16×16 map becomes $(15 \times 2) - 2 + 4 = 32$, so 32×32. Another is a 2×2 kernel at stride 2 with no padding: $(15 \times 2) + 2 = 32$.

Not every setting doubles exactly. A 3×3 kernel at stride 2 with padding 1 turns 16 into $(15 \times 2) - 2 + 3 = 31$, one short. Libraries have an extra setting, called output padding, that adds the missing row and column on one side. The next module shows why the size can come out one short.