Why “transposed”.

The name “transposed convolution” comes from writing a convolution as a matrix. It’s worth doing once, because it shows that the transposed convolution and chapter 7’s input gradient are the same operation.

A convolution is a matrix

Chapter 1 flattened an image into a list of numbers, and a dense layer multiplied that list by a matrix of weights: one row per output, one column per input. A convolution is a matrix multiplication too, just with a very particular matrix.

Take a 3×3 convolution with no padding at stride 1 on a 4×4 image, as in chapter 2’s example. Call the kernel’s weights a to i, row by row. Flatten the image into a list of 16 pixels and the output into a list of 4. Then the convolution is

$$y = M x$$

where $M$ has 4 rows, one per output, and 16 columns, one per pixel. Each row holds the nine weights in the columns of the nine pixels that output reads, and 0 in the other seven columns, because the output doesn’t use those pixels. The four rows hold the same nine weights, shifted along: that’s weight sharing, written as a matrix. A dense layer would have 64 different weights here. The convolution’s matrix uses 9, repeated, and fills the rest with zeros.

Its transpose

The transpose of a matrix swaps its rows and columns: row $j$ of $M^\top$ is column $j$ of $M$. $M$ is 4 × 16, so $M^\top$ is 16 × 4. It takes 4 numbers and gives back 16.

Read row $j$ of $M^\top$. It’s column $j$ of $M$: it lists every output that read pixel $j$, and the weight it read it through. So multiplying a list of 4 values $v$ by $M^\top$ gives pixel $j$ the sum of (value × weight) over the outputs that read it. That’s chapter 7’s input gradient exactly, with $v = G$: in chapter 7’s terms, the input gradient is $M^\top G$.

Now read it the other way, by columns of $M^\top$. Column $i$ of $M^\top$ is row $i$ of $M$: the kernel’s weights, placed on output $i$‘s window. Multiplying by $M^\top$ adds $v_i$ times that column into the result, for every $i$. That’s stamping: each value times the kernel, added onto its window. So a transposed convolution with a given kernel and stride is multiplication by $M^\top$, where $M$ is the convolution with the same kernel and stride.

Figure 8.3 starts with a 3×3 convolution at stride 2 on a 5×5 image, which also has a 2×2 output. $M$ is 4 × 25, and $M^\top$, 25 × 4, takes a 2×2 map to a 5×5 one: the transposed convolution of the last module. Click the rows of $M$ to see each output’s window. Then switch to $M^\top$ and click the pixels. The middle pixel, (2, 2), is read by all four outputs, each through a different weight, and that’s why it collected all four stamps in the last module. The corners are read by one output each.

Not an inverse

A transposed convolution restores the size of a convolution’s input, not its values. Transposing a matrix isn’t the same as inverting it, and $M$ has no inverse to begin with: it turns 25 numbers into 4, so many different images give the same 4 outputs. A transposed convolution’s weights are learned to fill in whatever helps the network’s final answer. Some papers call this layer a “deconvolution”, but that name is misleading, since undoing a convolution is a different problem.

The size can’t always be restored exactly either. With a 3×3 kernel at stride 2, chapter 3’s formula sends both a 5×5 and a 6×6 image to a 2×2 output, because of the rounding down: ⌊(5 − 3) / 2⌋ + 1 = 2 and ⌊(6 − 3) / 2⌋ + 1 = 2. The transposed convolution can only give one size back, $(2 - 1) \times 2 + 3 = 5$. Output padding is for exactly this: it picks the 6. The last module’s one-short case is the same rounding. A 3×3 convolution at stride 2 with padding 1 sends both a 31×31 and a 32×32 image to 16×16, since ⌊(31 + 2 − 3) / 2⌋ + 1 = 16 and ⌊(32 + 2 − 3) / 2⌋ + 1 = 16, so going back from 16 gives 31 unless output padding adds the missing row and column.