Moving the duck.

In chapter 1, moving the duck three pixels to the right broke the dense classifier. Its duck output had a weight for every pixel, tied to where the duck used to be, so the moved duck met the wrong weights, and the answer flipped to empty sky.

Figure 2.4 makes the same kind of move, on the 32×32 duck, and runs the edge kernel over the result.

Move the duck one pixel to the right. The output map moves one column to the right too, and nothing else about it changes. The red edge of the head is still red, with the same value, 1.78, one column further over. Of the 900 outputs, 870 are exactly the old outputs, moved one column. Move it two pixels and 840 are; three pixels, and 810 are. Compare chapter 1, where a three-pixel move turned the dense layer’s answer from duck to empty sky.

Why the map moves with the image

Take any output of the map, and the 3×3 window it reads. After the move, that same window’s pixels sit one column further right, and the output one column further right reads them. Its weights are the same nine weights, because every position uses the same kernel. Same pixels, same weights, same products, same sum. So every output reappears, unchanged, one column over.

Only the first columns are new. The image’s left edge repeats into the gap the duck leaves behind, so the windows in those columns read pixels that weren’t in the original. For a move of $d$ pixels that’s $d$ columns of 30 outputs, which is where the counts come from: $900 - 30 = 870$, $900 - 60 = 840$, $900 - 90 = 810$.

This property has a name. A layer is shift-equivariant when moving its input moves its output by the same amount and changes nothing else. A convolution is shift-equivariant because of the two properties from the end of chapter 1. Weight sharing means every position computes the same thing. Local connections mean each output depends only on its own small patch, so what happens elsewhere in the image can’t change it.

The dense layer has neither. Its weights are different at every position, so a pattern it learned at one place means nothing at another.

Equivariant, not invariant

Shift-equivariant doesn’t mean the output stays the same. The map moves, so any single output does change: the output where the head’s edge used to be is now plain sky, 0. A layer whose output didn’t change at all when the input moved would be shift-invariant, and a convolution isn’t.

That’s what you’d want from the first layers of a network. A convolution reports what it found and where, and the “where” moves with the duck. To answer a question like “is there a duck anywhere in this picture?”, a network still needs a later step that combines outputs from many positions. Chapter 5 covers pooling, one such step, and shows that it’s not fully shift-invariant either.

Everything in this module assumes the kernel stops at every pixel. Chapter 3 introduces settings that change where the kernel lands, starting with stride.