The first layer.

Looking at the kernels

The first layer’s 8 kernels are the easiest part of the network to look at: each is just 25 numbers, drawn as a 5×5 grid. Figure 9.2 draws each one above its channel’s map, red for positive weights and blue for negative, the colors the book uses for any value that can go below zero.

Most don’t look like chapter 2’s kernels. Sobel_x is a clean pattern, a column of negatives beside a column of positives, so it’s easy to say what it finds: places where the image gets brighter from left to right. A few trained kernels have a visible structure like that; kernel 0 has a blue band above a red one, something like sobel_x turned on its side. Most are blotchy, with red and blue patches of uneven shapes. Looking at the weights alone, it’s hard to say what they find.

A probe

So instead of reading the weights, test them. Pick an input simple enough to reason about, change one thing about it, and record how a channel responds. This chapter uses a straight white line through the middle of the image, and turns it.

For each angle from 0° (flat) to 175°, in steps of 5°, run the line through the network and record each channel’s largest value anywhere in its map. That’s its response to the line at that angle. Plotting a channel’s response against the angle gives its tuning curve: how strongly the channel responds as one property of the input changes, here the angle of the line. A flat curve means the channel doesn’t care about the angle. A curve with one sharp peak means it responds to lines at one angle and ignores the rest.

To put a number on that, this chapter uses a channel’s selectivity: (max − min) / max over its tuning curve. A channel that gives the same response at every angle has selectivity 0. A channel that goes completely silent at some angle, a response of 0 after ReLU, has selectivity 1.

Drag the slider in figure 9.2, or point at any of the charts, to turn the line. Each channel’s map shows where along the line it responds.

What the first layer does

Every first-layer channel has a favorite angle: its curve rises to a peak somewhere. But none of them is narrow. Their selectivities run from 0.11 to 0.70, so even the most selective channel still gives at least 30% of its peak response at every angle. Channel 1, at 0.11, responds almost the same whatever the angle.

That means no first-layer channel is a detector for lines at one angle. Each prefers some angles and responds more weakly to the rest, and the eight prefer different angles. Which angle a line has is spread across all eight channels at once: only by comparing the eight responses could you tell a line at 30° from one at 120°.

A straight line is one probe. Real digits have curves, ends and crossings too, and a tuning curve says nothing about how a channel responds to those. What it does show is that the trained kernels respond to the direction of a stroke, which is what you’d want from a first layer reading handwriting, and that they do it broadly, not one angle per channel.

The next module asks what the second layer does with these eight maps.