Counting parameters.

Every output of a dense layer has one weight per input and one bias. So a dense layer with $n_\text{in}$ inputs and $n_\text{out}$ outputs has

$$(n_\text{in} \times n_\text{out}) + n_\text{out} \text{ parameters.}$$

For an image, $n_\text{in}$ is the number of pixels, which grows with the square of the width. Double the width and there are four times as many pixels, so four times as many weights.

Real sizes

A 32×32 image has 1,024 pixels. A dense layer with 1,000 outputs on it has $(1{,}024 \times 1{,}000) + 1{,}000 = 1{,}025{,}000$ parameters.

Many image networks work on images 224 pixels on a side. That’s 50,176 pixels, and the same layer has $(50{,}176 \times 1{,}000) + 1{,}000 = 50{,}177{,}000$ parameters.

A phone photo is about 4,000 by 3,000 pixels, 12 million in all. A dense layer with 1,000 outputs on it has 12,000,001,000 parameters. Computers usually store each one in 4 bytes, so this one layer takes 48 gigabytes, more memory than most computers have. Using it means 12 billion multiplications for every photo. And this counts only grayscale, and only the first layer.

The convolution’s count

The rest of this book is about a different kind of layer, the convolution. Chapter 2 describes it in full; here is the part that matters for counting. A convolution has a small grid of weights, usually 3×3, and slides it across the image, computing one output at every position. The same nine weights are used at every position.

So a 3×3 convolution has 9 weights and 1 bias, 10 parameters, whatever the size of the image. It still produces one output per position, as many as there are pixels.

For a fair comparison, give a dense layer the same job: a 16×16 image in, 256 outputs out. It has $256 \times 256 + 256 = 65{,}792$ parameters. The convolution does it with 10.

The convolution can get away with so few because it gives up something the dense layer has: each output reads only a small patch of the image, not all of it. The next module shows why that’s not a loss.