Dense layers.

A layer is one step of a neural network: it takes numbers in and computes new numbers from them. A network is a sequence of layers, each one reading what the layer before it produced.

A dense layer

A dense layer is a layer in which every output is computed from every input. It’s also called a fully connected layer, and PyTorch calls it nn.Linear. To compute one output, it multiplies each input by a number of its own, adds up all the products, and then adds one more number.

The numbers that multiply the inputs are the weights. Each output has one weight for every input. The number added at the end is the bias, and each output has one. Together, the weights and biases are the layer’s parameters: the numbers it learns. Training is the process that adjusts them, little by little, until the network gives good answers. Chapter 7 shows how training reaches the weights of a convolution.

Here’s one output with four inputs. Each input is paired with its own weight:

Input	Weight	Input × weight
0.2	1	0.2
0.9	−1	−0.9
0.5	2	1.0
0.1	0	0

Add up the last column and add the bias, 0.5:

$$(0.2 \times 1) + (0.9 \times (-1)) + (0.5 \times 2) + (0.1 \times 0) + 0.5 = 0.2 - 0.9 + 1.0 + 0 + 0.5 = 0.8$$

The multiply-then-add part has a name. A dot product pairs up two lists of the same length, multiplies each pair, and adds up the results. So each output of a dense layer is the dot product of the inputs with that output’s weights, plus its bias.

More than one output

A layer usually has many outputs. Every output reads the same inputs, but each has its own weights and its own bias, so each computes something different. The outputs don’t read each other. They’re computed side by side, each from the inputs alone.

Add a second output to the example above, with four weights and a bias of its own. Each output has one weight per input, and one bias, so

$$\text{parameters} = (\text{inputs} \times \text{outputs}) + \text{biases}$$

where the number of biases equals the number of outputs. Here that’s $(4 \times 2) + 2 = 10$.

What an output means depends on the job. In a network that sorts images into categories, called a classifier, the last layer has one output per category, or class. Each output is a score for its class, and the network’s answer is the class with the highest score. The scores aren’t related by any rule inside the layer. Training sets each output’s weights from labeled examples, so that the right class tends to score highest.

A dense layer on an image

Flatten the duck, and a dense layer can read it: 256 inputs. Each output then has 256 weights, one per pixel. That means you can lay an output’s weights back out in the image’s shape, as a 16×16 picture of what that output responds to.

The weights alone already decide what an output computes. If every weight is 1/256 and the bias is 0, the output adds up all 256 pixels and divides by 256: it’s the image’s average brightness. If every weight is 0 except one, which is 1, the output copies that one pixel through.

Figure 1.2 is a dense layer working as a classifier with three classes: a duck on a pond, an empty pond, and empty sky. It has one output per class.

The weights here are set by hand rather than trained, so you can see why they work. Each class’s weights are its picture minus the average of the three pictures. A score adds up pixel × weight, so a bright pixel under a positive weight pushes the score up, and a bright pixel under a negative weight pushes it down.

• Duck on a pond. The weights are positive over the duck, where its picture is brighter than the average, and slightly negative over the open water.
• Empty pond. The weights are negative over the duck and slightly negative over the open water. Apart from the one pixel where the duck’s dark eye would be, there are no positive weights at all. This class scores well when those spots are dark, as they are in a picture of an empty pond.
• Empty sky. The weights are positive over the pond, where sky is lighter than water, and negative where the duck would be.

Across the top of the image, all three pictures show the same sky, so every class’s weights there are 0. Pixels that look the same in every class can’t help tell the classes apart.

The biases even things out. The pictures differ in overall brightness, and that would shift some scores up whatever the image showed. The empty pond is the darkest picture, so its output gets the largest bias, 8.27. The duck is the brightest, and its bias is −15.86.

Pick each image in turn. The duck scores 3.38 on its own class, against −3.51 for empty sky and −5.70 for empty pond. The empty pond scores 1.59 on its own class and 0.07 on empty sky; those two pictures differ only in the water. Click an output to see its weights, the products they make with the image, and how the sum and the bias give the score.

This layer has $(256 \times 3) + 3 = 771$ parameters. That’s for three classes and an image smaller than an icon. The next module counts what happens at real sizes.