Numbers, vectors and learning from zero

1.1 An image is numbers

Assumes: nothing. Gives: pixels as three numbers from 0 to 255, a picture as a block of numbers, and the normalisation that every picture goes through before the network sees it.

A computer stores a picture as a grid of tiny coloured squares called pixels. Each pixel holds 3 numbers: how much red, how much green and how much blue it shows. These three are the picture's colour channels. In a stored photograph each of them is a whole number from 0 to 255, where 0 means none of that colour and 255 the most that the file can hold. Fact

So a picture is a block of numbers. The network, the program that learns from these numbers (1.3 builds one), keeps such a block as a tensor, and the tensor's shape lists how long each of its sides is. In training, the long run of small corrections by which a network learns (1.4), it is shown pieces cut out of each photograph, called crops, each resized to a square of a fixed size: usually a part of the photograph, at most all of it. A crop 224 pixels on a side has the shape 3 × 224 × 224, which makes 150,528 numbers. Fact

Before those numbers enter the network they are normalised in two moves. First every value is divided by 255, so that it lies between 0.0 and 1.0. Then, channel by channel, the mean of that colour over ImageNet, a large public collection of photographs, is subtracted, and the result is divided by that colour's standard deviation, a measure of how widely the values spread around their mean. The means are 0.485 for red, 0.456 for green and 0.406 for blue; the standard deviations are 0.229, 0.224 and 0.225. Fact

Our runs use exactly these six numbers: they build their crops with SelfPatch's own augmentation code, which cuts the crops, changes them at random (Part 3) and, as its last step, applies this normalisation. Fact

After normalisation, 0 means "as strong as an average ImageNet photograph in this colour", a positive number means stronger, a negative number weaker, and a step of 1 is one standard deviation. Every value ends up between about −2.118 (no red at all) and 2.640 (full blue): small numbers on one scale, centred near zero. Toy

Lab 1A shows these numbers for any pixel you point at: on the drawn sample scene, on the course's photograph, or on a picture of your own, which never leaves your browser. Its mixer makes a pixel from three numbers you choose. Toy

1.2 Vectors, dot products and cosine similarity

Assumes: 1.1 (a pixel is three numbers). Gives: vectors, the dot product, the length of a vector, cosine similarity as one number from −1 to 1, and the unit sphere.

A vector is an ordered list of numbers, and how many numbers it holds is its dimension. A pixel is a vector of dimension 3. The network cuts each crop into squares of 16 by 16 pixels, called patches, and turns every patch into a vector of 384 numbers, called a token (Part 2). Fact

You can draw a vector of 2 numbers as an arrow from the origin: (2, 1) means 2 steps to the right and 1 step up. A vector of 3 numbers is an arrow in space. A vector of 384 numbers cannot be drawn, yet every rule below works for it in exactly the same way: the sums just run over 384 terms instead of 2 or 3. Toy

The dot product of two vectors multiplies their matching numbers and adds the products: a · b = a₁b₁ + a₂b₂ + … It is large and positive when the two arrows point the same way, near zero when they stand at right angles, and negative when they point apart. Toy

The length of a vector, written ‖a‖, is the square root of its dot product with itself, as Pythagoras says: ‖(2, 1)‖ = √(2² + 1²) = √5, about 2.236. Toy

The dot product also grows when either arrow gets longer. To compare directions only, divide it by both lengths. The result is the cosine similarity, cos(a, b) = a · b / (‖a‖ ‖b‖): the cosine of the angle between the two arrows. It always lies between −1 (opposite directions) and 1 (the same direction), with 0 for a right angle, and making an arrow longer or shorter never changes it. That is why "similar" can be one number between −1 and 1. Toy

Dividing a vector by its own length gives a vector of length 1 that points the same way, a unit vector. In two dimensions every unit vector ends on the circle of radius 1; in three dimensions, on a sphere; in 384 dimensions, on what is still called the unit sphere. For two unit vectors the cosine is simply their dot product. Toy

Every method in this course compares vectors this way. DINO's head, a small network that DINO puts after the one that makes the tokens (Part 3), scales a vector of 256 numbers to length 1, so those vectors lie on a sphere. SelfPatch ranks a patch's neighbours by the cosine similarity of their tokens. Our rule ranks every pair of patches in a crop by the cosine of their tokens in the teacher, the second of the two networks that DINO-style training keeps (Part 3). And the decoder of Part 6, which reads object outlines out of the network, links two patches when the cosine of their keys, vectors that the network's attention computes for every patch (Part 2), is above 0.2. Fact

Lab 1B lets you drag two arrows and watch the dot product, the lengths, the cosine and the angle, then type two vectors of three numbers and see every product and square written out. Toy

1.3 A layer: weights, a bias and a bend

Assumes: 1.2 (the dot product). Gives: what one layer of a network computes, what a bend such as ReLU or GELU adds, and a two-layer network drawn as arrows with its numbers.

A neural network is a chain of layers, and one layer takes a vector in and gives a vector out. Each output number is the dot product of the input with one row of weights, plus one more number, the bias: z = W x + b. PyTorch, which our code is written in, calls this a linear layer. Fact

With 3 inputs and 2 outputs, W has 2 rows of 3 weights and b has 2 numbers. The weights and the biases are the network's parameters: the numbers that training adjusts. Toy

The first layer of the real model turns each patch's 768 numbers into a token of 384: that takes 384 × 768 weights plus 384 biases, 295,296 parameters. The whole backbone, the main body of the network, has 21,664,128. Fact

After the weighted sums, the layer bends every output with a fixed function, its activation. The simplest is ReLU: it keeps a positive number and turns a negative one into 0, ReLU(z) = max(0, z). Fact

The real model uses a smoother bend, GELU: GELU(z) = z · Φ(z), where Φ(z) is the chance that a number drawn from the standard bell curve is below z. The transformer, the network of Part 2 that makes the tokens, is a stack of blocks, and inside every block a small two-layer network, the MLP, widens each token from 384 numbers to 1,536, applies PyTorch's exact GELU and narrows it back to 384. Fact

GELU lets large positive numbers through almost unchanged, turns large negative ones into almost 0, and bends gently in between: GELU(1) = 0.841, GELU(−0.5) = −0.154, and its lowest value, about −0.170, comes at z ≈ −0.752. Toy

Why bend at all? Without a bend, two layers in a row are one layer in disguise: a weighted sum of weighted sums is again a weighted sum. The figure shows it with numbers. Toy

Lab 1E is the first layer of the figure, live: type any input, weight or bias, choose the bend, and add the second layer. Toy

1.4 A loss, a gradient and a step downhill

Assumes: 1.3 (parameters). Gives: what a loss is, what a gradient says, how gradient descent and the learning rate move the parameters, and a line fitted to three points step by step.

How does a network find good weights? It needs a score of how wrong it is: one number, the loss, that is small when the outputs are good and large when they are bad. Training is the search for parameters that make the loss small. Toy

Take the smallest network there is, a line ŷ = w x + b with two parameters, and three points to fit: (1, 1), (2, 3) and (3, 2). For each point the error is the prediction minus the target, e = ŷ − y. The loss is the mean squared error, L = (e₁² + e₂² + e₃²) / 3: squaring makes every miss count as positive, and big misses count far more than small ones. Toy

The gradient says, for each parameter, how fast the loss changes when that parameter moves a little: ∂L/∂w and ∂L/∂b. Its sign points the way the loss rises, so the opposite way lowers it. Toy

Gradient descent repeats one move: compute the gradient, then step every parameter a little against it, w ← w − η ∂L/∂w. The learning rate η sets the size of the step: too small and training crawls, too large and each step overshoots the bottom and the loss grows. Toy

A real network does exactly this with every parameter at once. Its gradient comes from backpropagation: the loss's sensitivity is passed backwards through the layers, one layer at a time, each layer multiplying it by its own slope, as the walk of 1.3 did for one layer. Toy

Our runs do not step with plain gradient descent: they use AdamW, an optimiser that keeps running averages of each parameter's gradient and of its square and divides the one by the square root of the other (Part 5 walks through it). Their learning rate peaks at 1e-5 (0.00001) for the backbone and at 1e-4 (0.0001) for the heads, the small networks on top of the backbone. Fact

Lab 1C runs the worked example, and as many steps after it as you like. It draws the loss for every choice of w and b as a landscape, so you can watch each step go downhill, change the learning rate until the steps overshoot, and play the next steps as a film. Toy

1.5 Softmax and temperature

Assumes: 1.4 (a loss and its gradient). Gives: how softmax turns scores into shares that add up to 1, what dividing by a temperature does, entropy, and the arithmetic for three scores at three temperatures.

A network often ends with a list of scores, one per option, and needs shares: positive numbers that add up to 1, like the slices of a pie, also called a probability distribution. Softmax makes them: raise e, a constant close to 2.718, to the power of each score, then divide each result by the sum of all of them. A larger score always gets a larger share. Fact

Before the softmax, the scores can be divided by a temperature τ. A small τ stretches the gaps between the scores, so the largest takes almost everything and the shares get sharper; a large τ shrinks the gaps and the shares get flatter. Fact

DINO uses two networks: a student, which learns from the loss, and a teacher, whose shares the student learns to match (Part 3). This course meets three real temperatures: 0.1 for the student; 0.04 for the teacher in SelfPatch's code and in our runs (DINO's own recipe raises the teacher's from 0.04 to 0.07 over its first 30 epochs); and 0.2 in our rule's loss. Fact

Entropy measures how spread the shares are: H = −Σ p ln p, where ln is the natural logarithm. It is 0 when one option takes everything and largest when all the shares are equal, ln 3 ≈ 1.099 for three options. Computers also subtract the largest score from every score before raising e to it: the shares stay the same and the powers never grow too large to store. Toy

Lab 1D has the three scores, the temperature, the bars, the entropy and every step of the arithmetic, the shares across all temperatures at once, and, when you pick a target, the pull and the pushes of its gradient. Toy

1.6 Learning without labels

Assumes: 1.4 (a loss) and 1.5 (shares). Gives: what supervised and self-supervised learning are, why learning without labels is possible, and the danger that every such method has to avoid.

A label is a human-made answer attached to a picture: a name, a box or an exact outline. In supervised learning the loss compares the network's answer with the label, as the line of 1.4 was compared with its three targets. Toy

This thesis never shows a label to training: the runs read images only and record that no annotation was accessed, and no label enters a loss, a gate (an automatic check that reads a run's numbers), a setting of the run or the decoder (Part 6). Fact

In self-supervised learning the task is made from the pictures themselves. DINO's authors call their method self-distillation with no labels: the network sees several crops of one photograph and has to describe them in ways that agree. Fact

Why can that work? A photograph is full of regularities: neighbouring pixels usually belong to the same thing, and a dog seen in a large crop is still a dog in a small one. A task that can only be solved by noticing such regularities forces the network to store them in its numbers, and nobody has to say what a dog is. Toy

The three methods of this course make their tasks in different ways. DINO asks different crops of the same photograph for matching descriptions; SelfPatch asks each patch to agree with its most similar neighbours; our rule asks each patch to agree with the most similar patches anywhere in its crop and to disagree with the crop's least similar fifth. Fact

Such a task has a cheap way out. If every picture gets the same description, every pair agrees perfectly, and the network has learned nothing about pictures. This failure is called collapse, and every method without labels has to rule it out. Toy

DINO's authors name two forms of collapse, an output uniform along all dimensions and an output dominated by one, and use centring against the second and sharpening against the first (Part 3). Fact

Labels appear in this project only outside training, to measure how good the masks of a checkpoint are. A checkpoint is a saved copy of the network's parameters, and its masks are the object outlines that the decoder reads out of it. The masks are measured on a validation panel of labelled photographs: each read is recorded in a ledger, none is used to choose a checkpoint, and the test panels stay sealed (Part 6). Fact

1.7 What a GPU does, and why batches exist

Assumes: 1.3 (a layer) and 1.4 (a step). Gives: what a GPU is for, what a batch, a step and an epoch are, and the numbers of our runs.

A GPU is a chip that does thousands of small multiplications at the same time, which is exactly what the layers of 1.3 ask for: the same arithmetic on many numbers at once. Toy

A batch is the group of images whose gradients are averaged before one step: an average over many images points more steadily downhill than the gradient of one image, and the GPU computes many images at once anyway. One pass over all the training images is an epoch. Toy

Our runs trained on one NVIDIA RTX A4000 card of a shared university machine with 8 such cards of 16 GB each. Each step used a batch of 256 images, which the card computed as 8 micro-batches of 32 images, adding up their gradients before taking the step. Fact

Run 1's one-card step took a median of 3.706 seconds over its steps 2 to 50: the middle value once those times are sorted. It was measured once, on one seed, the number that fixes a run's random choices, such as the crops it cuts (Part 5). It says how long a step took, nothing more. Reading

That is the whole toolkit: numbers, vectors and their cosines, layers with a bend, a loss walked downhill, shares from a softmax, and a task that needs no label. Part 2 builds the Vision Transformer out of these pieces. Toy