// part 3 of 8

Part 3

DINO

Two networks that learn to agree about pictures without a single label. This part builds DINO one piece at a time: the student and the teacher, the crops each one sees, the head and its prototypes, centring and sharpening, the loss over the pairs of crops, the teacher that is an average of the student's past, and the recipe. Every equation gets a walk with numbers you can check by hand, every decision a why card, and the last section shows what three real models make of a real photograph.Toy

3.1 Two networks that agree without labels

Assumes: Part 1 (a loss, a gradient step, softmax with a temperature) and Part 2 (a crop becomes tokens). Gives: what DINO trains, why it needs a student and a teacher, what collapse is, and how the papers score the features afterwards.

Training changes a network's weights until its outputs become useful. Show it a batch of pictures, measure how wrong its outputs are with one number, the loss, compute the gradient and move every weight a small step downhill (Part 1). One such round is a step, and one pass over all the training pictures is an epoch. In supervised learning the loss compares each output with a label, such as the word 'dog' written under a photograph.Toy

DINO has no label anywhere. It trains on the 1,281,167 photographs of ImageNet without using their labels, and its authors call it DINO and describe it as a form of self-distillation with no labels.Fact

Distillation trains a student network to reproduce the outputs of a teacher network. In DINO the two networks share one architecture and differ only in their weights. Only the student learns from the loss. The teacher is built from the student's own past, as an exponential moving average of the student's weights, and at step 0 it is an exact copy of the student. A teacher made from the student itself is why the method is called self-distillation.Fact

Why two networks? Picture one network that must give the same answer for two crops of one picture and that learns from its own answer. The cheapest way to agree needs no looking at all: give every picture the same answer, and the two crops of any picture always agree. That failure is called collapse. A teacher that moves slowly gives the student a target that is more than its own current guess, and two operations on the teacher's side, centring and sharpening (3.4), close the two lazy exits.Toy

DINO's supplement names the two forms of collapse: whatever the input, the output is uniform along all its dimensions, or one dimension dominates it.Fact

The authors report that copying the student's weights into the teacher fails to converge, while freezing the teacher for an epoch works surprisingly well. They describe the moving average as a form of model ensembling, in which several models answer together, similar to Polyak-Ruppert averaging with an exponential decay, an average of the weights over the steps of training in which older steps count less and less, and they observe that this teacher performs better than the student throughout training, so it gives the student targets of higher quality.Fact

If the teacher comes from the student, where does anything new come from? From the views. The student must give, for one crop of a picture, the answer the teacher gives for another crop of the same picture. Describing what the picture shows is one way to do that, and centring and sharpening block the ways that need no description.Toy

One step of training, in five moves, as DINO's Algorithm 1 and its multi-crop rule run it:Fact

cut each picture of the batch into crops and change each crop at random (3.2);
pass every crop through the student and the two large crops through the teacher; each network ends in a head that scores the crop against many prototypes (3.3);
turn the scores into probabilities: the teacher's centred and sharpened, the student's softer (3.4);
measure the loss, the cross-entropy over the pairs of crops, and move the student's weights against its gradient (3.5);
move the teacher's weights a little towards the student's, and the centre a little towards the teacher's latest average (3.6).

One training step of DINO. The teacher sees the two global crops, the student every crop; the teacher's scores are centred and divided by its low temperature, the student's by its higher one; the cross-entropy over the 18 pairs of crops trains the student alone, and after the step the teacher and the centre each move a little.Fact

The why cards quote one yardstick again and again, so here it is first. How do the papers tell good features from bad ones? They freeze the trained network, so that its features no longer change, and only then bring in ImageNet's labels, to measure. A k-NN classifier stores the features of the labelled training pictures and gives a new picture the label chosen by a weighted vote of the k stored features nearest to its own; DINO lets the 20 nearest vote, the number that worked best in most of its runs. A linear classifier is one linear layer fitted with the labels on top of the frozen features. The accuracy is the share of ImageNet's validation pictures whose right class comes out first: for DINO's ViT-S/16, 74.5 % with k-NN and 77.0 % with a linear classifier. The labels judge the features after training and never enter it.Fact

A vote worked by hand, with k = 3: if the three stored pictures nearest to a new one are two dogs and a cat, the k-NN classifier answers 'dog'. The figure draws it in two dimensions, with a straight boundary beside it, the kind a linear classifier fits.Toy

Measuring frozen features with labels, a toy. Each point is one picture's features, drawn in two dimensions where real features have many more. The new picture's three nearest labelled pictures, joined to it by thin lines, are two dogs and a cat, so the k-NN vote says dog; the dashed line is a straight boundary of the kind a linear classifier fits with the labels, and the new picture falls on the dogs' side. Neither measurement changes the features.Toy

The cards also quote DINO's Table 7, which compares training choices on ViT-S/16 after 300 epochs; there DINO's own row reads 72.8 % k-NN and 76.1 % linear accuracy. The 74.5 % and 77.0 % above belong to the released ViT-S/16, whose run was configured for 800 epochs.Fact

The same table trains three other self-supervised methods on the same ViT-S/16, and several cards compare DINO with them. MoCo-v2 has a momentum encoder and a contrastive loss, InfoNCE, that compares each image with other images. BYOL trains its features to match a momentum encoder's outputs too, with a predictor on the student and a mean squared error as its loss. SwAV has no momentum teacher and uses an algorithm called Sinkhorn-Knopp on its prototypes; DINO's head is inspired by SwAV's prototype layer (3.3).Fact

Decision

Train a student network to match a teacher network that has the same architecture and is built from the student's own past: self-distillation, with no label anywhere.

Why

The authors cast self-supervised learning as knowledge distillation in which no teacher exists in advance, so they build the teacher from past iterations of the student, and they call the result self-distillation with no labels.Fact

Evidence

Without the momentum teacher, DINO's framework does not work: row 2 of Table 7 reaches 0.1 % k-NN and 0.1 % linear accuracy against 72.8 % and 76.1 % for DINO, both measured afterwards with ImageNet's labels.Fact

What it does to training

At every step the gradient flows into the student only; the teacher's answers are fixed targets for that step, and the teacher moves afterwards, by averaging the student's weights.Fact

If you changed it

A teacher that is a copy of the student, or the student of the previous iteration, fails at 0.1 % k-NN accuracy; the student of the previous epoch works, at 66.6 % against 72.8 % for the moving average.Fact

3.2 Many views of one picture: multi-crop

Assumes: 3.1 and the patches of Part 2. Gives: the crops and their changes, who sees which crop, and why small crops answer for large ones.

Each image becomes several views, which are random crops. In SelfPatch's recipe, which our runs use, each image gives 2 large global crops and 8 small local crops per step, 10 in all. A global crop covers 0.25 to 1.0 of the image's area and is resized to 224 × 224 pixels; a local crop covers 0.05 to 0.25 and is resized to 96 × 96, both with bicubic interpolation (2.2). Mixing large and small crops in this way is called multi-crop.Fact

The released DINO ViT-S/16 used 10 local crops, so its student saw 12 crops of every image. DINO's code defaults differ again (global crops of 0.4 to 1.0 of the area, local crops of 0.05 to 0.4, 8 local crops), and the paper describes global views as covering, for example, more than 50 % of the image and local views less than 50 %.Fact

Each crop is also changed at random. These changes are augmentations, and they make two views differ in more than their framing. The recipe of the official code:Fact

mirror left to right with probability 0.5;
colour jitter with probability 0.8, with strength 0.4 for brightness and contrast, 0.2 for saturation and 0.1 for hue; a strength s multiplies the brightness, contrast or saturation by a random factor from 1 − s to 1 + s;
turn grey with probability 0.2;
Gaussian blur with probability 1.0 for the first global crop, 0.1 for the second and 0.5 for each local crop, at a random radius of 0.1 to 2.0 pixels;
solarise, which keeps every value below 128 and turns every value v of 128 or more into 255 − v, only in the second global crop and with probability 0.2.

Now the split. In DINO all crops pass through the student while only the 2 global crops pass through the teacher, which, the authors write, encourages local-to-global correspondences. SelfPatch changes the split, as Part 4 explains: its teacher processes all 10 crops.Fact

What does that ask of the student? A local crop of 96 pixels holds 6 × 6 = 36 patches against the 196 of a global crop, often a corner of the scene. From that corner the student must produce what the teacher says about a view that covers most of the picture. Describing the part in a way that agrees with the whole is a strong hint about what the whole contains.Toy

Lab 3A draws one random set of crops on a picture: the drawn sample, your own image, or one of the course's photographs. Press 'New crops' a few times, switch the changes on, then switch between DINO and SelfPatch to see which crops the teacher gets.Toy

Ten views of one picture

Which views of the picture does each network see?

Toy

What you are looking at: 2 global and 8 local crops, outlined on the picture and shown as the small pictures the networks receive, with switches for the augmentations, for the method (DINO or SelfPatch) and for highlighting the crops that the teacher or the student sees.Fact

Real model versus this toy: the crop rule, its area ranges and the two output sizes are the real ones. The colour jitter and the blur are simplified stand-ins: this toy multiplies each colour by a random factor and blurs with a box filter. A switch that is on applies to every crop it can, unless 'Only with the recipe's chances' is on too, which applies each change by chance with the recipe's probabilities, as training does.Toy

Decision

Cut every image into two global crops of 224 pixels and several local crops of 96 pixels: in SelfPatch's code and our runs the global crops cover 0.25 to 1.0 of the area and 8 local crops cover 0.05 to 0.25; the released DINO run used 10 local crops.

Why

All crops pass through the student and only the global views through the teacher, which, the authors write, encourages local-to-global correspondences.Fact

Evidence

Without multi-crop, k-NN accuracy falls from 72.8 % to 67.9 % (Table 7, row 4). In the supplement, two global and ten local crops reach 74.6 % linear accuracy after 24.2 hours of training where two global crops alone reach 72.5 % after 45.9 hours, at a peak memory of 15.4 GB per GPU against 9.3 GB.Fact

What it does to training

Each image gives 18 loss pairs, 16 of which ask a small crop for a large crop's answer, and a 96-pixel crop holds 36 patches against 196, so the extra views cost far less than extra global crops would.Toy

If you changed it

The supplement moves the scale s that separates local from global crops: 0.08 gives 65.6 % and 0.32 gives 69.8 % k-NN accuracy, best near 0.3. Adding 2, 6 and 10 local crops raises the linear accuracy each time, by less from 6 to 10 at 300 epochs.Fact

Decision

Change every crop at random: mirror it, jitter its colours, turn it grey, blur it, and solarise the second global crop, each with a fixed probability.

Why

DINO follows BYOL's augmentations, colour jittering, Gaussian blur and solarisation, together with multi-crop.Fact

Evidence

The probabilities of the official code: mirror 0.5, colour jitter 0.8, greyscale 0.2, blur 1.0 for the first global crop, 0.1 for the second and 0.5 for each local crop, and solarise 0.2 for the second global crop only.Fact

What it does to training

Two views of one image now differ in colour, sharpness and brightness as well as in framing, so an answer that rests on colour alone cannot make them agree; an answer that describes what the image shows can.Toy

If you changed it

Without the changes, two views of one picture differ only in their framing. Switch them on and off in Lab 3A and compare the small pictures.Toy

3.3 The head and the prototypes

Assumes: 3.1, a layer and GELU (Part 1), the class token (Part 2). Gives: how a crop becomes scores over 65,536 prototypes, and why a head sits between the backbone and the loss.

The student and the teacher must each give their answer for a crop in a form that can be compared. The projection head produces it. In DINO the output of the class token feeds the head, and the head exists only for training: the features that later tasks use are the backbone's output.Fact

The head is a three-layer MLP. It maps the class token's 384 numbers to 2,048, then to 2,048 again, then to a 256-number bottleneck, with GELUs in between; this MLP alone has 5,509,376 parameters. The bottleneck vector is then L2-normalised, scaled to length 1, so that it lies on the surface of a sphere. A last weight-normalised layer, which stores each of its weight vectors as a direction and a length, compares it with 65,536 learned directions, the prototypes, and gives one score per prototype.Fact

In SelfPatch's code, and so in our runs, every prototype also has length 1, so each score is the cosine between the crop's vector and that prototype, a number from −1 to 1; the released DINO run let those lengths be learned.Fact

One picture of the prototypes, as an analogy only: 65,536 categories that the network invents for itself, and the student must sort one crop into the same categories as the teacher sorts another crop of the same picture.Toy

The course's real models are SelfPatch-family networks, and their backbone has no class token: a 224-pixel crop leaves it as 196 tokens of 384 numbers. Converting DINO's backbone for that network drops 1,536 parameters (the class token, its position row and the final LayerNorm), so the converted network no longer computes DINO's own forward pass. In those networks the class output that feeds this head comes from SelfPatch's aggregation head, its own class token attending over the patch tokens (Part 4).Fact

The projection head of DINO. Bar heights hint at layer widths without being to scale; the circle stands for the sphere of 65,536 prototypes, a few of them drawn, with the head's output vector pointing out from its middle. The two small charts at the bottom are the teacher's and the student's probabilities over five prototypes from the worked example of 3.5, a toy; the widths and the temperatures are DINO's.Fact

Decision

Put a projection head between the backbone and the loss: a three-layer MLP from 384 to 2,048 to 2,048 to a 256-number bottleneck, L2 normalisation, then a weight-normalised layer of K = 65,536 outputs, and no batch normalisation anywhere.

Why

In the supplement's words, as in other self-supervised frameworks, a projection head improves the accuracy greatly, while the features used downstream are the backbone's; the design is inspired by the projection head with a prototype layer of SwAV, and it has no batch normalisation.Fact

Evidence

With the L2 bottleneck, 2, 3 and 4 linear layers give 62.2, 68.0 and 69.3 % k-NN accuracy; without it, 1 and 2 layers give 61.6 and 62.9 % and 3 or 4 layers fail at 0.1 %. GELU against ReLU: 69.7 against 68.9 %; batch normalisation in the heads: 68.6 against 69.7 % without.Fact

What it does to training

The head turns a crop's summary into the scores the loss compares, and the gradient of the loss flows back through the head into the backbone. Whatever is peculiar to the training task can settle in the head, while the backbone's tokens, which later tasks and our decoder read, stay general.Toy

If you changed it

Without the L2 bottleneck, a head of three or four linear layers does not train (0.1 %): the bottleneck is what keeps a deep head stable.Fact

3.4 Centring and sharpening

Assumes: 3.3 and softmax with a temperature (Part 1). Gives: the teacher's distribution, the two forms of collapse, and how two opposite pushes avoid both.

Scores become probabilities through a softmax after division by a temperature τ, and a smaller τ makes the result sharper, so the top score takes more of the probability. The result is a probability distribution: one number per prototype, none negative, all adding up to one.Fact

The student divides its scores by τ_s = 0.1. SelfPatch's code and our runs divide the teacher's by τ_t = 0.04 for the whole run, so the teacher's distribution is sharper than the student's; DINO itself raises the teacher's temperature from 0.04 to 0.07 during the first 30 epochs.Fact

Nothing so far says what the right answer is, so the two networks could agree in a useless way, by giving every picture the same answer. The authors watch two numbers. The teacher's entropy, a measure of how spread out its distribution is, goes to 0 when one prototype dominates and to its largest value, ln K, when every prototype gets the same share. The KL divergence from the teacher to the student, a distance between the two distributions that is 0 when they are equal (3.5 works one out), goes to 0 in both cases: a KL of zero means a constant output.Fact

DINO's answer to both forms is a pair of opposite pushes. Centring keeps a running average of the teacher's scores, the centre, and subtracts it from the teacher's scores before the softmax; after every step the centre moves a tenth of the way towards the batch's average, c ← 0.9 c + 0.1 × (batch mean of the teacher's outputs). Sharpening is the low teacher temperature. Centring stops one prototype from dominating yet pushes towards a uniform output, sharpening pushes the other way, and the authors find that using both together avoids collapse with a momentum teacher.Fact

In DINO the centre averages the teacher's outputs for the 2 global crops; SelfPatch's code averages the image-level outputs of all 10 teacher crops.Fact

The teacher's distribution, with centring and sharpening

P_t(x)⁽ⁱ⁾ = exp((g_θt(x)⁽ⁱ⁾ − c⁽ⁱ⁾) / τ_t) / Σ_k=1..K exp((g_θt(x)^(k) − c^(k)) / τ_t)

c ← m · c + (1 − m) · (1/B) Σ_b=1..B g_θt(x_b)

P_s(x′)⁽ⁱ⁾ = exp(g_θs(x′)⁽ⁱ⁾ / τ_s) / Σ_k=1..K exp(g_θs(x′)^(k) / τ_s)

DINO's Eq. (1) and Eq. (4); its Algorithm 1 applies the centre and the teacher's temperature together.Fact

x, x′: two views of one image: x a global crop of 224 × 224 pixels, seen by the teacher; x′ another crop, global or a local one of 96 × 96 pixels, seen by the student
g_θt(x): the teacher's scores for x: one number per prototype, a vector of K = 65,536 numbers in DINOFact
θ_t, θ_s: all the teacher's and all the student's weights, backbone and head: two lists of numbers of the same length, one number per weight
c: the centre: one number per prototype, a vector of 65,536 numbers that the gradient never trains
τ_t, τ_s: the temperatures, two scalars: 0.04 for the teacher in our runs, 0.1 for the studentFact
m: the centre's rate, a scalar: 0.9 (in 3.6 the same letter names the teacher's momentum, another number)Fact
B: the number of teacher outputs the centre averages at one step, a whole number: 2 for each image of the batch in DINO, 10 for each image in SelfPatch's codeFact
P_t(x), P_s(x′): two probability distributions over the K prototypes, each adding up to one

Three prototypes and a batch of two pictures, A and B, each seen in one global crop. The teacher's scores are A = [0.30, 0.10, 0.20] and B = [0.30, 0.25, 0.05]. Prototype 1 scores 0.30 for both: a lead that every picture shares, the start of the first form of collapse. Logarithms are natural, so entropies come in nats, and every value is rounded after the arithmetic, which keeps every digit: redoing a step from the rounded numbers can move its last digit.Toy

Without centring, at τ_t = 0.04: A's scores become 7.50, 2.50 and 5.00 after the division, and the softmax gives 0.918, 0.006 and 0.075; B's become 7.50, 6.25 and 1.25 and give 0.776, 0.222 and 0.001. Prototype 1 wins both pictures.Toy

The centre. The batch mean of the scores is [0.300, 0.175, 0.125]. One update from a centre of [0.20, 0.15, 0.15] moves each entry a tenth of the way: 0.9 × 0.20 + 0.1 × 0.300 = 0.2100, then 0.1525 and 0.1475. To keep the arithmetic short, suppose recent batches looked like this one, so that the centre has settled at their mean, c = [0.300, 0.175, 0.125].Toy

With centring, at τ_t = 0.04: A − c = [0.000, −0.075, 0.075], divided by 0.04 gives [0.000, −1.875, 1.875]; the exponentials are 1.000, 0.153 and 6.521, which add up to 7.674, so the probabilities are 0.130, 0.020 and 0.850. B mirrors A: 0.130, 0.850 and 0.020. The shared lead is gone, and each picture has its own winner.Toy

Teacher's distribution	A: 1	A: 2	A: 3	B: 1	B: 2	B: 3	Entropy of A (nats)
no centring, τ_t = 0.04	0.918	0.006	0.075	0.776	0.222	0.001	0.305
centred, τ_t = 0.04	0.130	0.020	0.850	0.130	0.850	0.020	0.482
centred, τ = 0.1	0.279	0.132	0.590	0.279	0.590	0.132	0.934

Each entropy is −Σ p ln p over the three probabilities; for A centred at 0.04, −(0.130 ln 0.130 + 0.020 ln 0.020 + 0.850 ln 0.850) = 0.482 nats. With centring but a teacher as soft as the student, τ = 0.1, A gives 0.279, 0.132 and 0.590, and the entropy rises from 0.482 to 0.934 nats, close to ln 3 = 1.099 for a flat answer: a target this soft tells the student little. B's centred rows hold A's numbers in another order, so their entropies are A's; without centring B's entropy is 0.541. Probabilities are rounded to three decimals, so a row can add up to 0.999 or 1.001.Toy

No gradient reaches the teacher: P_t is held fixed (stop-gradient), and the centre is a running average that the gradient never trains. The teacher's distribution still steers learning, because the gradient of the loss on the student's score for prototype i is (P_s⁽ⁱ⁾ − P_t⁽ⁱ⁾) / τ_s (3.5 derives it).Fact

Take a student that gives A a flat answer, 1/3 ≈ 0.333 on each prototype, and the centred, sharp target, whose probabilities are the exponentials above divided by their sum: 1.000/7.674, 0.153/7.674 and 6.521/7.674, that is 0.130, 0.020 and 0.850. Then (1/3 − 1.000/7.674) / 0.1 = 2.03, (1/3 − 0.153/7.674) / 0.1 = 3.13 and (1/3 − 6.521/7.674) / 0.1 = −5.16. The three add up to 0, as these pushes always do, because both distributions add up to one; working from the rounded 0.333 and 0.850 instead would move the last digit of the third. A step against the gradient lowers the student's scores for prototypes 1 and 2 and raises its score for prototype 3; the sharper the target, the larger the push towards its winner.Toy

In DINO's collapse study, the teacher's entropy goes to 0 without centring and to its largest value without sharpening, and the KL divergence to the student goes to 0 in both; with both operations the entropy stays in between.Fact

Our runs log the same diagnostic: the entropy of the teacher's centred and sharpened distribution over 65,536 prototypes on the two global crops.Fact

Run 2 started from DINO's backbone with fresh heads: its teacher entropy was 9.8944 nats at step 1 and 10.6397 at step 463, not far below ln 65,536 = 11.0904, the entropy of a flat answer, then fell to 6.4316 at step 4,620 and 5.5077 at step 9,240. Run 1, which started from the released SelfPatch checkpoint with trained heads, read 1.2665 at step 1 and 1.2173 at step 9,240. Neither run ended at either end that DINO's study ties to collapse: run 1's 1.2173 nats lies between ln 3 and ln 4, as spread out as a choice among three or four equally likely prototypes, and run 2's teacher moved away from the flat end as it trained. Lab 5E in 5.8 draws both curves across the run's 9,240 steps.Reading

Lab 3B puts a toy head with 12 prototypes on screen. Lower the teacher's temperature and watch its bars sharpen; switch centring off; choose the 'One wins' or the 'All equal' scores, the two forms of collapse; let the student take a few learning steps and watch its bars, and the gradient under them, change. The cross-entropy it prints is the loss that 3.5 works out by hand.Toy

Temperatures, centring and collapse

What do the two temperatures and the centre do to the teacher's target and to the student's gradient?

Toy

What you are looking at: the teacher's probabilities over 12 prototypes and the student's, from made-up starting scores, the gradient of the cross-entropy on each of the student's scores, a slider for each temperature, a centring switch, three sets of scores (usual, one prototype winning everywhere, every prototype equal), buttons that let the student learn or update the centre, and the cross-entropy and its two parts in nats.Toy

Real model versus this toy: the real head has 65,536 prototypes and computes its scores from the picture; this toy starts from 12 made-up scores, and its learning step moves the student's scores directly, where a real step changes the network's weights. The real centre is the mean over a whole batch of pictures; here the batch is this one picture, so with the 'One wins' scores centring takes the lead off completely and leaves a flat teacher. The starting temperatures, 0.04 for the teacher and 0.1 for the student, and the centre's rate, 0.9, are the real ones.Fact

Lab 3E runs a whole training in miniature: a student and a teacher, each a tiny network, on four pictures and three prototypes. Each picture's answer is a point of a triangle whose corners are the three prototypes, so you can watch the answers move as training runs. Press play, then switch centring or sharpening off and play again.Toy

Collapse, and how DINO avoids it

What happens to the answers when centring or sharpening is switched off?

Toy

What you are looking at: a film of 300 training steps of a toy student and teacher on four pictures, A to D: each picture's answer as a point of the triangle of three prototypes (blue dots for the teacher, green rings for the student, faint lines for the teacher's path), and the teacher's entropy and its KL divergence to the student over the steps, the two numbers of DINO's collapse study.Toy

Real model versus this toy: the toy keeps DINO's rules (the teacher a moving average of the student, the centre a moving average of the teacher's scores with rate 0.9, temperatures 0.04 and 0.1, two noisy views per picture, the cross-entropy, stop-gradient) and shrinks everything else: four pictures of two numbers each, a network of nine weights, three prototypes, and a teacher that moves at 0.98 per step, far faster than the real 0.996 to 1. Its outcome also depends on its random start: with centring off it collapses on most starts, not on every one. It shows the mechanism; it measures nothing about the real model.Toy

Decision

Subtract a centre c, a moving average of the teacher's scores over batches with rate m = 0.9, from the teacher's scores before its softmax.

Why

Centring prevents one dimension from dominating, while it encourages a uniform output. It depends only on first-order batch statistics and acts like a bias term added to the teacher, and because the centre moves as a moving average, DINO works across batch sizes.Fact

Evidence

In the collapse study, without centring the teacher's entropy falls to 0 while its KL divergence to the student falls to 0: a collapse. The centre's rate matters: 0, 0.9 and 0.99 give 69.1, 69.7 and 69.4 % k-NN accuracy, while 0.999, too slow, collapses to 0.1 %.Fact

What it does to training

A prototype that scores high for every picture raises the centre by the same amount and loses its lead, so the targets differ from picture to picture and the student has to look at each picture to match them.Toy

If you changed it

Switch centring off in Lab 3E: at the lab's default start the four pictures' answers run into one corner of the triangle, the first form of collapse.Toy

Decision

Give the teacher a low temperature, τ_t = 0.04 rising to 0.07 over the first 30 epochs in DINO (0.04 for the whole run in SelfPatch's code and our runs), against τ_s = 0.1 for the student.

Why

Sharpening has the opposite effect to centring: it pushes towards one dominant dimension. Applying both operations balances their effects, which the authors find sufficient to avoid collapse in the presence of a momentum teacher.Fact

Evidence

k-NN accuracy for a teacher temperature of 0: 43.9 %; 0.02: 66.7 %; 0.04: 69.6 %; 0.06: 68.7 %; 0.08: 0.1 %, a collapse; 0.04 rising to 0.07: 69.7 %. Above 0.06, training collapses unless the temperature starts lower and warms up.Fact

What it does to training

A target sharper than the student's own answer asks the student to commit: the gradient (P_s − P_t) / τ_s pushes hardest on the prototypes the teacher favours, as the walk of 3.4 shows.Toy

If you changed it

Without sharpening, the teacher's entropy rises to its largest value, −log(1/K), and its KL divergence to the student goes to 0: the second form of collapse. In Lab 3E the answers drift to the middle of the triangle.Fact

3.5 The loss over the pairs of crops

Assumes: 3.4, a logarithm and a gradient (Part 1). Gives: the cross-entropy, worked with five prototypes; the 18 pairs of one image; and what one step of the gradient does to the student.

Now the two networks can be compared. For one pair of crops the teacher gives a probability distribution over the prototypes, and so does the student. DINO trains the student to match the teacher by minimising the cross-entropy H(teacher, student) = −Σ teacher × log(student) with respect to the student's weights.Fact

The teacher's probabilities are held fixed while the gradient is computed, a step called stop-gradient, so only the student learns from the loss.Fact

One image gives several pairs. The teacher sees 2 crops and the student 10, and a crop is never compared with itself, so the loss averages 2 × 9 = 18 pairs; with the 10 local crops of the released DINO run the same rule gives 2 × 11 = 22 pairs. DINO's Eq. (3) writes it as a sum over each global view x seen by the teacher and every view x′ ≠ x seen by the student.Fact

The cross-entropy over the pairs of crops

H(a, b) = −Σ_i=1..K a⁽ⁱ⁾ log b⁽ⁱ⁾

min over θ_s of Σ_{x ∈ {x₁^g, x₂^g}} Σ_{x′ ∈ V, x′ ≠ x} H(P_t(x), P_s(x′))

DINO's Eq. (2) and Eq. (3); our trainer averages the 18 terms of one image.Fact

H(a, b): the cross-entropy of two distributions over the same K prototypes: one scalar, in nats when the logarithm is the natural one
a⁽ⁱ⁾, b⁽ⁱ⁾: the teacher's and the student's probabilities for prototype i
x₁^g, x₂^g: the two global crops, 224 × 224 pixels each, the only crops the teacher sees
V: every view of the image: 10 crops in our runs, 12 in the released DINO runFact
P_t(x), P_s(x′): the teacher's centred and sharpened distribution for x, held fixed, and the student's distribution for x′; vectors of 65,536 probabilities in DINO
θ_s: the student's weights, the only ones the gradient moves

Five prototypes, where the real head has 65,536. The teacher's scores are 0.20, 0.15, 0.10, 0.05 and 0.00, divided by τ_t = 0.04; the student's are 0.15, 0.20, 0.05, 0.00 and −0.05, divided by τ_s = 0.1. The centre is taken as zero, logarithms are natural, and every value is rounded to three decimals after the arithmetic, which keeps every digit: redoing a cell from the rounded numbers can move its last digit. In Lab 3D this is the pair of teacher G1 and student G2.Toy

Prototype	1	2	3	4	5	Sum
teacher score	0.20	0.15	0.10	0.05	0.00
teacher probability, τ_t = 0.04	0.715	0.205	0.059	0.017	0.005
student score	0.15	0.20	0.05	0.00	−0.05
student probability, τ_s = 0.1	0.296	0.489	0.109	0.066	0.040
log of the student probability	−1.216	−0.716	−2.216	−2.716	−3.216
term: −(teacher probability) × log	0.870	0.147	0.130	0.046	0.015	H = 1.208

Each term is −(teacher probability) × log(student probability), and H is their sum, 1.208 nats. Most of it comes from prototype 1, which the teacher gives 0.715 and the student only 0.296; the student currently favours prototype 2. H can never fall below the teacher's own entropy, here 0.825, and reaches it only when the student's probabilities equal the teacher's; the difference, 0.382, is the KL divergence.Toy

The image's loss averages 18 such numbers, one per pair. Lab 3D shows all of them at once and works any pair out when you click it.Toy

The gradient of H on the student's score s⁽ⁱ⁾ for prototype i is ∂H / ∂s⁽ⁱ⁾ = (P_s⁽ⁱ⁾ − P_t⁽ⁱ⁾) / τ_s: the student's probability minus the teacher's, divided by the student's temperature. Here it is −4.19, 2.84, 0.50, 0.49 and 0.35. A step against the gradient raises the score of prototype 1, where the student gives too little, and lowers the others, prototype 2's most of all.Toy

One step with a learning rate of 0.01 moves the scores to 0.192, 0.172, 0.045, −0.005 and −0.054; the student's probabilities become 0.440, 0.359, 0.101, 0.061 and 0.038, and H falls from 1.208 to 0.994 nats. The student cannot set its scores directly: backpropagation carries these pushes back through the head and the backbone and gives every weight its share of the gradient, and the optimiser then moves each weight a little.Toy

The SelfPatch paper writes this distance as a KL divergence, which is H minus the teacher's entropy; the teacher's entropy has no gradient, so the two losses have the same gradients and train the student identically.Fact

In our runs this term is DINO's global term as the trainer computes it: 65,536 prototypes, each of the two global teacher views centred and sharpened at 0.04 and compared with every other student view at 0.1, 18 terms over the ten views.Fact

The runs' telemetry follows it step by step. Run 1, from the released SelfPatch checkpoint, read 4.2292 at step 1, 3.7014 at step 463, 3.2344 at step 4,620 and 2.8194 at step 9,240. Run 2, whose heads started fresh on DINO's backbone, began at 10.5406, not far below ln 65,536 = 11.0904, the cross-entropy between two flat answers, read 10.8685 at step 463, then fell to 7.3341 and 6.5651. Lab 5E in 5.8 draws both runs' curves across all 9,240 steps.Real

Lab 3D lays out the pairs of one image as a grid: the teacher's two global crops down the side, the student's crops across the top. Every cell holds one pair's cross-entropy; click one to see it worked out over five prototypes, then let the student learn a step and watch the numbers fall.Toy

The 18 pairs

Which student view learns from which teacher view, and how much does each pair ask?

A crop is never compared with itself, so every pair asks two different views to agree, and 16 of the 18 pairs ask a local crop for a global crop's answer.Toy

If you changed it

A teacher that also saw local crops would add targets made from small views. SelfPatch's teacher does process all ten crops, for its patch targets, while its image-level targets still come from the two global crops.Fact

3.6 The teacher is an average of the student's past

Assumes: 3.1 and 3.5. Gives: the moving average, its cosine schedule, and how much of the starting network the teacher still holds after an epoch.

After every step, each of the teacher's weights moves a small step towards the matching weight of the student: teacher ← m × teacher + (1 − m) × student. The number m is the momentum. In DINO it follows a cosine schedule from 0.996 up to 1 during training; our runs go from 0.9995 up to 1. SelfPatch's official default is 0.996, and the note it prints about that setting recommends 0.9995 for a batch of 256.Fact

Roughly, the teacher averages the last 1 / (1 − m) steps: 2,000 steps at m = 0.9995 and 250 at m = 0.996.Fact

The teacher's moving average and its cosine schedule

θ_t ← λ θ_t + (1 − λ) θ_s

λ(k) = 1 − (1 − λ₀) · (cos(π k / n) + 1) / 2

DINO's update rule with λ on a cosine schedule from λ₀ to 1; our trainer computes the same schedule from 0.9995.Fact

θ_t, θ_s: every weight of the teacher and the matching weight of the student; the backbone alone has 21,664,128 of themFact
λ: the momentum, a scalar just below 1; the text above and Lab 3C call it m
λ₀: its value at the start: 0.996 in DINO, 0.9995 in our runs
k, n: the current step and the run's number of steps: 9,240 in our runsFact

One weight: the teacher holds 0.500 and the student 0.800. At λ = 0.996 the teacher becomes 0.996 × 0.500 + 0.004 × 0.800 = 0.4980 + 0.0032 = 0.5012: it closed 0.4 % of the gap. At λ = 0.9995 it becomes 0.50015, a twentieth of a per cent of the gap. With the student held at 0.800 for 100 steps, the gap shrinks to 0.996¹⁰⁰ = 0.670 of its start at λ = 0.996, and to 0.9995¹⁰⁰ = 0.951 at λ = 0.9995.Toy

The schedule of our runs: λ(0) = 0.9995; halfway, at step 4,620, cos(π / 2) = 0, so λ = 1 − 0.0005 × 1/2 = 0.99975; at step 9,240, cos(π) = −1, so λ = 1.Toy

Follow the share of the step-0 teacher, the network that training started from. Each step keeps λ of that share. At a constant 0.9995, the 462 steps of one epoch keep 0.9995⁴⁶² ≈ 0.794. Under our cosine schedule the teacher keeps 0.794 of its step-0 weights after one epoch, 0.334 after five, 0.151 after ten and 0.099 at the end of a 20-epoch run. At DINO's settings the teacher moves on much faster: one ImageNet epoch at a batch of 1,024 is 1,251 steps, and 0.996^1,251 ≈ 0.0066.Fact

The teacher receives no gradient: the moving average is its only update. Each step moves every teacher weight by (1 − λ)(θ_s − θ_t), a fixed fraction of its distance to the student's weight, so the teacher follows the student's path smoothed over roughly the last 1 / (1 − λ) steps. The student moves by the gradient; the teacher moves by the student.Fact

Our runs' telemetry shows the schedule at work: λ = 0.9995 at step 1, 0.999503 at step 463, 0.99975 at step 4,620 and 1.0 at step 9,240, as the formula gives. Lab 5E in 5.8 draws the whole curve.Real

Through the first epoch, then, the teacher, and with it every target the student gets, is mostly the network that training started from.Fact

Lab 3C draws the share of the step-0 teacher step by step. Press 'DINO setting' and 'Our setting' in turn and compare how fast the share falls, then switch on our cosine schedule to see the slower real curve.Toy

A teacher made of the student's past

How much of the starting teacher is left after many small averaging steps, and so how long do the student's targets still come from the starting network?

Toy

What you are looking at: the share of the step-0 teacher still inside the teacher after each step, at the momentum on the slider; at rest m = 0.9995 over the 462 steps of one epoch of our runs, which ends at 0.794. Below it, one made-up weight: the green student weight wanders as each batch pushes it, and the blue teacher weight follows the average of that path, slowly and smoothly.Fact

Real model versus this toy: the upper curve is the real arithmetic of the averaging rule with m held fixed; in our runs m climbs along a cosine towards one, as the dashed curve shows. The weight in the lower chart is invented; a real weight moves wherever the loss pushes it.Toy

Decision

Build the teacher as an exponential moving average of the student's weights, θ_t ← λ θ_t + (1 − λ) θ_s, after every step.

Why

The authors find this momentum encoder particularly well suited: it performs a form of model ensembling similar to Polyak-Ruppert averaging, and the teacher performs better than the student throughout training, so it guides the student with targets of higher quality.Fact

Evidence

The supplement compares teachers: a copy of the student 0.1 %, the student of the previous iteration 0.1 %, the student of the previous epoch 66.6 % and the momentum teacher 72.8 % k-NN accuracy.Fact

What it does to training

The targets change slowly: at λ = 0.9995 the teacher is roughly an average of the last 2,000 steps, so one unusual batch cannot drag the targets around.Fact

If you changed it

Copying the student fails to converge, while freezing the teacher for an epoch works surprisingly well. Compare DINO's 0.996 with our 0.9995 in Lab 3C.Fact

Decision

Raise the momentum λ along a cosine from 0.996 to 1 during training (0.9995 to 1 in our runs).

Why

DINO states the cosine schedule from 0.996 to 1. SelfPatch's official code keeps 0.996 as its default, and its help text recommends 0.9995 for a batch of 256, the batch of our runs.Fact

Evidence

Our runs' telemetry: λ = 0.9995 at step 1, 0.999503 at step 463, 0.99975 at step 4,620 and 1.0 at step 9,240.Real

What it does to training

The teacher moves most at the start, where the momentum is lowest, and less and less as it climbs to 1, so the last steps train the student against an almost fixed target. Under our schedule the teacher still holds 0.794 of its step-0 weights after one epoch and 0.099 after twenty.Fact

If you changed it

A constant 0.9995 would leave 0.0992 of the step-0 teacher after 4,620 steps, where the cosine schedule leaves 0.151. Lab 3C draws both curves.Toy

3.7 What DINO learns

Assumes: 3.1 to 3.6 and attention (Part 2). Gives: what the trained network's attention shows, and why a decoder that never sees a label can read objects out of its keys.

What does all this produce? The DINO authors report that the features of self-supervised ViTs contain explicit information about the segmentation of an image, which pixels belong to which object, and that this information does not emerge as clearly with supervised ViTs or with convolutional networks.Fact

They checked this against outlines drawn by people. Masks made by keeping 60 % of a ViT-S/16's attention mass overlap the object masks of the PASCAL VOC validation pictures with a Jaccard similarity of 45.9 for DINO, 27.3 for a supervised ViT and 22.0 for random weights, a measurement made with human masks after training.Fact

Labels also measure the features with the yardstick of 3.1: on ImageNet, a k-NN classifier fitted with labels on DINO's ViT-S/16 features names the right class first for 74.5 % of pictures, and a linear classifier for 77.0 %, both on frozen features that training no longer changes.Fact

Two object finders that work without labels, MaskCut and TokenCut, read one particular part of the last block: the keys of its attention, which they use as patch features. The thesis's label-free decoder does the same: at 480 pixels it takes 900 × 384 numbers from the teacher's last block, one key of 384 numbers for each of the 900 patches, subtracts the picture's mean key from every key, and links two patches when the cosine of their keys is above 0.2, giving every other pair a tiny weight instead (Part 6).Fact

The idea fits in three moves, which the figure draws. Link two patches when their keys are similar. If the patches of an object have similar keys and differ from the background, the links form a dense island over the object. A cut then splits the patches into two groups so that few links cross between the groups, and one group becomes the object's mask. Part 6 builds the decoder in full and runs it on real keys.Toy

From keys to a mask, schematic. An idealised six-by-six grid, drawn by hand and computed from no network, in which the patches of one object have similar keys. Left, each patch's key drawn as a direction; middle, links between neighbouring patches whose keys are similar; right, the group that a cut separates, which becomes the mask.Toy

Decision

Read the scene's objects from what DINO's training leaves in the last block, its attention and its keys, with no label.

Why

The authors report that self-supervised ViT features contain explicit information about the segmentation of an image, which does not emerge as clearly with supervised ViTs or with convolutional networks.Fact

Evidence

Masks that keep 60 % of a ViT-S/16's attention mass overlap the object masks of the PASCAL VOC validation pictures with a Jaccard similarity of 45.9 for DINO, 27.3 for a supervised ViT and 22.0 for random weights, measured with human masks after training.Fact

What it does to training

Nothing in the loss names an object. One reading, the course's own: to give one answer for a small crop and a large crop of the same picture, the network has to find what the crops share, and that is usually an object.Toy

If you changed it

The supplement measures the overlap for other self-supervised training: 45.1 for DINO without multi-crop, 46.3 for MoCo-v2, 47.8 for BYOL and 46.8 for SwAV, against 27.3 for supervised training, and the authors observe the scene layout and object boundaries across these methods. Its masks keep 80 % of the attention mass, where the main text says 60 % beside the same 22.0, 27.3 and 45.9: the paper's two texts disagree on that share.Fact

3.8 The recipe in numbers

Assumes: 3.1 to 3.6, gradient descent and a learning rate (Part 1). Gives: every setting of the released DINO ViT-S/16 and what each one is for.

A few more terms first. The learning rate sets the size of each step. A warm-up raises it gradually at the start of training. Weight decay pulls every weight a little towards zero at each step; a bigger number means a stronger pull. AdamW is the optimiser: it scales each weight's step by the recent history of that weight's gradients and applies the weight decay as a separate pull (5.7 works one AdamW step with numbers). Stochastic depth lets the student skip a block's residual branches at random. Gradient clipping would shrink any gradient that grows past a set size.Toy

Setting	Released DINO ViT-S/16	What it is for	Source
Training pictures	ImageNet: 1,281,167 photographs, about 1.28 million, used with no labels	the pictures the two networks learn to agree about	Fact
Epochs	800	longer training raised k-NN accuracy from 70.9 % (100 epochs) to 74.5 % (800)	Fact
Batch	1,024 images per step, 64 on each of 16 GPUs	how many pictures one step averages; the centre averages the teacher over it	Fact
Steps per epoch	1,251	1,281,167 pictures in batches of 1,024, the last partial batch dropped	Fact
Optimiser	AdamW	steps sized for each weight from its recent gradients	Fact
Learning-rate rule	0.0005 × batch / 256	a larger batch takes a proportionally larger step	Fact
Peak learning rate	0.002, after a linear warm-up of 10 epochs	small steps while the network and the centre settle, then the full step	Fact
Final learning rate	1e-5, after a cosine decay	small steps at the end, so the weights settle	Fact
Weight decay	0.04 → 0.4, along a cosine	a stronger pull towards zero as training goes on	Fact
Teacher momentum	0.996 → 1, along a cosine	a teacher that follows the student slowly, then not at all (3.6)	Fact
Teacher temperature	0.04 → 0.07 over 30 epochs	sharp targets (3.4)	Fact
Student temperature	0.1	a softer student than teacher	Fact
Centre rate	0.9	a centre that follows the teacher's batch averages (3.4)	Fact
Prototypes	65,536	many soft categories (3.3)	Fact
Crops	2 global crops of 224 pixels covering 0.25 to 1.0 of the area, and 10 local crops of 96 pixels covering 0.05 to 0.25	small views that answer for large ones (3.2)	Fact
Stochastic depth	0.1	noise in the student alone	Fact
Gradient clipping	none: its setting is 0	no limit on the size of a gradient	Fact
Last layer of the head	held fixed for 1 epoch at the start	DINO's code notes that this typically helps training	Fact

A worked example of the learning-rate rule at a batch of 1,024: 0.0005 × 1,024 / 256 = 0.002, and the released run's log reaches that peak at epoch 10, when the warm-up ends.Fact

A student that may lose a block must still give the teacher's answer, so no single block carries the answer alone, while the teacher computes its targets with the whole network.Toy

If you changed it

DINO's paper and supplement, as read for this course, report no study of it, and nothing later in this course depends on it.Toy

3.9 Real similarity under three models

Assumes: cosine similarity (Part 1), tokens (Part 2) and 3.7. Gives: what one patch of a real photograph finds similar under DINO, under the released SelfPatch checkpoint and under run 2 of this thesis.

Everything so far ran on toys. This lab runs on the real models. For each of the course's eight photographs, the bundle holds the final block's 196 patch tokens of its 224-pixel view, 384 numbers each, under five models, and this lab reads three of them: the converted DINO ViT-S/16 (run 2's start: DINO's backbone in the SelfPatch network, without its class token and final LayerNorm), the released DINO+SelfPatch checkpoint, and the teacher of run 2 at the end of its 20 epochs.Real

Click a patch and the lab computes, under each model, the cosine between that patch's token and every other token of the view, and paints it over the photograph on one shared colour scale: the deeper the blue, the more similar. The anchor wears a thick outline, and its eight most similar patches wear orange ones. Eight is the k of our rule on a 224-pixel crop (Part 5), so on this view the orange patches under run 2 are the anchor's top-k positives as the rule picks them, up to the rounding of the stored numbers.Fact

The lab opens on the photograph of a dog and a cat by a door, at patch 34, a patch on the dog's face in row 2 and column 6 of the 14 × 14 grid. Under run 2's teacher its most similar token has a cosine of 0.8137 with it, computed from the float32 tokens.Real

Things to try: click the cat, then the floor, then the door; compare how far the blue spreads under each model, and whether the orange patches stay on the same object. Choose another photograph with the picker.Toy

Real similarity

What does one patch of a real photograph find similar under DINO, the released SelfPatch checkpoint and run 2?

Real Toy

What you are looking at: three cosine maps of one anchor patch over a real photograph's 224-pixel view, one per model, on one colour scale; the anchor outlined thickly, its eight most similar patches in orange; under each map the eight patches with their cosines, their mean and the least similar patch; beside them how many of the eight the models share.Real

Real model versus this toy: the tokens and so the cosines are the real models' numbers, stored as 8-bit integers that change no cosine between two tokens by more than 0.00621. The photograph under the maps is redrawn by your browser from the same file and the same square with its own resampling filter, where the bundle used PIL's bicubic one, so its pixels need not equal the ones the models saw; the maps do not depend on it.Real

Glossary of this page

AdamW: The optimiser DINO uses. It scales each weight's step by the recent history of that weight's gradients, and it applies weight decay as a separate small pull towards zero.
Augmentation: A random change applied to a training crop, such as a mirror flip, a colour change or a blur, so that two views of one image differ.
Backpropagation: The procedure that computes the gradient for every weight by passing the effect of the loss backwards through the network, layer by layer.
Batch: The pictures used together in one step of training.
Bottleneck: The narrow layer of 256 numbers at the end of the projection head's MLP, just before the L2 normalisation.
Centre: In DINO, a running average of the teacher's scores, one number per prototype, updated after every step.
Centring: Subtracting the centre from the teacher's scores before the softmax, so that no prototype can win for every picture.
Collapse: A useless outcome of training in which the networks give every picture the same answer: one prototype takes all the probability, or the probability spreads evenly over all of them.
Colour jitter: An augmentation that changes a crop's brightness, contrast, saturation and hue by random amounts.
Cosine schedule: A setting that moves from a start value to an end value along half a cosine wave: slowly at first, fastest in the middle and slowly at the end.
Crop: A rectangle cut out of a picture and resized. In training the networks receive crops of the photographs.
Cross-entropy: DINO's loss for one pair of crops: minus the sum, over the prototypes, of the teacher's probability times the logarithm of the student's probability. It is smallest when the student's probabilities equal the teacher's.
Distillation: Training a student network to reproduce a teacher network's outputs.
Entropy: A measure of how spread out a probability distribution is: zero when one outcome takes all the probability, largest, the logarithm of the number of outcomes, when all of them are equally likely.
Epoch: One pass over all the training pictures.
Exponential moving average (EMA): A running average in which each new value gets a small weight and older values fade away step by step. DINO's teacher is an exponential moving average of the student's weights.
Gaussian blur: An augmentation that softens a crop by averaging each pixel with its neighbours, the nearer ones weighing more.
Global crop: One of the two large crops taken from each image at every step. In SelfPatch's code and our runs it covers 0.25 to 1.0 of the image's area and is resized to 224 × 224 pixels.
Gradient: For every weight, how the loss would change if that weight moved a little. Training moves each weight in the direction that lowers the loss.
Gradient clipping: Shrinking any gradient that grows past a set size, so that one step cannot jump too far.
InfoNCE: A contrastive loss: a view must pick out its partner among negatives, scored by the share that their similarity takes in a softmax over the partner and the negatives. MoCo-v2 uses it between images; our rule uses it between the patches of one crop (Part 5).
Jaccard similarity: The area two masks share divided by the area they cover together: one for identical masks and zero for masks that do not overlap.
k-NN classifier: A simple test of features with labels: a picture gets the label that wins a vote among the k labelled pictures whose features lie closest to its own. DINO weights the vote and lets the 20 nearest take part.
Key: In attention, the vector a token offers to be matched against the other tokens' queries. MaskCut, TokenCut and the thesis's decoder read the last block's keys as patch features.
KL divergence: Cross-entropy minus the teacher's entropy: a distance between two probability distributions that is zero when they are equal.
L2 normalisation: Dividing a vector by its length so that its length becomes one; the vector then lies on the surface of a sphere.
Label-free decoder: A fixed procedure that turns a network's features into object masks without using any labels. The thesis's decoder reads the teacher's last-block keys.
Learning rate: The size of each step of training, set by a schedule over the run.
Linear classifier: A single linear map from features to class scores, fitted with labels: a common test of what the features already contain.
Local crop: One of the small crops taken from each image at every step, 8 of them in SelfPatch's code and our runs. Each covers 0.05 to 0.25 of the image's area and is resized to 96 × 96 pixels.
MaskCut: A label-free object finder that links patches with similar keys and splits the resulting graph with normalised cuts, one object at a time.
Momentum: The number m in the teacher's update, just below one, which sets how slowly the teacher follows the student. The centre's rate is the same kind of number: the larger it is, the slower the centre moves.
Multi-crop: DINO's way of making views: a few large global crops and several small local crops of the same image.
Nat: The unit of entropy and cross-entropy when the logarithm is the natural one.
Probability distribution: A list of numbers, one per possible outcome, none negative and all adding up to one.
Projection head: The small network after the backbone that turns a crop's summary into scores over the prototypes. It exists for training; later tasks read the backbone.
Prototype: One of the learned directions that the projection head compares its output with; DINO's head has 65,536 of them.
Segmentation: Dividing a picture into regions, for example one region per object.
Self-distillation: Distillation in which the teacher is built from the student itself: in DINO, as a moving average of the student's past weights.
Sharpening: Using a low teacher temperature so that the teacher's probabilities clearly favour a few prototypes.
Solarise: An augmentation that keeps every pixel value below 128 and turns every value v of 128 or more into 255 − v, so the bright parts turn dark.
Step: One round of training: compute the loss on a batch, compute the gradient, and move the weights one small step. Also called an update.
Stochastic depth: A training trick in which the student randomly skips a block's residual branches, its attention and MLP steps, for some pictures.
Stop-gradient: Holding a quantity fixed while the gradient is computed, so that no learning flows through it; DINO applies it to the teacher's outputs.
Student: The network that learns from the loss.
Supervised learning: Training that uses labels to say what each output should be.
Teacher: The network whose outputs the student learns to match. In DINO it is an average of the student's past weights and receives no gradient.
Temperature: A number τ that divides the scores before a softmax: a smaller τ gives a sharper distribution, a larger τ a flatter one.
View: One changed version of a training image: a crop with its augmentations.
Warm-up: A period at the start of training in which the learning rate rises gradually to its peak.
Weight decay: A small pull of every weight towards zero at each step, which keeps weights from growing large. A bigger weight-decay number means a stronger pull.