SelfPatch

4.1 The gap DINO leaves

Assumes: patch tokens and the class token (Part 2), and DINO's loss over pairs of crops (Part 3). Gives: why SelfPatch adds a loss for every patch, and the evidence its authors give.

When a 224-pixel crop passes through the backbone, 196 patch tokens come out, one per 16 by 16 square of the crop. DINO's loss reads none of them directly. Its projection head is attached to one extra token, the class token, whose job is to gather information from the whole sequence. Fact

The SelfPatch authors point out what this leaves out: a Vision Transformer also produces an output for every patch, yet image-level methods such as DINO train only the class token's output. They argue that, as a result, such networks may fail to capture meaningful relations among patches, and that a positive pair made of two augmented views of one image is architecture-agnostic: it would train any network the same way and does not fully use the patch outputs that a Transformer has. Fact

Think about what reaches a patch token during DINO's training. The loss produces a gradient for the class token's output only. The patch tokens that leave the last block receive no gradient at all. The patch tokens inside the network receive some, through the attention that the class token pays to them in the blocks above, but only as much as the class token happens to look at them, and only towards whatever helps the summary of the whole crop. Two patches of the same dog get no direct instruction to resemble each other. Toy

What evidence do the authors give that the gap matters? Three kinds. A picture: in their Fig. 1 they take some patches of a ViT-S/16 trained with DINO and, for each one, keep the 10 % of patches that it attends to most in the last block. Those patches are not semantically related to the patch that asks, which their introduction calls collapsed self-attention maps; with DINO + SelfPatch the selections improve, which they take as evidence of better patch-level representations. Scores: their method improves DINO on three dense tasks, each measured with labels after pre-training (4.7). And an ablation, an experiment that changes one part at a time: they switch the patch loss off while keeping everything else, the new aggregation module of 4.3 included. With the patch loss's weight λ = 0 the DAVIS score is 55.3, about DINO's 55.1, and with λ = 0.1 it is 57.0, so the gain comes from the patch loss itself. Every ablation of the paper pre-trains a small ViT-Ti/16, a smaller Vision Transformer than ViT-S/16 with the same 16 by 16 patches, on COCO and scores DAVIS 2017 video segmentation by label propagation, which uses labelled masks. Fact

Why should this matter to a study that finds objects without labels? The decoder of Part 6 finds objects by comparing the features of patches with one another, so it depends on exactly the patch-level outputs that DINO trains only through the class token. SelfPatch gives every patch a lesson of its own; our study, in Part 5, asks how that lesson should choose the patches it learns from. Toy

4.2 Neighbours as positives

Assumes: cosine similarity (Part 1), the 14 by 14 grid of patches (Part 2), and the teacher (Part 3). Gives: the window, the cosine, the top k, the wrap at the border, and the positive set P(i).

SelfPatch's idea fits in one sentence: similar neighbouring patches are treated as positive examples, because adjacent patches often share a common semantic context. A patch of fur usually sits next to more fur, and a patch of door next to more door. So the lesson for each patch is: your token should resemble the tokens of the neighbours that look like you. Fact

Three words carry the method. The anchor is the patch whose lesson is being built; every patch takes a turn as the anchor. Its candidates are the 8 patches around it, the other cells of the 3 by 3 window centred on it. Its positives are the candidates that the rule keeps, and the anchor's lesson is built from them only. Fact

How similar is similar? SelfPatch measures it by the cosine of two patches' tokens: 1 when their 384 numbers point the same way, 0 when they are unrelated. The paper's Eq. (3) writes the cosine on the tokens of the online network, the paper's name for the student; the official code, which our trainer runs, computes it on the teacher's tokens, without gradient, inside the teacher's forward pass. The teacher is the slow average of the student from Part 3, so the choice of positives changes slowly from step to step. Fact

The official rule then keeps each anchor's 4 most similar candidates as its positives, so k = 4. Any selection rule, this one included, can be written in one form: it keeps the candidates whose similarity reaches a threshold θ (the Greek letter theta), P(i) = {j ∈ N(i) : s_ij ≥ θ_i}. The official rule sets θ_i to the 4th largest of the anchor's own 8 similarities, so it keeps exactly 4. Our earlier selector study, told in Part 5, changed only how θ is set. Fact

What about an anchor on the border of the grid? It has fewer than 8 real neighbours. The official code answers with a wrap-around: the window continues on the opposite edge, as if the grid were drawn on a doughnut, so a patch on the left border counts patches on the right border as neighbours and every anchor has exactly 8 candidates. The paper speaks of the 8 adjacent patches and does not mention the wrap; it comes from the code. Fact

On the 14 by 14 grid of a global crop, the top-left anchor 0 has the candidates 1, 13, 14, 15, 27, 182, 183 and 195, while an inner anchor such as 15 has the ordinary ring 0, 1, 2, 14, 16, 28, 29 and 30. How often does the wrap matter? A 224-pixel crop has 196 × 8 = 1,568 anchor-to-candidate links, and 164 of them wrap: 10.5 %. A 96-pixel crop has 36 × 8 = 288 links, and 68 of them wrap: 23.6 %, because a larger share of a small grid's cells lies on its border. On any G by G grid the count is 12G − 4: each of the 4 corner anchors has 5 wrapped candidates and each of the 4(G − 2) other border anchors has 3, and 20 + 12G − 24 = 12G − 4. Fact

4.3 The aggregation head

Assumes: attention with queries, keys and values, LayerNorm and the MLP (Part 2), and softmax (Part 1). Gives: how 4 positives become one target, the query token's share, and why a learned summary.

An anchor now has 4 positives: 4 tokens of 384 numbers each. The student will be asked to predict one target per anchor, so the 4 tokens must become one. A plain average would be the simplest merge. SelfPatch uses a small learned network instead, the aggregation module, which summarises a set of tokens into one token. Fact

Recall the attention of Part 2: every token asks a question with its query, and every token answers with its key and value. Class attention, a design the paper borrows from Touvron et al.'s CaiT, keeps the answering and drops most of the asking: only one token asks. That token is the query token, 384 numbers that the network learns, drawn at the start from a truncated normal distribution with standard deviation 0.02: small random numbers around 0, from a bell curve whose far tails are cut off. It attends to itself and to the set it summarises, and its answer is the summary. Fact

The module stacks 2 class-attention blocks, without LayerScale, a further idea of the same CaiT paper that the supplement says SelfPatch leaves out; the official code's block carries the name but holds no such parameters. Each is a Transformer block in which only the query token is updated: attention with 6 heads, whose result is added back onto the query token, then an MLP four times as wide as the token, added back the same way. In the first block the learned query token asks; in the second, the first block's summary asks, and the positives answer again. A final LayerNorm rescales the result. The module has 3,550,080 parameters. Fact

Three details matter later. The target is built from the positives only, with the anchor's own token left out, so an anchor never learns from itself. The order of the positives does not matter: the module adds no position embedding of its own, although the tokens still carry their positions from the backbone. And the same module, run on all of a crop's tokens, makes the crop's image-level token, which takes the place of DINO's class token, because SelfPatch's backbone has none. Fact

How much of a target comes from the query token itself? Each anchor's set is its 4 positives plus the query token, a sequence of 5. Attention turns the scores into shares that add up to 1, so if all 5 scores were equal each token, the query token included, would get 1/5 = 0.2. In general, at equal scores the query token keeps 1 / (d + 1) of the attention when it summarises d tokens: 0.5 for a single positive and 0.11 for all 8. Fact

The query token is the same for every anchor, so the fewer positives an anchor has, the more its target leans towards one shared vector. Under the official rule every anchor has 4; Part 5 tells of rules under which an anchor can have 1 or all 8, and this share is one reason that matters. Toy

In the second block the question comes from the first block's summary, so the module can give less say to a neighbour that disagrees with the rest of the set: the cleaning role the paper gives it. The walk below works this out with numbers. Fact

Who trains the module? The student runs the module's query token and its two blocks only to make the image-level token; it never runs them on neighbours. Of the whole module, only the final LayerNorm also touches the student's patch tokens: each patch's own token passes through it on the way to the patch head (4.4). So the student's query token and blocks, the part that weighs the positives, learn only through the image-level loss, while the final LayerNorm also feels the patch loss. The teacher, which builds every patch target with its own copy of the module, is an average of the student's. The part that weighs the positives learns from the whole-picture lesson, never from the patch lesson. Fact

4.4 The patch loss and the total

Assumes: the projection head and its prototypes, centring and sharpening, the cross-entropy and the stop-gradient (Part 3). Gives: the patch loss, the asymmetry between teacher and student, the total L = L_global + 0.1 · L_patch, and why each piece is there.

The teacher and the student treat an anchor differently, on purpose. The teacher builds the target from the anchor's positives, merged by the aggregation module. The student predicts that target from the anchor's own token, without the aggregation module. This difference is the asymmetry of SelfPatch: own token versus aggregated positives. Fact

Why not let both sides use the neighbours? Because pulling the tokens of neighbouring patches together invites collapse (Part 3): the easiest way to make every patch agree with its neighbours is to make every token the same, and a network that outputs one constant token has learned nothing. The authors found that the asymmetry, together with the stop-gradient, avoids the collapse that comes from pulling patch representations together. This course takes that finding from the authors; it does not show why it works. Fact

Two details come from the code. The student's prediction starts from LN(z_i): its own token first passes through the aggregation module's final LayerNorm, a step the paper's equation leaves out. And the student's tokens, like the teacher's, are the backbone's last outputs, 196 tokens of 384 numbers for a 224-pixel crop, without a class token and without a final LayerNorm of the backbone. Fact

A target and a prediction are each still 384 numbers. As in Part 3, a projection head turns such a vector into one score per prototype, and a softmax with a temperature turns the scores into a probability distribution. SelfPatch gives the patches a head of their own, the patch head, with the same layers as the image-level head, 384 → 2,048 → 2,048 → 256 → 4,096: it ends in 4,096 prototypes against the image-level head's 65,536. The teacher divides its scores by 0.04 and the student by 0.1, in both heads, and the patch head has its own centre, 4,096 numbers, one per prototype. Fact

The patch loss takes every anchor, compares the teacher's target distribution with the student's prediction by the cross-entropy of Part 3, and averages. Every one of the 10 crops gets patch targets, and each student patch is compared with the target built in the same crop. That is why SelfPatch's teacher processes all 10 crops, while DINO's teacher sees only its 2 global crops. The patch loss is the average of the 10 per-crop losses. Fact

The two losses are then added. The image-level loss L_global is DINO's loss of Part 3, now read from the image-level token and still averaged over 18 pairs of crops. The patch loss L_patch enters with a weight λ (the Greek letter lambda) of 0.1: L = L_global + 0.1 × L_patch. In words: the total loss is the image-level loss plus one tenth of the patch loss. The student's weights move to make this total smaller; the teacher follows by averaging. Fact

The paper writes the distance between the two distributions as a KL divergence, following DINO; the code uses the cross-entropy of Part 3. The two differ only by the teacher's entropy, which has no gradient, so they train the student the same way. Fact

4.5 The model in numbers

Assumes: parameters, tensors and shapes (Part 2), and the projection head (Part 3). Gives: what each part of SelfPatch costs, and how much work one optimiser step does.

Every learned number in a network is a parameter. The table counts them for the SelfPatch student as the training code builds it. Fact

Of the student's 54,128,384 parameters, 54,058,752 are trained by gradient descent. The other 69,632 stay frozen: they are the fixed prototype lengths of the two heads, 65,536 + 4,096. Most of the image-level head sits in its last layer, 16,842,752 parameters, a direction of 256 numbers and a length for each of its 65,536 prototypes; the patch head's last layer holds 1,052,672. The three parts after the backbone hold about 32.5 M parameters, which exist for training only. The teacher is an identical copy with the same 54,128,384 parameters, all changed only by averaging. Fact

Now count the work in one optimiser step. Our runs use a batch of 256 images per step. Each image gives 2 global crops of 196 anchors and 8 local crops of 36 anchors: 680 anchors per image. With 8 candidates each, that is 5,440 candidate links, and 3,136 of them, 57.6 %, come from the two 224-pixel crops. Fact

So one step handles 174,080 anchors and 1,392,640 candidate links, 256 × 680 × 8. Under the official rule each anchor ranks its own 8; the batch rule of our earlier selector study, told in Part 5, took one median over all 1,392,640 of them. Fact

Inside the computer these numbers travel in tensors: blocks of numbers arranged along several axes, with a shape written as sizes joined by ×. For example B × 196 × 384 means B images, 196 tokens per image and 384 numbers per token. The work is shared between several graphics processors, and B is the number of images on one of them: 256 divided by the number of processors. Fact

The shapes come from our architecture document, which lists them per crop size; in B × 196 × 8 the last size is the 8 candidates of each anchor, and the teacher fills these rows without gradient. The patch term's targets are large: for one micro-batch of 32 images over the ten crops they are 32 × 680 rows of 4,096 numbers of 4 bytes each, about 357 MB, and the student's patch probabilities are about as large. Fact

4.6 One step, from end to end

Assumes: 4.1 to 4.5, and AdamW and the EMA (Part 3). Gives: the order of one training step with its tensor shapes, one picture followed through a whole step of a small network, and the loop by which the model trains itself.

Sections 4.1 to 4.5 took the method apart. This section puts it back together twice: Lab 4B steps through one optimiser step in nine stages, with the real tensor shapes, and Lab 4C follows one picture through a whole step of a small network that keeps the real design. First, the loop that both of them are one turn of. Toy

Nothing in this loop needs a label. The targets come from the teacher, the teacher is an average of the student's past weights, and the positives come from the teacher's own tokens. What keeps the loop from settling on one answer for every patch is a set of guards: the stop-gradient, so the student cannot move its own targets; the averaging, so the targets change slowly; the asymmetry of 4.4; and the centring and sharpening of Part 3, which SelfPatch applies in both heads. Fact

The SelfPatch paper draws the whole method in its Figure 2: two rows, image-level self-supervision above patch-level self-supervision, and a side panel for choosing positives. The figure below redraws it in this course's colours, green for the student, blue for the teacher and orange for the positives, and keeps the paper's own labels, the online network for the student and the momentum network for the teacher. Fact

Read the notation once and the paper's equations become readable: f is the backbone, h the aggregation module, g the image-level projection head and g′ the patch-level one, D the distance, sg the stop-gradient, P(i) the positives of patch i, N the number of patches, x and x⁺ the two views, θ the weights, and a tilde marks the teacher's copies. With it the paper's image-level loss is D(g(y), sg(g̃(ỹ))), its patch loss is (1/N) Σ_i D(g′(f⁽ⁱ⁾), sg(g̃′(ỹ⁽ⁱ⁾))), with each student patch predicting from its own token and the teacher from its aggregated positives, and the total adds them with λ. These are the paper's Eqs. (6) to (8). Fact

The positives listed in the figure's side panel are toy: Lab 4C's small network picked them with random weights on global crop 1 of the drawn sample. That crop covers 65 % of the picture and is flipped left to right, colour-jittered and blurred, as the lab's second stop shows, so the scene appears mirrored. The figure was computed once with the page's own painting of the sample; your browser paints the sample with its canvas, whose pixels can differ slightly, so the lab's live map can list other toy positives. Toy

Lab 4C runs a miniature of this network in your browser and shows its numbers. It keeps the real design: 12 blocks with 6 attention heads each, 196 patches of 768 numbers per 224-pixel crop, the aggregation module and both heads. Only the widths shrink and the weights are random: a token holds 48 numbers where the real one holds 384, and the lab writes each toy width beside the real one. It computes three of the 10 crops. Fact

Its step follows the recipe of our earlier selector study, the runs that kept SelfPatch's loss (Part 5): a peak learning rate of 5e-4, 9,240 steps in 20 epochs, gradient clipping at 3.0, and the first epoch with the prototypes held fixed. Fact

The lab walks through the step in fourteen stops, in four stretches. From pixels to tokens (stops 1 to 4): a real step cuts 10 crops from the picture; a 224-pixel crop is 3 × 224 × 224 numbers, cut into 196 patches of 16 by 16 pixels, 768 numbers each, and the patch embedding turns each patch into a token, 768 → 384, and adds its position row. Fact

Through the backbone (stops 5 and 6): the tokens pass through 12 blocks. In each, every token gathers from the others by attention and passes through the MLP, both results added back onto it. Each block hands the next one the same shape, so a global crop leaves the backbone as 196 × 384 numbers. Fact

Targets, guesses and losses (stops 7 to 11): the teacher runs the network on all 10 crops without gradient. For each patch it keeps the 4 most similar of its 8 neighbours, merges them into a target and makes probabilities at temperature 0.04. The student predicts from the patch's own token at temperature 0.1. Cross-entropy scores each guess, and L = L_global + 0.1 × L_patch. Fact

The update (stops 12 to 14): the backward pass gives each student weight its gradient. Each tensor's gradient is clipped to a length of at most 3.0, AdamW moves the weights, and the teacher averages. Fact

How does the update use the gradient? AdamW keeps two running averages for every weight, its moments: m of the gradient, keeping 0.9 of its old value at each step, and v of the squared gradient, keeping 0.999. Both start at 0, so their first averages are too small, and dividing them by 1 − 0.9^t and 1 − 0.999^t at step t makes up for it. The step then subtracts the learning rate times m / (√v + ε) from the weight, with ε = 1e-8, so a weight whose gradient keeps the same sign moves by about the learning rate, whatever the gradient's size. Weight decay also pulls the weight matrices towards zero. Fact

In the selector study's recipe the learning rate warms up linearly to 5e-4 over 4,620 steps, then falls along a cosine towards 1e-6 by step 9,240. The last layer of each head, which holds its prototypes, stays fixed for 1 epoch, the first 462 steps; DINO's code notes that this helps training. Fact

Does the step help? The lab runs the student again on the same crops, against the same targets, and shows the loss before and after. Each weight that is not held fixed moves against its own gradient, so a small enough step lowers the loss; a larger one can overshoot, and the lab says which happened. Repeated over many steps, this is how the patch loss teaches: each patch's own token learns to predict what the teacher makes of the neighbours that look like it. Toy

4.7 SelfPatch's own results, and why they are not our endpoint

Assumes: 4.1 to 4.4. Gives: how SelfPatch was trained and judged, our check of its released checkpoint, and why none of its numbers can answer our question.

SelfPatch pre-trains ViT-S/16 on ImageNet for 200 epochs with a batch of 1,024 images. Its official script starts from random weights and loads no pretrained model: DINO's objective and SelfPatch's are trained together from the first step, with DINO's optimiser and schedule, AdamW with a linear warm-up over 10 epochs and a cosine decay. DINO's rule sets the peak learning rate to 0.0005 × 1,024 / 256 = 0.002. Fact

The patch loss adds about 21 % to the training time per epoch, so the authors train 200 epochs against the 300 of their DINO baseline, a separate public model, to keep the comparison fair; they report that DINO + SelfPatch is 1.24 times faster than DINO in total running time. One caveat: the official script's default is 300 epochs and the command in its instructions sets none, while the paper states 200. Fact

The paper's ablations, the source of every DINO-versus-variant score in 4.1 to 4.4, use less: a small ViT-Ti/16 pre-trained on COCO's training images for 200 epochs at a batch of 256, with 2 global and 2 local crops and 4,096 prototypes in both heads, about 23.7 M image presentations against about 256 M for the main run. Fact

How did the authors show that SelfPatch helps? Through downstream tasks: after the label-free pre-training, the network is put to work on a task with human labels, such as finding and outlining every object, and scored. For COCO the paper fine-tunes a Mask R-CNN detector with an FPN, standard detection networks that this course does not take apart, under the standard 1× schedule, a fixed length of fine-tuning, using COCO's human labels, and scores box and mask average precision (AP), from 0 to 100. For ADE20K it fine-tunes a Semantic FPN, a standard network that gives every pixel a class, on labelled scene photographs and reports mIoU. DAVIS 2017 carries labelled masks from frame to frame of a video with the frozen patch features; its score (J&F)_m is the average of a region similarity J and a contour accuracy F. Fact

DINO here is a public ViT-S/16 trained for 300 epochs, and DINO + SelfPatch is trained for 200. Every number of the table comes from supervised fine-tuning or from label propagation. Fact

Our study measures something else. The label-free decoder of Part 6 reads the frozen tokens of a checkpoint and finds masks without any labels, and only then are the masks scored once against human outlines. A fine-tuned detector's AP and a label-free decoder's score are two kinds of number that cannot be compared, and none of the paper's numbers can tell us whether a rule for choosing positives helps label-free object discovery. Fact

We have checked one of the table's numbers ourselves, as an artefact check. Our re-evaluation of the released SelfPatch Mask R-CNN checkpoint on all 5,000 images of COCO val2017 gave 42.09 box AP, against 42.1 in the paper, and 38.46 mask AP, which rounds to the paper's 38.5. It shows that the released checkpoint, the scoring program, the data and the configuration fit together; it says nothing about whether our rules help label-free segmentation. Fact

Which run made the released DINO + SelfPatch file is an open question: it is a SelfPatch run from scratch of 200 or 300 ImageNet epochs, since the paper's dense-task model has 200 epochs, the instructions' command uses the code's default of 300, and the file carries no record of its epochs. It is not a DINO model fine-tuned with SelfPatch. Fact

Our adapted DINO baseline, also a supervised Mask R-CNN result, scored 36.19 box AP and 33.47 mask AP, below the paper's 40.8 and 37.3: DINO's file would not load into SelfPatch's detection code as it was, and our own adapter is a diagnostic that does not reproduce the paper's DINO row. The gap between 38.46 and 33.47 must not be read as a measured SelfPatch improvement. Fact

4.8 Real neighbours

Assumes: 4.2, and the eight photographs of the course (Start here). Gives: the released model's real window positives on real photographs, and the observation our own rule starts from.

Everything so far used toy numbers. Lab 4E uses real ones: the 196 tokens of a photograph's 224-pixel view under the released DINO + SelfPatch checkpoint, its teacher's weights, computed once by the course's bundle script and stored with 8 bits per number. Click any patch: the lab ranks its 8 window neighbours by their real cosines, keeps the top k, and beside them shows the cosine to every patch of the crop. Real

On the released model the patches most like an anchor are often outside its window: in our premise tests only 43.6 % of an anchor's 8 most similar patches lie inside its 3 by 3 window, at a mean grid distance of 2.65, and of an anchor's 4 most similar patches in the whole crop, 58.7 % lie inside its window at 224 pixels. A patch's most similar partners are often further away than one step. Fact

Try the map in Lab 4E on the followed photograph. Anchor 34, where the lab starts, is an exception: 7 of its 8 most similar patches lie in its window. Click patch 0 in the top-left corner: only 3 of its 8 do, and the other five lie two to four rows or columns away. Real

That observation is where our rule begins. Part 5 keeps DINO's image-level loss, drops SelfPatch's window and patch loss, and ranks all 195 other patches of a 224-pixel crop to choose positives anywhere in it, with negatives and a contrastive loss: L = L_global + λ L_con. Fact