The pipeline film

One real photograph, a dog and a cat by a door, followed from its pixels to the regions that the frozen decoder reads out of a trained model, in twelve stops that you can play, pause and step through. Fact

7.1 How to watch the film

Assumes: nothing; each stop names the section that teaches it. Gives: how to move through the film, and what in it is real.

Play runs the twelve stops one after another, and Pause holds the picture. Back and Next, the numbered squares and the left and right arrow keys move one stop at a time, and Home and End jump to the first and the last stop. The speed buttons play at half, normal or double speed. If your system asks for less motion, Play shows the finished picture of each stop and moves on without animating it. Fact

Every stop is also a still picture with its own controls: point at it, click it, or move its slider, and its numbers follow. The panel beside the picture explains the stop, with a mark on every statement. Fact

Every real number in the film comes from the course's data files. Most are run 2's at the end of its 9,240 steps: its teacher's tokens and attention and its student's tokens on the 224-pixel view of this photograph, and its teacher's keys on the 480-pixel view. Stop 12 adds the regions that the decoder finds under the converted DINO and released SelfPatch, and stops 8 and 10 quote run 2's training log at four of its steps. Fact

Run 2 trained our rule from the converted DINO start, with fresh SelfPatch heads, for 20 epochs over the 118,287 images of COCO train2017, without a single label. Fact

Seven things in the film are this page's own, and each says so: the pixel values of stop 1, as your browser decodes the photograph; the crops other than the bundle's view; the scaling of each attention map in stop 4; the shades of stop 5; the flat slice of stop 9; the arithmetic of the AdamW update and of the moving average in stop 10; and the two objects that stand in for a truth in stop 12. Toy

The pipeline film

What happens to one real photograph, from its pixels to the regions that the decoder reads out?

Real Toy

What you are looking at: a film in twelve stops. The strip at the top plays, pauses and steps it, and its numbered squares jump to a stop. Under it, the stop's picture with its own controls and numbers, and beside it (below it, on a phone) the explanation of the stop with a mark on every statement.

Real model versus this toy: the real numbers were computed once by the course's bundle script (section 7.3) from run 2's final checkpoint and, for the regions of stop 12, from the converted DINO and released SelfPatch, or read from run 2's training log for the steps that stops 8 and 10 quote, and stored in the data files. This page draws them and recomputes some of them (the sets, the InfoNCE term, the gradient and, on request, the decoder) with the course's checked runtime. The pixel values of stop 1, the crops other than the bundle's view, the scaling of the maps of stop 4, the shades of stop 5, the slice of stop 9, the AdamW and moving-average arithmetic and the toy objects of stop 12 are this page's own. The decoder here reads the keys of the photograph's centre square at 480 pixels, where the readings' exports resize the whole photograph to 480 by 480, and its bilateral refinement and the resize of its masks to the photograph's size are not repeated. Fact

7.2 The twelve stops

Assumes: nothing. Gives: what each stop shows, what in it is real, and where the course teaches it.

The stops run in the order of the design of this part, from the pixels to a reading. Fact

Stop	What it shows	Marks	Taught in
1	The photograph as pixels of three numbers each, and their normalisation	Real Toy	Part 1, section 1.1
2	The bundle's 224-pixel view, and the crops of a training step	Real Toy	Part 3, section 3.2
3	The 14 by 14 patches, patch 34 and its token	Real	Part 2, sections 2.1 and 2.2
4	Where patch 34 takes its attention from, in every block and head	Real Toy	Part 2, sections 2.3 to 2.6
5	The tokens of the teacher and of the student of run 2	Real Toy	Part 3, sections 3.1 and 3.6
6	The 196 by 196 cosines of the teacher's tokens and the two quantiles	Real	Part 5, sections 5.2 and 5.3
7	The positives and negatives of patch 34, and the caps	Real	Part 5, sections 5.2, 5.3 and 5.5
8	The shares of each denominator and the InfoNCE term at any temperature	Real	Parts 1 and 5, sections 1.5 and 5.4
9	The gradient on the student's token, split into the pull from the positives and the push from the negatives	Real Toy	Part 5, section 5.4
10	AdamW updates of a toy weight, and the teacher's moving average after each	Real Toy	Parts 1, 3 and 5, sections 1.4, 3.6, 5.7 and 5.8
11	The key rule, the links, the Fiedler vector and the regions of the decoder	Real	Part 6, sections 6.2 and 6.3
12	IoU, the Hungarian matching and the other scores, on two toy objects	Real Toy Reading	Part 6, sections 6.5 and 6.7

7.3 Where each number came from

Assumes: nothing. Gives: what a SHA-256 sum, a checkpoint and a commit are, and the files, models, training log and code behind every real number of the film, each with its sum.

A SHA-256 sum is a fingerprint of a file: 64 characters, each a digit or a letter from a to f, computed from every byte of the file. Change one byte and the sum changes completely, so two files with equal sums hold, in practice, the same bytes. Fact

A checkpoint is a file that holds a network's weights at one moment of its training. Our runs write one every 5 epochs and call it an envelope: it holds the weights of the student and of the teacher, the optimiser's running averages, the run's settings and everything else that training needs to go on from that step. Fact

A commit is a saved version of a project's code in the tool Git. Its name is a fingerprint of 40 characters that changes when any of its files changes, so a commit's name pins down one exact version of the code. Fact

The data files were computed once by this course's script build/make_bundle.py, on the main processor (the CPU) of the compute server gpu01. The script loaded each model, read its tokens, keys and attention, and ran the decoder with the export package: the copy of our code on gpu01 from which run 2's exports for its readings were computed. The bundle's manifest, a file that lists every input and output of the script with its SHA-256 sum, records them all; the film reads the five files below. Fact

File	What the film reads from it	Bytes	SHA-256
`images/000000114269.jpg`	the photograph	`121,477`	`ec62e965a2699adb4d043c0468817062bc7aa5019358fdf8ececd104d3de36e8`
`images.json`	the size of the photograph and how its 224 and 480 pixel views were cut	`12,045`	`b53a00fb0c8834973c8ecad222dd0b25a8386420b6b63d11e61d9fb4eb6f80ce`
`followed.json`	run 2's student and teacher tokens, the similarity matrix, the sets of every anchor, the InfoNCE values and shares, the gradient with its pull and push, the attention of patch 34 in every block, and the decoder's stages and regions under run 2, the converted DINO and released SelfPatch	`6,516,367`	`a992c9c90622b1222e1540653f8a04801b5345653c893c0aa8257f2d37700be0`
`000000114269_con2_tokens224.json`	run 2's teacher tokens of the 224-pixel view, stored as 8-bit integers	`104,966`	`fd6bde7ceaec650154b4a3bbce70cb1c814bc82a857fb7e0baf6777043e1abc4`
`000000114269_con2_keys480.json`	run 2's teacher keys of the 480-pixel view, stored as 8-bit integers	`480,046`	`84302738a38eeac63c4e49615e164289ebb876b62a5bffa71c3ffc89d06828fd`

Run 2's checkpoint is its final envelope ckpt_epoch020.pt, of 813,644,913 bytes and SHA-256 29d298a885ee0c99521e13d0d9846500a2565ad2b89b0bb59f20e97ae4303dec, written after step 9,240 of the run with seed 906. The run started from DINO's released weights, the file of SHA-256 1566d50496f27f52f07fea6094fa29b2fdd6fae89da65bdd3ebc3b24ef6b7eb7, placed in the SelfPatch network with fresh heads. The envelope's teacher weights give the tokens, the attention and the keys, and its student weights give the student's tokens. Fact

Stop 12's converted DINO is DINO's backbone from that same file, placed in the SelfPatch network by the export package's loader, and released SelfPatch is the released checkpoint dino_selfpatch.pth, of SHA-256 3656c2f42b724ee8ee546c06eb154f7235b56845c39fd52e40bf8cfc29ef7e43, read with its teacher's weights. The steps that stops 8 and 10 quote come from run 2's training log progress.jsonl, one line per step, of SHA-256 36a0ddc9f8c209d84f3131fc396716870ebfd2c646440141fdc79059e09d1571. The bundle's telemetry.json keeps its step 1, every tenth step and the first step of each epoch, and the film quotes those values through the course's fact sheet. Fact

The script has SHA-256 d7ca2087cb3cb2362eb351c17e2eb86c8360053f2d258f807e6b84324fb73b67. It ran at 2026-10-13T00:22:14Z under Python 3.12.14, the programming language of our code, and PyTorch 2.12.0+cu126, with the export package at commit 604d6a3cb21ee8b539e5440be9fc7208bcc80b6b. It carries the rule's code copied unchanged from cluster/contrastive.py:119-212 at commit 3c49ec44d0bbd0951084f82656baa8ece68376b9, with which it computed the sets, the InfoNCE values and the gradient, and it ran the decoder with the frozen configuration of SHA-256 b9cbe9d561c04ed9ef50b7e762cab558245253ec1fe1697f9c9e37345d053e75. Fact

Tokens and keys are stored as 8-bit integers with one scale per row. That changes no cosine between two tokens by more than 0.00621, and none between two keys by more than 0.00367, or 0.00692 after the key rule. Fact

The bundle read images only: no label, mask, box, category or caption of any photograph was opened, no GPU was used, and its numbers enter no scientific result. Fact

The page's own code was checked against these numbers: its sets of every anchor equal the bundle's, its InfoNCE values agree within 1e-6, its gradient equals the one from PyTorch, and its decoder gives the Python decoder's masks. Fact

To follow any number further, click its mark: a Real mark names the data file and the member that hold the number, and a Fact or Reading mark names its source. Fact

Glossary of this page

AdamW: The optimiser of our runs and of DINO. For every weight it keeps a running average of the gradient and of the gradient's square, steps by about the learning rate against the averaged gradient, and shrinks the weight a little by its weight decay.
Attention: The part of a transformer block in which every token takes a weighted mix of all the tokens. The weights come from a softmax over the scores of the token's query against every key, so they add up to 1.
Attention head: One of the copies of attention that run side by side in a block, each on its own slice of the numbers. ViT-S/16 has 6 heads of 64 numbers each.
Checkpoint: A file that holds a network's weights at one moment of its training. Our runs write one every 5 epochs and call it an envelope.
Commit: A saved version of a project's code in the tool Git. Its name is a fingerprint of 40 characters that changes when any of its files changes, so it pins down one exact version of the code.
Cosine similarity: The cosine of the angle between two vectors: their dot product divided by both lengths. It runs from −1 for opposite directions through 0 to 1 for the same direction.
Crop: A rectangle cut from an image and resized to a fixed size: 224 by 224 pixels for a global crop, 96 by 96 for a local crop.
Envelope: Our runs' name for a checkpoint: the weights of the student and of the teacher, the optimiser's running averages, the run's settings and everything else that training needs to go on from that step.
Fiedler vector: The eigenvector of the second-smallest eigenvalue of a graph's Laplacian. Splitting its values at their mean cuts the graph where it has few links, which is how the decoder finds its normalised cut.
Gradient: How much, and in which direction, a quantity changes when each of its inputs changes a little. Training moves every weight a small step against the gradient of the loss.
Hungarian matching: The one-to-one pairing of true objects with predicted regions that makes the sum of their IoUs as large as possible; an object left without a region scores 0.
InfoNCE: A contrastive loss. For an anchor and one positive it is minus the log of the positive's share in a softmax over the positive and all the negatives, so it falls as the positive outscores the negatives.
IoU: Intersection over union: the pixels that two masks share divided by the pixels that either covers, from 0 for no overlap to 1 for the same mask.
Key: One of the three vectors, query, key and value, that attention makes from each token. The decoder reads the keys of the last block as the features of the patches.
Key rule: The decoder's first step: the image's mean key is taken away from every key, so that what all the keys share does not link every patch with every other.
Moving average: Here, the teacher's weights: after each step every teacher weight moves a small part of the way to the student's weight, so the teacher is a smoothed average of the student's past.
Negative: For an anchor, a token that the rule asks the student to move away from: one of the tokens of the crop least similar to the anchor, at or below τ_lo.
Normalisation: Dividing every pixel value by 255, then taking away a mean and dividing by a standard deviation for each colour, so that the network reads numbers of a similar size.
Normalised cut: A split of a graph into two parts that cuts few links compared with how linked each part is inside; the decoder finds it with the Fiedler vector.
Patch: A 16 by 16 square of pixels, the piece that the Vision Transformer reads; a 224-pixel view holds 196 of them.
Pixel: One dot of a photograph, stored as three numbers from 0 to 255 for its red, green and blue.
Positive: For an anchor, a token that the rule asks the student to move toward: one of the anchor's most similar tokens in the crop, by the teacher's cosine.
Quantile: The value below which a given share of a list of numbers lies: a fifth of the values lie at or below the 0.20 quantile.
SHA-256 sum: A fingerprint of a file: 64 characters, each a digit or a letter from a to f, computed from every byte of it. Change one byte and the sum changes completely, so two files with equal sums hold, in practice, the same bytes.
Student: The network that the loss trains by gradient descent.
Teacher: The network whose weights are a moving average of the student's. It chooses the targets and receives no gradient.
Temperature: The number that scores are divided by before a softmax: a small temperature makes the softmax sharper, a large one flatter.
Token: The list of 384 numbers that the Vision Transformer keeps for each patch.