Pixels to Islands

0.1 The thesis in one sentence

Assumes: nothing. Gives: what the thesis tries to do, the question this course centres on, and what label-free means.

The thesis studies label-free multi-object segmentation: finding the separate objects in everyday photographs, those of the COCO and VOC collections, without a human telling the computer, while it learns, what or where they are. Fact

The question that this course centres on is the thesis's current one: does a patch objective whose positives are the most similar patches anywhere in the crop, and whose negatives are the crop's least similar fifth, change the quality of the masks that a label-free decoder reads out of the network? Part 5 builds that rule and tells the selector study that came before it as history. Fact

A label is a human-made answer attached to a picture: the outline of an object, a box drawn around it, the name of its category or a caption. Label-free means that no label enters training. The training package has no reader for labels and refuses to start if a label file can be reached from its folder of images, so the networks learn from the photographs alone. Fact

Outside training, labels are read in two places, and every use is recorded. The validation panel, 1,000 COCO photographs, is read with its labels in a ledger, one row for every use: its labels may inform the decoder's settings, checks of the metrics, the recipe and the length of a campaign, and never a choice between runs or of a checkpoint. The decoder's settings were fixed that way once, on the converted DINO network, before any network that we trained was scored (6.1 tells why). A separate test panel stays sealed until one final reading, and the contrastive study opens no sealed panel. Fact

0.2 How to read this course

Assumes: nothing. Gives: the four marks, the colours, how a lab, an equation walk and a why card work, and how to move through the course with a keyboard or on a touch screen.

Every statement of the course wears one of four marks, a small badge at the end of its paragraph that says where the statement comes from. Click or tap a mark, or reach it with the Tab key and press Enter, and a box opens that says what its kind means and names its source. Fact

• Fact Fact: a statement copied from a paper or from our own code, protocols or reports. Its badge has a thin solid border, and its source is a short tag such as [DINO §3.1]; section 8.3 says what every tag points to.
• Real Real: a number computed once from the real models on real photographs, stored in the course's data files and drawn by the page. Its badge has a thick blue border, and its box names the data file and the member that hold the number.
• Toy Toy: the page's own simplification, computed in your browser to show how an idea works. Its badge is grey, dashed and hatched, and its numbers measure nothing about the real model.
• Reading Reading: what we ran and what it showed, quoted from our reading notes and claimed no further. Its badge has a dotted orange border.

Colours keep one meaning in every lab and every figure, as the key below shows. Two badges borrow two of these colours as a second cue, blue for Real and orange for Reading; the word on a badge always says its kind. Fact

• green: the student network
• blue: the teacher network
• orange: a positive, a patch that should agree with the one you follow
• purple: a negative, a patch that should disagree with it
• grey: ignored or empty

A lab is a panel you can work with. Its header gives its name, the question it answers and its marks. Its controls sit on the left and its drawing on the right, one above the other on a narrow screen, and the drawing is there before you touch anything: move a slider, click a square or press a button, and it changes at once. Under it, "What you are looking at" says what the drawing shows, and a lab that wears the Toy mark adds "Real model versus this toy", where it says what the real network does differently. Fact

A lab that draws real numbers fetches its data files only when you scroll near it, and until then it says "loading the real numbers…". Some labs let you pick one of the course's eight photographs, and a few take a picture of your own, which never leaves your browser. Fact

Every equation comes with an equation walk in five blocks: the formula; every symbol with its size; a worked example with small numbers that you can redo by hand; the gradient, which says what moves, in which direction and by how much; and what the term does over training. Fact

Every choice of the papers and of our protocol that the course explains has a why card with five fields: Decision, Why, Evidence, What it does to training, and If you changed it. Part 8 collects them all in its why ledger. Fact

The bar at the top links all nine pages and shows how far down the page you are, and each page ends with links to the page before and the page after. Sections are numbered in reading order, and a glossed term, set in italics and underlined, opens its definition when you click or tap it. Fact

With a keyboard, the Tab key reaches every link, control, glossed term and mark; Enter or Space opens a term's definition or a mark's source, and Escape closes it again. A drawing too wide for the screen joins the Tab order, so that you can scroll it from the keyboard. The film of Part 7 steps with the left and right arrow keys and jumps with Home and End. On a touch screen, tap wherever a mouse would click. Fact

Only the films move by themselves, and only once you press Play. If your system asks for less motion, a film shows the finished picture of each stop and moves on without animating it. Fact

0.3 The map

Assumes: nothing. Gives: the eight parts in reading order, a sentence on each, the time its text takes to read and the number of its labs.

The parts build on each other, so read them in order the first time; a reader who knows the basics can start at Part 2 or Part 3 and use the glossary. The times are the minutes that each page's text takes at 200 words a minute, as the build measures them; the labs take as long as you let them. Part 8's time counts its own text only, since its cards and glossary entries repeat the earlier parts. Fact

1. Part 1 Numbers, vectors and learning from zero 42 min, 5 labs An image as numbers, vectors and cosine similarity, a layer, gradient descent and softmax with a temperature, each with an example you can redo by hand, and how a network can learn without labels. Fact
2. Part 2 The Vision Transformer 38 min, 4 labs Patches and tokens, attention worked for four tokens, one block and the twelve blocks of ViT-S/16, ending on the real attention of a trained network. Fact
3. Part 3 DINO 59 min, 6 labs A student and a teacher that agree on what an image is without labels: the crops, the head and its prototypes, centring and sharpening, the loss, and the teacher as an average of the student's past. Fact
4. Part 4 SelfPatch 63 min, 5 labs Patches that learn from their neighbours: the 3 by 3 window, the top 4, the aggregation head and the patch loss beside DINO's. Fact
5. Part 5 Our rule 68 min, 7 labs The thesis's contrastive rule: positives anywhere in the crop, negatives from its least similar fifth, the InfoNCE term and its gradient, the three runs and what their readings showed. Fact
6. Part 6 Reading the model 53 min, 5 labs How a frozen decoder reads objects out of a trained network, and how its masks are scored without cheating. Fact
7. Part 7 The pipeline film 9 min, 1 lab One real photograph followed from its pixels to its masks, as a film of twelve stops that you can play, pause and step through. Fact
8. Part 8 Why ledger, glossary, sources 19 min, for looking things up Every why card in one place with a filter, the glossary of every page, and the sources behind every mark. Fact

0.4 You will be able to

Assumes: nothing. Gives: ten things you can do after the course, each with the section that teaches it.

These are the ten outcomes the course is built for; each names the section where it is taught. Fact

• Read a photograph as numbers, and compute the cosine similarity of two vectors by hand (1.1, 1.2).
• Run gradient descent by hand on three points, and say what the learning rate does (1.4).
• Compute attention for four tokens, and say why it divides by the square root of the width (2.3).
• Follow one transformer block number by number, and say why it has residual connections and LayerNorm (2.4).
• Say why DINO needs a teacher that is an average of the student's past, and what centring and sharpening prevent (3.4, 3.6).
• Say what SelfPatch adds to DINO, and how one patch picks its positives (4.1, 4.2).
• State our rule exactly: its positives, its negatives, its InfoNCE term and the pull and push of its gradient on one token (5.2, 5.4).
• Decode regions from the keys of a network with MaskCut, and say what each frozen setting of the decoder does (6.2).
• Score predicted regions against true objects with a Hungarian match, and say what each score rewards (6.5).
• Follow one real photograph from its pixels to its masks, and find the reason behind any choice in the why ledger (7.1, 8.1).

0.5 What this course is not

Assumes: nothing. Gives: the limits of what you are reading.

It is not the thesis. The owner has not started writing the thesis, and this course is teaching material, written to explain the work. Fact

It makes no scientific claim of its own. Every number of three or more digits on its pages comes from its fact sheets or from the toy values of its labs, and its checks refuse any other; a reading of our runs is quoted as its note gives it, and claimed no further. Fact

Its toys are not the real model. A Toy mark says so wherever a number is the page's own, and each toy lab says what the real network does differently. Fact

No picture in it was made by an image generator. The drawings are the pages' own, and the only photographs are eight COCO train2017 images stored in the course's data files, read without any label, mask, box, category or caption. There is no video and no GIF: the films are drawn by the page. Fact

This is version 1 of the course. A second wave is planned to add labs on one real step of the rule's gradient, on the schedules of our runs, on the token of one patch through the twelve blocks and on the decoder's stages, with more worked examples and smoother animations; a third will take in the owner's feedback. Fact