// part 6 of 8

Part 6

Reading the model

Training changes the numbers inside the network, and nothing in training says where one object ends and the next begins. This part shows how we read objects out of a trained network without letting a label steer anything: a frozen decoder cuts the patches of one picture into regions, a fixed set of scores compares those regions with outlines drawn by people, and a linear probe asks a second question of the same tokens. Fact

6.1 Why a frozen decoder

Assumes: what a token and a key are (Part 2), and that three runs trained the network with our rule (Part 5). Gives: what reading a model means, why the decoder never changes, and the one place where labels may enter.

A trained backbone takes a picture and gives back a vector for every patch. A score needs more than vectors: it needs regions, groups of patches that the model treats as belonging together. The decoder turns vectors into regions. Ours has no weights of its own and learns nothing. It is a fixed recipe, MaskCut, that reads the keys of the final block and splits a graph of one picture's patches in two, again and again. Fact

We call it frozen because its settings never change. They were chosen once, by a grid of 144 decoder passes over the 1,000 images of the COCO v4 validation panel with the untrained converted DINO, the start of run 2, ranked by AP50 and then by FG-ARI, two scores that 6.5 defines, and fixed before any trained checkpoint was scored. Fact

The design states the reason in one line: the settings are chosen on the zero-step DINO baseline, "not on any trained model, so the yardstick cannot favour an arm". Fact

Picture the other way of doing it. If the decoder were tuned again for each checkpoint, a better score could come from a better-tuned decoder as easily as from a better network, and the two could never be told apart. With one decoder for every checkpoint, a difference between two scores can only come from the keys it is given, which means from training. Toy

The reading path. A checkpoint's frozen teacher (blue) turns each picture into keys; the frozen key rule and MaskCut turn the keys into at most five regions, which are sealed with their hashes before any label is read. Only the evaluator, in the shaded half, opens the truth masks, and its scores enter a reading note that decides nothing. Fact

The decoder reads the teacher's weights, as SelfPatch and DINO evaluate the teacher; at step zero teacher and student are the same network, so the choice of the settings does not depend on which of the two the grid read. Fact

In the decoder's reading, labels enter at one place only, the evaluator; the linear probe of 6.6 is a separate measurement with labels. Training reads images and records annotations_accessed as false, and no label enters training, a gate of a run, a value of a run, the choice of its checkpoint or the decoder. Every run is read at its final checkpoint, the one saved at the end of epoch 20, so no score ever chooses a checkpoint. Fact

Decision

One decoder, MaskCut with fixed settings, chosen once on the untrained converted DINO and never changed after a trained checkpoint was scored. Fact

Why

The settings are chosen on the zero-step DINO baseline, "not on any trained model, so the yardstick cannot favour an arm". Fact

Evidence

The settings won a grid of 144 passes over the 1,000 validation images (feature, side, τ, region count, score and refinement), ranked by AP50 then FG-ARI, two scores of section 6.5; the winner, with up to 5 regions, beat its twin with 3 by only 0.00039 AP50. Fact

What it does to training

Nothing: the training package holds no annotation reader and no evaluation code, so the decoder never runs during training and sends no gradient. It only reads the teacher's keys after training. Fact

If you changed it

Tuning the decoder per checkpoint would mix the decoder's tuning and the network's training in one score. Even the frozen choice carries some selection optimism, because it was made on these very images: above about 0.6 FG-ARI points it would make one of the report's verdicts indeterminate, and one research lane estimates up to about 1 point. Fact

Decision

What we never do: choose a checkpoint, a training value or the decoder by a score that used labels. Fact

Why

Every run is read at its final checkpoint, so no score ever chooses a checkpoint, and no COCO label enters training, a gate, a value of the run, the choice of its object or the decoder. Fact

Evidence

The study's 15 rows in the validation ledger, from the first early look on 9 October 2026 to the last probe read on 11 October, carry the decision none, 11 of them, or "continue or stop", the 4 rows of the early looks; none of them steered a run. Fact

What it does to training

Training stays label-free: the trainer opens images only and records annotations_accessed as false. Fact

If you changed it

Picking the best of several checkpoints by its validation score would turn the validation labels into a training signal, and the score reported for the winner would be optimistic. Toy

6.2 MaskCut on the keys

Assumes: cosine similarity (Part 1) and the keys of the last block (Part 2). Gives: every step of the decoder with every number of its frozen configuration, and two of its equations worked by hand.

Here is what the decoder does with one photograph, step by step. Every number comes from the frozen configuration, configs/decoder_v2_frozen_v2.json, which every export records by its hash. Fact

One stage of the decoder, and the loop around it. The key rule runs once; then each stage turns the keys into a graph, splits it in two, keeps the side that holds the seed patch as a mask, and paints the masked keys to zero before the next stage. Fact

Keys. The decoder resizes the picture to 480 by 480 pixels, a 30 by 30 grid of 16-pixel patches, runs the teacher, and keeps one key per patch from the final block: 900 keys of 384 numbers. A key is k_i = W_k·LN1(z_i) + b_k, the key part of the last block's attention, computed from the layer-normalised output z_i of the block before it. Fact

The key rule. Before comparing the keys, the decoder subtracts the picture's mean key from each of them: the key rule mean_subtract. The first walk below works it by hand, and 6.4 tells why it exists. Fact

Links. Every pair of patches gets a link weight: one when the cosine of their keys is above τ = 0.2, and ε = 1e-5 otherwise, the link of a patch with itself included. The table of all link weights is the affinity matrix A, 900 by 900. Adding up a row gives that patch's degree, and the degrees sit on the diagonal of a second matrix, D. Fact

The letter τ is used again here. This τ is a threshold on a cosine. It has nothing to do with the temperature of our InfoNCE term in Part 5, which also happens to be 0.2, or with DINO's temperatures in Part 3. Fact

Split. The decoder wants to cut the patches into two groups so that the links it breaks are weak compared with the links it keeps inside each group: the normalised cut. There are far too many ways to split 900 patches to try them one by one, so MaskCut solves one equation instead, (D − A) v = λ D v, and takes the Fiedler vector v₂, the solution with the second-smallest eigenvalue λ₂. Patches joined by strong links get similar numbers in v₂, so its values change most across the weakest seam of the graph. Every patch above the mean of v₂ goes to one side, the rest to the other. Fact

Which side is the object. Call the side above the mean the chosen side. The patch with the largest entry of v₂, ignoring its sign, and the grid's 4 corners decide whether the chosen side is kept: the two sides are swapped when at least 3 of the corners fall on the chosen side, or when that patch does not. The seed patch is then the patch with the largest entry on the side that is kept: the largest of v₂ itself, or of −v₂ after a swap. The mask is the seed's 4-connected component: every patch that can be reached from the seed by steps up, down, left or right without leaving its side. Fact

So the seed is not always the patch with the largest entry. Under released SelfPatch, the fourth cut of the followed photograph has its largest entry at row 12, column 9, counting from row 0 and column 0 at the top left, on the chosen side; but 3 corners lie on that side too, so the sides swap, the seed moves to row 21, column 21, and the region of 216 patches does not hold the patch at row 12, column 9. Lab 6A shows this stage. Real

MaskCut's authors give two intuitions for this rule: object patches should stand out more than background patches, so the object's side holds the patch with the largest entry; and, as "a simple but empirically effective object-centric prior", the object's side should not hold most of the picture's corners. Their text says fewer than two of the four; the code, and our decoder, flip at three or more. Fact

More objects. Up to 5 stages run one after another. Before each new stage the keys of the patches in the previous masks are set to zero, which the code calls painting, and the cut runs again. From the second stage on, a mask is dropped when its IoU with the previous stage's mask, the patches the two share divided by the patches they cover together, is above 0.5 (overlap) or when it covers at most 1 % of the grid (small), and a mask that adds no new patch is empty. So at most five regions come out, each numbered by its stage. Fact

When a cut fails. When λ₂ and λ₃ are equal or nearly so (|λ₃ − λ₂| < 1e-8), or fewer than three eigenvalues exist, or one is not finite, v₂ is not unique and any split would be arbitrary: a degenerate cut. It ends the loop, and the frozen rule keep_prior keeps the masks found before it. At the first stage there are none, so the picture gets no region at all. Fact

Score and polish. Each kept mask gets the ncut score clip(1 − λ₂, 0, 1): a clean cut has a small λ₂ and scores near one. Finally the masks are resized to the photograph's own size, their outlines are refined by a bilateral solver, and the whole decoder runs its numerical libraries on one thread. Fact

The key rule: subtract the mean key

k̃_i = k_i − k̄, k̄ = (1/N) Σ_j k_j

k_i: the raw key of patch i, a vector of 384 numbers
N: the number of keys of one picture, a scalar: 900 at 480 pixels
k̄: the picture's mean key, 384 numbers, one per picture
k̃_i: the key the decoder compares, 384 numbers

Take four keys with two numbers each instead of 384: k₁ = (5, 2), k₂ = (5, −2), k₃ = (3, 2), k₄ = (3, −2). Their mean is k̄ = (4, 0), so the centred keys are (1, 2), (1, −2), (−1, 2) and (−1, −2). All four raw keys point to the right, because they share the offset (4, 0); the centred keys keep only how they differ, up or down. Toy

The cosines show it. Raw: cos(k₁, k₂) = (5·5 + 2·(−2)) / (√29 · √29) = 21/29 = 0.724, and cos(k₁, k₃) = (15 + 4) / (√29 · √13) = 19 / 19.416 = 0.979. Centred: cos(k̃₁, k̃₂) = (1 − 4) / (√5 · √5) = −0.6, and cos(k̃₁, k̃₃) = (−1 + 4) / 5 = 0.6. At τ = 0.2 the raw keys link every pair, while the centred keys link only k₁ with k₃ and k₂ with k₄: two clean groups. Toy

The six pairs of the four keys, before and after the key rule
pair	raw cosine	linked at τ = 0.2	centred cosine	linked at τ = 0.2
1, 2	0.724	yes	−0.6	no
1, 3	0.979	yes	0.6	yes
1, 4	0.567	yes	−1	no
2, 3	0.567	yes	−1	no
2, 4	0.979	yes	0.6	yes
3, 4	0.385	yes	−0.6	no

The real keys behave the same way. On three COCO v4 pictures, 36.3 to 47.3 % of the converted DINO's key pairs lie above τ = 0.2 with raw keys, and 17.6 to 20.8 % after the key rule. Fact

No gradient flows here. The key rule has no weight and runs after training, on keys that no longer change. Its effect is pure geometry: subtracting one vector from every key leaves every difference k_i − k_j as it was and removes only what all the keys share, the key offset of 6.4 among it, which the network's own softmax ignores anyway. Toy

The rule acts the same way on every checkpoint, and it matters more when training grows the shared part. In our pilot, the student's key offset grew from DINO's 4.320 to 6.76 in 5 epochs, a growth that mean subtraction removes and that keys without the bias alone would keep. Fact

The affinity and the normalised cut

A_ij = 1 if cos(k̃_i, k̃_j) > τ, else ε; D = diag(Σ_j A_ij); (D − A) v = λ D v; side = { i : v_2,i > mean(v₂) }; score = clip(1 − λ₂, 0, 1)

A: the affinity matrix, 900 by 900: one link weight per pair of patches, itself included
τ, ε: scalars: the link threshold 0.2 and the hair-thin weight 1e-5
D: the degree matrix, 900 by 900, zero except on its diagonal, where it holds each row sum of A
v, λ: a solution of the equation: a vector of 900 numbers, one per patch, and a scalar, its eigenvalue
v₂, λ₂: the Fiedler vector and the second-smallest eigenvalue; λ₃ is the third-smallest

Use the four centred keys of the first walk on a 2 by 2 grid, k₁ and k₂ on top, k₃ and k₄ below. Each key is linked to itself and to its partner (k₁ with k₃, k₂ with k₄), and every other pair gets ε. So every row of A holds 1, 1, ε, ε in some order, and every degree is 2 + 2ε. Toy

Try v = (1, −1, 1, −1). Row 1 of (D − A) v is (2 + 2ε)·1 − (1·1 + ε·(−1) + 1·1 + ε·(−1)) = 2 + 2ε − 2 + 2ε = 4ε, and row 1 of D v is 2 + 2ε. So λ = 4ε / (2 + 2ε) = 2ε / (1 + ε), about 0.00002, and the other three rows agree. The four eigenvalues are 0, 0.00002, 1 and 1: the first solution, (1, 1, 1, 1), splits nothing, and v₂ = (1, −1, 1, −1) puts k₁ and k₃ above its mean, 0, and k₂ and k₄ below. The largest entry, ignoring its sign, is k₁'s, the first of four of equal size, and it lies above the mean; only 2 of the 4 corners do, so nothing is swapped, and the seed, the largest entry of v₂, is k₁; the mask is {k₁, k₃}, joined from top to bottom, and its ncut score is 1 − 0.00002 = 0.99998. Toy

With the raw keys every pair was linked: A is all ones, every degree is 4, D − A = 4I − 11ᵀ, and the four eigenvalues are 0, 1, 1 and 1. Since λ₂ = λ₃, the cut is degenerate and the picture gets no region. Toy

The real first cut of run 2 on the followed photograph has λ₂ = 0.0238, a clean cut in this sense. Real

Nothing here is trained, so no gradient flows. What moves the answer is τ. A pair whose cosine crosses τ jumps from a full link to ε, so the cut can change all at once at one value of τ and stay put between such values. Lab 6A's τ slider shows this on real keys. Toy

Over training the keys change, and the graph with them. On the followed photograph the converted DINO, run 2's start, has 111,518 pairs of keys above 0.2 at the first cut, and run 2 after its 20 epochs has 180,919; run 2's decoder keeps a single region, 400 of the 900 patches. Real

Decision

The decoder reads the final block's attention keys, 384 numbers per patch, from the teacher. Fact

Why

MaskCut and TokenCut, two label-free object finders, read the key vectors of the final attention layer as their patch features, and the grid kept keys over tokens on the zero-step DINO. Fact

Evidence

The grid's 144 passes tried both features, keys and tokens, at every other setting, 72 passes each (3 sides, 2 values of τ, 2 region counts, 3 scores, 2 refinements); the winning configuration reads keys. Fact

What it does to training

Nothing directly. Our rule pulls and pushes the final tokens; the keys are computed inside the same final block from the output of the block before it, so the decoder sees training through the weights that make that block's keys. Toy

If you changed it

Reading tokens was one of the grid's options and lost on the zero-step DINO. The linear probe of 6.6 reads tokens, and it ranks the models in the opposite direction to the decoder. Reading

Decision

The decoder reads its keys at 480 by 480 pixels: a 30 by 30 grid of 900 keys. Fact

Why

The grid tried three sides, 224, 448 and 480, on the zero-step DINO, and the winner reads 480. Fact

Evidence

The frozen configuration records side 480, and the linear probe's protocol notes that its own tokens, at 224 pixels on a 14 by 14 grid, are not the decoder's keys. Fact

What it does to training

Nothing: training sees two 224-pixel crops and eight 96-pixel crops of each image, and the decoder reads the same weights at a size that training never used, on a finer grid than any training crop. Fact

If you changed it

At 224 pixels the grid would be 14 by 14, so a mask could only follow an object's outline in steps of a fourteenth of the picture. A finer grid costs more: the cut solves an equation with one unknown per patch, 900 at 480 pixels, and its work grows much faster than the number of patches. Toy

Decision

Two patches are linked when the cosine of their keys is above τ = 0.2; at or below it their link weighs ε = 1e-5. Fact

Why

The grid tried τ = 0.15 and τ = 0.2 on the zero-step DINO, and the winner uses 0.2; the strict "above" is MaskCut's own affinity. Fact

Evidence

At this τ the key rule matters: on three COCO v4 pictures the share of the converted DINO's key pairs above 0.2 is 36.3 to 47.3 % with raw keys and 17.6 to 20.8 % after the key rule. Fact

What it does to training

Nothing: it is a setting of the reader, unrelated to the temperature τ of the InfoNCE term or to DINO's temperatures. Fact

If you changed it

A lower τ links more pairs, and the graph drifts towards complete, where every cut is degenerate; a higher τ breaks it into fragments. Move the τ slider of Lab 6A and watch both happen on real keys. Toy

Decision

At most 5 regions per picture, with MaskCut's stop rules: from the second stage, a mask that overlaps the previous one above IoU 0.5, or covers at most 1 % of the grid, is dropped; a degenerate cut ends the loop and keeps what came before. Fact

Why

The grid tried 3 and 5 regions and kept 5; the stop rules are those of MaskCut's archived loop, with the masks found before a degenerate cut kept. Fact

Evidence

The winner with 5 regions beat its twin with 3 by only 0.00039 AP50, a detection score of section 6.5; and the cap limits every score: a decoder with at most 5 regions reaches an average recall, AR100, of 6.17 on the zero-step start, against 12.7 to 27.1 for detector pipelines that propose many more masks. Fact

What it does to training

Nothing; it limits what a reading can credit. With at most K masks and n objects, even perfect masks score min(K, n) / n on a picture. Fact

If you changed it

On this panel the ceiling of perfect masks under the 5-region cap is a Hungarian IoU of 0.7957; fewer regions would lower it, and more regions would let the decoder propose fragments that the matching cannot use. Fact

6.3 Real decoding

Assumes: the steps of 6.2. Gives: the decoder run in your browser on the real keys of eight photographs under three models, and checked against the Python decoder.

Lab 6A runs the decoder of 6.2 on real keys. They were computed once from three real models, the converted DINO (the start of run 2), the released SelfPatch checkpoint (the start of runs 1 and 3) and the teacher of run 2 at the end of its 20 epochs, on the eight photographs of the course, and stored with 8 bits per number. The cut itself is computed in your browser by this page's copy of the decoder. Real

On the followed photograph, the dog and the cat by the door, at τ = 0.2 and with the frozen key rule, this page's decoder gives the same regions as the Python decoder that the bundle ran on the same keys, for all three models, and the lab shows the two side by side. On the other photographs, and at other values of τ, no Python run exists to check it against. Real

On that photograph the Python decoder kept 2 regions under the converted DINO, 3 under released SelfPatch and 1 under run 2, whose single region covers 400 of the 900 patches. Real

Two differences from a reading remain. These keys are those of the photograph's centre square, its short side resized to 480 pixels and the middle square cut out, while the readings' exports resize the whole photograph to 480 by 480; and the bilateral refinement of 6.2 is not run here. Real

MaskCut on real and toy keys

How does the frozen decoder cut a picture into regions, stage by stage, and what does τ change?

Real Toy

What you are looking at: in the real mode, the 480-pixel view of a course photograph with the regions the decoder keeps (blue, numbered by their stage), the Fiedler vector of the chosen stage on the 30 by 30 grid, the histogram of all pairwise cosines of the keys after the key rule with τ marked, and every stage's outcome; for the followed photograph at τ = 0.2 with the frozen key rule, the Python decoder's outcome beside the page's. The toy mode is the first explainer's Lab J: 64 toy keys on an 8 by 8 grid, with every link of the affinity matrix visible. Try this: choose released SelfPatch, switch the key rule to raw keys, and read what happens to the first cut; then subtract the mean again, show stage 4, and find the seed far from the patch with the largest entry of the Fiedler vector. Real

Real model versus this toy: the real mode reads the real keys, stored with 8 bits per number, and runs this page's copy of the decoder with the frozen rules and numbers, without the bilateral refinement and without the resize to the photograph's own size; its keys are those of the photograph's centre square. The toy mode's keys are a fixed random mix of each cell's pixels plus a few hand-made features of colour, texture and position, and it stops after three stages. Real

6.4 The key-bias effect

Assumes: the decoder of 6.2 and cosine similarity. Gives: why a number that the network itself ignores can break the decoder, and why the key rule subtracts the mean key.

One released model broke our decoder completely, and the reason teaches something about every decoder that compares raw keys. With raw keys the released DINO+SelfPatch checkpoint scored 0 on every score where higher is better: every image hit a degenerate cut at the very first stage. Fact

The culprit is the key bias b_k in the formula of 6.2: one fixed list of 384 numbers that the last block adds to every key. Its length is 28.25 in the release and 0.041 in DINO. Counting the shift β1 of the block's first LayerNorm too, the part that every key shares, δ_k = W_k·β1 + b_k, is 32.09 long in the release and 4.320 in DINO. Fact

Inside the network this shared part, the key offset, does nothing. It adds the same amount to all of one query's attention scores, and softmax reacts only to differences between scores, so it cancels: zeroing the bias changes the released network's tokens by at most 1.9e-6. The decoder is different. It compares raw keys by their cosine, and a cosine does not cancel a shared offset. Fact

Why a shared offset fools a cosine. Two keys, blue because the decoder reads the teacher's keys, drawn flat on the page. Add the same long offset to both (grey, because the network ignores it) and they point almost the same way, so their cosine is close to one whatever they looked like before. Subtract the mean of all the keys and the offset is gone again. Real keys have 384 numbers instead of two. Toy

That is what the release's keys do: their mean pairwise cosine is 0.66 to 0.70, with 100 % of pairs above τ = 0.2, while the converted DINO's mean is 0.17 to 0.23, with 36 to 47 % of pairs above it. When every pair clears τ, every link is a full one and the graph is complete. Every split then breaks links of the same strength, λ₂ and λ₃ are both 1, and the first cut is degenerate; the zero-step scoring saw exactly the eigenvalues 0, 1 and 1 at the first stage. Fact

The report tested the cause directly on 12 images. Zeroing the bias gave the release 1 to 2 regions on all 12, and adding the released bias, or a random bias of length 28, to DINO made DINO degenerate on all 12. Fact

The lesson concerns the decoder: raw-key spectral decoders are sensitive to parameters the network never uses, so the way keys are prepared must be label-free and the same for every family of models before any two are compared. Three key rules exist in our code: raw keys, bias-free keys (subtract b_k) and per-image mean subtraction. On 29 September 2026 the owner chose mean subtraction. Fact

With mean subtraction or bias-free keys the released features decode normally at τ = 0.2: 1 to 3 regions per image on 20 of 20 images, with a key geometry like DINO's, a mean cosine of about 0.18. Fact

The two rules differ where it matters for training. Mean subtraction removes any part that every key shares, the LayerNorm's shift included; bias-free keys remove b_k only. In our pilot, after 5 epochs, the student's key bias was 0.107 long while its key offset had grown from DINO's 4.320 to 6.76: the growth is the part that mean subtraction removes and bias-free keys would keep. Fact

Decision

Before the cut, each picture's mean key is subtracted from each of its 900 keys: the key rule mean_subtract. Fact

Why

Raw-key spectral decoders are sensitive to unused bias parameters, so the key rule must be label-free and the same for every model family before arms are compared; on 29 September 2026 the owner chose mean subtraction. Fact

Evidence

Under raw keys the released checkpoint, with a key bias 28.25 long against DINO's 0.041, scored 0 on every score where higher is better; with the key bias out of the way its features give 1 to 3 regions per image on 20 of 20 images. Fact

What it does to training

Nothing, because it runs after training; but it keeps the reading fair when training grows the shared part of the keys, as the pilot's student offset grew from 4.320 to 6.76 in 5 epochs. Fact

If you changed it

With raw keys the release degenerates at its first cut, where the eigenvalues read 0, 1 and 1; with bias-free keys the LayerNorm's part of the offset would stay. Lab 6B shows both on toy keys and the first on real keys. Fact

Lab 6B reproduces the failure. Lengthen the shared vector of the toy keys and watch the cosines crowd towards one until the cut degenerates; then choose a key rule. Below the toy, the same histogram is drawn from the real keys of a photograph. Toy

A bias the network ignores

What happens to the cut when every key gets the same extra vector, and which key rule takes it away?

Toy Real

What you are looking at: above, the toy keys of Lab 6A's toy mode with two shared vectors added to every key, a key bias and a LayerNorm shift, each of its own length: the histogram of all pairwise cosines with τ = 0.2 marked, the share of pairs above it, the regions the toy decoder keeps, and a flat sketch of the same effect; then the key rule, raw, bias-free or mean subtraction. Below, the histogram of the cosines of a photograph's real keys under one of three models, raw or with the mean subtracted, and what the first cut does. Toy

Real model versus this toy: the toy offsets point in two fixed random directions of the toy key space, and the bias-free rule knows the toy bias exactly. The real offset is the release's constant key part δ_k, 32.09 long against 4.320 in DINO, on keys of 384 numbers over a 30 by 30 grid; the real panel offers raw keys and mean subtraction only, because the course's data hold no key bias. Fact

6.5 Scoring a mask without cheating

Assumes: what a region is (6.1 to 6.3). Gives: the primary endpoint worked by hand, the secondary scores and which way is good for each, and why every label read is written down.

A score compares the decoder's regions with truth masks, outlines that people drew around the objects of each photograph of the validation panel. This is where labels enter, after the regions are sealed. Fact

The contrastive study's primary endpoint is Hungarian matched IoU. Each truth object is paired with at most one predicted region, and each region with at most one object, so that the sum of the pairs' IoU is as large as possible; the score is the mean of each object's matched IoU over all its truth objects, an object left without a region counting 0. Fact

IoU, intersection over union, is the area two masks share divided by the area they cover together: one for a perfect match, zero when they do not touch. Why a matching at all? Without it, one large region could take credit for every object it touches. One to one, it is credited for one object at most, and every object the decoder missed keeps its share of the denominator. Toy

A reading also says how far its differences could move with other photographs. For the difference between two checkpoints, the evaluator draws 1,000 images from the panel's 1,000 with replacement, so that some come twice and some not at all, uses the same draw for both checkpoints, computes the difference on that draw, and repeats this 2,000 times, 500 times for AP. The middle 95 % of those differences, from the 2.5th to the 97.5th percentile, is the 95 % image-bootstrap interval that the readings quote. It covers the sampling of images only; a second training seed is outside it. Fact

Read an interval this way. If it holds zero, a difference of that size can come from which photographs happen to be in the panel, so the reading cannot tell the two checkpoints apart. If it leaves zero out, the difference is larger than the choice of photographs explains, which still says nothing about a second training seed. Toy

Hungarian matching and the IoU

IoU(T, P) = |T ∩ P| / |T ∪ P|; π* = the one-to-one π that makes Σ_t IoU(T_t, P_π(t)) largest; score = (1/n) Σ_t IoU(T_t, P_π*(t)), an unmatched t counting 0

T_t: the truth mask of object t: one yes or no per pixel
P_p: the predicted region p: one yes or no per pixel
|·|: a count of pixels, a whole number
π: an assignment: each object to at most one region, no region used twice
n: the number of truth objects of the picture, a whole number

A strip of 30 pixels holds three truth objects, hatched, and two predicted regions, blue; P1 has two pieces. Toy

The six IoUs: T0 with P0 shares pixels 4 to 9 (6) out of 0 to 13 (14), 6/14 = 0.4286; T0 with P1 shares 4 out of 12, 0.3333; T1 with P0 shares 4 out of 10, 0.4; T2 with P1 shares 2 out of 14, 0.1429; the other two pairs share nothing, 0. Toy

The IoU of every object with every region
object	P0	P1
T0	0.4286	0.3333
T1	0.4	0
T2	0	0.1429

A greedy matching takes the largest IoU first, T0 with P0 (0.4286); P1 is left, and its best remaining partner is T2 (0.1429). The total is 0.5714 and the score (0.4286 + 0 + 0.1429) / 3 = 0.1905. Toy

The Hungarian matching looks for the largest total over all six one-to-one assignments and finds T0 with P1 (0.3333) and T1 with P0 (0.4), total 0.7333; T2 stays unmatched and counts 0. The score is (0.3333 + 0.4 + 0) / 3 = 11/45 = 0.2444. Giving up the single best pair lets two objects be matched instead of one with a weak partner. Toy

The project's own evaluator gives exactly these numbers on this strip: the course's tests record its answers. Fact

A looser secondary score, mBO-i, lets one region serve several objects: each object takes its best IoU with any region, so (0.4286 + 0.4 + 0.1429) / 3 = 0.3238, with P0 counted for both T0 and T1. It sits above the Hungarian 0.2444 because nothing stops P0 from being used twice. Toy

On the validation panel, run 2's Hungarian matched IoU is 17.41 per cent and its start's 17.23, means over the panel's 988 defined images. Reading

No gradient: the score is computed after training, with labels, and nothing is fitted to it. It also moves in jumps. Moving one pixel changes one IoU a little, but it can change which assignment is best, and then the score jumps. Try it in Lab 6C. Toy

Read at epochs 5, 10 and 20, run 1's primary endpoint moved by −0.11, −0.24 and +0.04 points against the released start, every interval around zero: the score that training never sees did not move. Reading

Lab 6C draws three truth objects and two predicted regions on a 48 by 48 grid. Drag a region, resize it from its corner, or add one, and watch the table of IoUs, the matching and the score follow. Toy

Matching and scoring

How are predicted regions matched to the true objects, and what is each object's share of the score?

Toy

What you are looking at: three truth objects, A, B and C, outlined in ink and hatched, and the predicted regions, blue and numbered, on a 48 by 48 grid. The table gives the IoU of every object with every region; the outlined cells are the Hungarian matching, and the score is the mean of the matched IoUs over the three objects. Where two regions overlap, the lower number keeps the pixels. Try this: press "Greedy loses" and compare the two matchings; then drag region 2 until it covers B alone and add a third region over C. Toy

Real model versus this toy: the real truth masks are COCO's object outlines at the photograph's own size, with void pixels left out, and the real regions are the decoder's at most five, refined and resized. The evaluator matches them with the same Hungarian rule, gives a pixel that two regions cover to the earlier one, as this drawing does, and a reading is the mean of the score over the panel's 988 defined images. Fact

One number cannot say everything about a set of regions, so every reading reports secondary endpoints beside the primary one. Each has a direction: five are credits, where higher is better, and the background false-positive rate is a cost, where lower is better. Fact

mBO-i, a credit: the mean over the truth objects of the best IoU with any predicted region; one region may serve several objects. Fact
Foreground union IoU, a credit: the IoU of all predicted regions together with all truth objects together. Fact
Recall, a credit: the share of the truth objects' pixels that some region covers. Fact
Foreground ARI, a credit: the adjusted Rand index between the truth objects and the predicted regions over the truth objects' pixels, the predicted background counting as one more region; 0 is chance level and a value can be negative. Fact
Background false-positive rate, a cost: the share of the truth background's pixels that some region covers. Fact
Regions per image counts the regions that hold at least one pixel that is not void, and the readings also count the images where a degenerate cut ended the decoder's loop. Fact

Two more scores come from object detection, the way MaskCut and the methods after it report their results. Every region becomes a detection with a confidence, its ncut score, and every truth object counts as one class, object. Within each picture, from the most confident detection down, a detection is a hit when its IoU with a truth object that no more confident detection has taken reaches a threshold. Then all the detections of the panel are ranked by confidence, and going down that list, precision is the share of hits so far and recall the share of truth objects found so far. Average precision, AP, averages over the levels of recall from 0 to 1 the best precision reached at that recall or beyond. AP50 uses the threshold 0.5. AP100 averages AP over the ten thresholds 0.5, 0.55 and so on up to 0.95, allowing up to 100 detections per image, of which our decoder uses at most 5. Average recall, AR100, is the recall reached with up to 100 detections per image, averaged over the same ten thresholds. Both are computed over all the panel's images at once, and the readings give them in per cent. Fact

Where two regions overlap, the evaluator gives each pixel to the earlier region (the first slot wins), and every endpoint except mBO-i reads that map. A prediction with no region on a picture that holds a truth object gets no credit at all: 0 on every credit and 1, the worst value, on the background false-positive rate; without this zero-region rule an empty prediction would score a perfect foreground ARI on a picture with one object. Fact

A picture counts for the pixel scores only when it holds at least one retained truth object, an object with a pixel that is not void; COCO's crowd regions and pixels that two objects share are void. Of the panel's 1,000 images, 988 are defined in this sense. Every pixel score of a reading, from Hungarian IoU to the background false-positive rate, is a mean over those 988, and so are its interval and the regions per image; AP and AR pool all 1,000 images, and the count of degenerate cuts runs over all 1,000 too. Fact

Foreground ARI

ARI = (Σ_ij C(n_ij, 2) − E) / (½ (Σ_i C(a_i, 2) + Σ_j C(b_j, 2)) − E), E = Σ_i C(a_i, 2) · Σ_j C(b_j, 2) / C(n, 2), C(m, 2) = m (m − 1) / 2

n_ij: a table of counts: the pixels of truth object i that fall in predicted region j, the predicted background counting as one region
a_i, b_j: the table's row sums (pixels of object i) and column sums (truth-object pixels in region j)
n: the number of truth-object pixels of the picture, a whole number
C(m, 2): the number of pairs among m pixels
E: the pairs that a random grouping with the same row and column sums would put together, on average

Ten truth-object pixels belong to objects 1, 1, 1, 1, 2, 2, 2, 3, 3, 3, and the decoder put them in regions 1, 1, 2, 2, 2, 2, 0, 0, 3, 3, where 0 is the predicted background. Count them in a table, objects as rows and regions 0 to 3 as columns: object 1 is (0, 2, 2, 0), object 2 is (1, 0, 2, 0) and object 3 is (1, 0, 0, 2). Toy

Pairs kept together by both the truth and the prediction: four cells hold 2 pixels, one pair each, so 4. Pairs within a truth object: C(4, 2) + C(3, 2) + C(3, 2) = 6 + 3 + 3 = 12. Pairs within a region (columns of 2, 2, 4 and 2 pixels): 1 + 1 + 6 + 1 = 9. All pairs: C(10, 2) = 45. Expected by chance: 12 · 9 / 45 = 2.4; the most possible: (12 + 9) / 2 = 10.5. ARI = (4 − 2.4) / (10.5 − 2.4) = 1.6 / 8.1 = 16/81 = 0.1975, which the project's evaluator also gives. Toy

A prediction that groups the pixels exactly as the truth does scores 1, whatever numbers it gives its regions; a random grouping scores about 0. Fact

No gradient: like every score here it is computed after training, with labels. It rewards keeping each object's pixels together and apart from other objects, and it ignores the background pixels entirely, so a region that spills onto the background costs nothing here (the background false-positive rate sees that instead). Toy

Our rule moved this score in opposite directions from the two starts: run 1's foreground ARI fell 1.74 points below its start, and run 2's rose 1.94 above its own. Reading

Lab 6D scores the same drawing as Lab 6C with every secondary endpoint at once. Its buttons build the classic failures: one region over everything, two objects in one region, one object split in two, and no region at all. Toy

The other scores

What do union IoU, foreground ARI and the false-positive rate say about the same drawing?

Toy

What you are looking at: the drawing of Lab 6C, shared with it, painted by what each pixel is: covered and inside an object (blue over hatching), covered but on the background (blue only), inside an object but missed (hatching only). Beside it, every pixel endpoint of a reading with its value, the direction that is good, and the counts it is made of, with the table of pixels that foreground ARI counts. Try this: press "One region over everything", then "Two objects merged", and see which scores fall and which do not. Toy

Real model versus this toy: the real endpoints are computed at the photograph's own size, with crowd pixels and pixels of two objects left out as void, and averaged over the 988 defined images of the panel. This toy has 2,304 pixels, no void and one picture. Fact

Two rules keep the labels from steering anything. The validation panel is open, but every process that opens one of its labels writes a line in a ledger: the date, the script, the commit, the purpose and what was decided. Validation labels may inform the decoder's settings, metric checks, the recipe and the campaign's length, and never a choice between arms or of a checkpoint. A sealed test panel is opened once, by one evaluator run, after every prediction is sealed and the pre-registration is committed. Fact

Our contrastive study reads only the COCO v4 validation panel and never opens the sealed test panels, COCO v6 and VOC segmentation-train. The panel is much read: run 2's final reading was its 41st ledgered read. Fact

Decision

The primary endpoint of the contrastive study is Hungarian matched IoU, a mean over each picture's truth objects with a fixed denominator. Fact

Why

The early looks and the readings name it the primary endpoint, as it had been the project's primary score before the campaign made AP50 and FG-ARI its co-primaries; no note of the study gives a further reason for the choice. Fact

Evidence

Run 1 left it where its start was (+0.04 points, the interval from −0.58 to +0.63), while the secondary endpoints moved: the matched IoU nets a tighter foreground and a looser grouping to zero. Reading

What it does to training

Nothing: it is computed after training, with labels, once per checkpoint read, and no value of it selects anything. Fact

If you changed it

Matching one to one asks the decoder to separate instances: a region over two objects is credited for one of them, and a missed object costs its full share. A score that lets one region serve several objects, such as mBO-i, forgives merging. Compare the two in Lab 6D. Toy

Decision

Every reading reports, beside the primary endpoint, mBO-i, foreground union IoU, recall, foreground ARI, the background false-positive rate, AP100, regions per image and degenerate cuts. Reading

Why

Each endpoint has its direction, five credits and one cost, and each sees a different property of the same regions. Fact

Evidence

Against its start, run 2 holds the primary endpoint (+0.18 points, the interval from −0.59 to +0.92) while its background false-positive rate falls by 15.39 points, its union IoU rises by 3.79 and its foreground ARI by 1.94, with 2.58 points less recall. Reading

What it does to training

Nothing: the secondaries are read after training, like the primary endpoint, and none of them selects anything. Fact

If you changed it

With the primary endpoint alone, the clearest statement of the three runs would be invisible: the rule on the raw tokens changes what the decoder groups, most from the DINO start, and the head version leaves it where its start was. Reading

Decision

The validation labels are open, and every read of them is a ledger row; the sealed test panels stay closed in this study, and a test panel is opened once, at the end. Fact

Why

"The validation split is open but ledgered", and "the test split is opened once", after every prediction is sealed and the pre-registration is committed. Fact

Evidence

The panel has been read many times (run 2's final reading was its 41st row), and the sealed COCO test panel is smaller, 493 images, with intervals about 1.42 times wider. Fact

What it does to training

Nothing: no row of the study's ledger steered a run, each carrying the decision none, or "continue or stop" at an early look. Reading

If you changed it

A panel read again and again, with choices made after each read, slowly becomes part of the design, and its scores grow optimistic; the ledger makes every read visible, and the sealed panel keeps one honest read for the end. Toy

6.6 The linear probe

Assumes: a linear layer and softmax (Part 1) and patch tokens (Part 2). Gives: what a linear probe measures, what mIoU counts, and why the probe is a secondary reading.

The decoder asks whether the patches of one picture fall into object-shaped groups. A linear probe asks the same frozen network a different question: how well can one linear layer read COCO's categories off each token? It measures linear separability of categories, so two objects of one class carry one label, and the decoder's grouping into instances plays no part. Fact

The probe reads each image's final-block patch tokens at 224 pixels, 196 tokens of 384 numbers on a 14 by 14 grid. It standardises every dimension with the training tokens' mean and standard deviation, and one linear layer maps each token to 81 logits, one per class: COCO's 80 categories and background. Fact

Each token's label is the one that most of its patch's 256 pixels hold in COCO's instance masks, background counting as a label. The layer is fitted with Adam at a learning rate of 1e-3 on the cross-entropy over the 81 logits, for 20 epochs over a fixed subset of 2,000 COCO train2017 images in batches of 64 images (12,544 tokens), 640 steps in all. Nothing in the probe is tuned, and the backbone stays frozen: the gradient changes the layer's weights and biases and stops at the tokens. Fact

The probe is read once per checkpoint on the panel's 196,000 tokens. For each class, IoU counts the tokens labelled correctly as that class over the union of the tokens that are that class and the tokens predicted as it; mIoU is the mean of those IoUs over the classes present, so a rare class weighs as much as background. Token accuracy, aAcc, is the share of tokens labelled correctly. Fact

Background holds 70.1 per cent of the probe's training tokens, so token accuracy mostly reflects background, and mIoU is the informative column. Reading

The probe's logits, its loss and mIoU

z = W x + b; p_c = e^z_c / Σ_c' e^z_c'; L = −log p_y; ŷ = argmax_c z_c; IoU_c = TP_c / (true_c + predicted_c − TP_c); mIoU = the mean of IoU_c over the classes present

x: one standardised token: 384 numbers
W, b: the layer: 81 rows of 384 weights, and 81 biases
z, p: 81 logits and the 81 probabilities that softmax makes of them
y, ŷ: the token's true class and the predicted one
L: the cross-entropy of one token, a scalar; the fit averages it over a batch
TP_c, true_c, predicted_c: counts of tokens over the panel: correct as c, truly c, predicted as c

Take a token with two numbers, x = (2, 1), and three classes: background, dog and cat. Let the rows of W be (0, 0), (0.5, 0) and (0, 0.5), and b = (0.5, 0, 0). Then z = (0.5, 1, 0.5). Softmax raises e to each logit, e^0.5 = 1.6487 and e¹ = 2.7183, adds the three, 1.6487 + 2.7183 + 1.6487 = 6.0157, and divides: p = (0.2741, 0.4519, 0.2741). If the token is a dog, L = −log 0.4519 = 0.7944, and the prediction, the largest logit, is dog: correct. Toy

Now count ten tokens of a panel: 7 background, 2 dog, 1 cat. The probe gets every background token right, one dog right, calls the other dog background, and calls the cat a dog. Background: 7 correct over a union of 8, IoU 0.875. Dog: 1 correct over a union of 3, IoU 0.3333. Cat: 0 correct, IoU 0. mIoU = (0.875 + 0.3333 + 0) / 3 = 0.4028, while 8 of 10 tokens are right, an aAcc of 0.8. A probe that says background for every token gets aAcc 0.7 and mIoU 0.7 / 3 = 0.2333: token accuracy hides the miss of every object, and mIoU does not. Toy

The gradient of L on the logits is p minus the one-hot label, whose entry y_c is 1 for the token's class and 0 for the others: (0.2741, −0.5481, 0.2741) for the dog token. Each bias b_c gets exactly that number, and each row of W gets it times x, so the dog row's gradient is −0.5481 · (2, 1) = (−1.0963, −0.5481). Gradient descent subtracts every gradient times the learning rate: the dog row moves towards x, the other rows move away from it, and the dog's bias rises while the others fall. Toy

One step at a learning rate of 0.5 moves each logit z_c = W_c·x + b_c by −0.5 · (p_c − y_c) · (x·x + 1), where x·x = 2·2 + 1·1 = 5 comes from the row's step and the 1 from the bias's step. That is −3 times each logit's gradient: (−0.8222, +1.6444, −0.8222), so the logits go from (0.5, 1, 0.5) to (−0.3222, 2.6444, −0.3222). Softmax now gives p = (0.0467, 0.9067, 0.0467), and the token's loss falls from 0.7944 to −log 0.9067 = 0.0980. The token itself also receives a gradient, but the backbone is frozen, so it stops there. Toy

The real fit reaches, on its own training tokens, a token accuracy of 79.19 per cent for run 1's tokens; on the panel the same probe reads 75.98. The toy curve of Lab 6E shows the loss of a fit falling step by step. Reading

Why is the probe a secondary reading? It uses labels, after training, once per checkpoint with its own ledger row, and no value of it selects a checkpoint, a training value, the decoder or a panel. Its numbers also cannot be compared with published linear segmentation numbers, which use other datasets, label sets, resolutions and schedules. Fact

Decision

A linear probe of the frozen tokens is read beside the decoder, once per checkpoint, with labels, after training, with the decision none. Fact

Why

It asks whether the frozen token space differs in how linearly its patch tokens separate COCO's categories, measured independently of the frozen decoder: did our term change the token space while the decoder's primary endpoint stayed flat? Fact

Evidence

Run 1's tokens are more linearly separable than its start's by 1.35 mIoU points, where the decoder did not move; and the probe and the decoder rank the converted DINO and released SelfPatch in opposite orders. Reading

What it does to training

Nothing: the backbone is frozen, the probe's gradient stops at the tokens, and no value of it selects a checkpoint, a training value, the decoder or a panel. Fact

If you changed it

Fine-tuning the backbone with the labels would let them change the tokens being measured, so the result would describe the labels' training as much as the self-supervised tokens, closer to SelfPatch's supervised downstream numbers, which cannot be compared with this probe either. Toy

Lab 6E trains a probe in your browser on a toy: tokens with two numbers instead of 384, three classes instead of 81. Press Train and watch the layer's boundaries settle, the loss fall and the mIoU of a second toy picture rise; then try the probe that answers background everywhere. Toy

The linear probe

What does one linear layer on frozen tokens learn, and what does mIoU count?

Toy

What you are looking at: two toy pictures of 14 by 14 tokens, each token with two numbers. The probe trains on the first and is read on the second. The scatter shows the second picture's tokens and the three areas where the layer predicts background, dog or cat; the two grids show the true labels and the predicted ones, a cross marking each mistake; the table counts what mIoU counts. Click a token to follow its logits, its loss and its gradient. Toy

Real model versus this toy: the real probe reads 384 numbers per token and 81 classes, fits on 2,000 training images with Adam at 1e-3 for 640 steps, and is read once on the 1,000 panel images; this toy reads 2 numbers and 3 classes from one toy picture, with plain gradient descent, as often as you press the button. Fact

6.7 Our numbers so far, read with these definitions

Assumes: the endpoints of 6.5, the probe of 6.6 and the three runs (Part 5). Gives: every checkpoint's reading in one table, and how to read it.

Every number below is a reading: one validation read of the COCO v4 panel per checkpoint, from the reading notes of 10 and 11 October 2026 and the probe's reading. The six pixel scores, in per cent, and the regions per image are means over the panel's 988 defined images; AP100, in per cent, pools all 1,000 images; degenerate is a count of images out of all 1,000; and the probe's mIoU, in per cent, counts the 196,000 tokens of all 1,000 images. Each run has one seed, so an interval holds image sampling only, no seed spread, and no verdict of "different" or "equivalent" is declared. Reading

The final readings, per cent (↑ higher is better, ↓ lower is better)
checkpoint	Hungarian IoU ↑	mBO-i ↑	union IoU ↑	recall ↑	FG-ARI ↑	BG false positives ↓	AP100 ↑	regions	degenerate	probe mIoU ↑
converted DINO, the start of run 2	17.23	19.58	32.68	84.03	22.62	56.59	0.68	2.38	0	14.41
pilot: DINO with SelfPatch's window term	16.67	19.00	32.63	81.25	22.22	54.52	0.75	2.22	1	20.88
run 2: our rule from the DINO start	17.41	19.72	36.47	81.46	24.56	41.20	1.47	1.77	2	21.11
released SelfPatch, the start of runs 1 and 3	15.58	17.68	30.66	81.99	22.98	60.37	0.41	2.11	2	23.80
LC0 row twin: SelfPatch's loss on COCO	15.71	17.86	30.94	82.53	23.18	60.26	0.44	2.09	4	24.68
LC0 image twin	15.68	17.81	30.85	82.52	22.77	60.19	0.44	2.09	4	24.59
run 1: our rule from the released start	15.62	17.77	31.47	81.42	21.24	56.02	0.46	1.91	4	25.15
run 3: our rule through the projection head	15.95	18.02	30.73	81.85	22.81	60.29	0.46	2.13	7	25.31

Regions are a mean per image; degenerate counts the images, of the panel's 1,000, where a degenerate cut ended the decoder's loop. In every reading this happened after the first stage, since no export records a decoder failure, so each of these images kept the masks found before the cut. The decoder columns come from the three validation readings, the last column from the probe's reading. Reading

Read a row for one checkpoint's profile, and a column to compare checkpoints on one endpoint, with its arrow in mind: the background false-positive rate is the only column where lower is better. The rows come in two families, each start followed by what training made of it. Toy

On the primary endpoint the three runs sit where their starts sat. Run 2 is +0.18 points above the converted DINO (interval −0.59 to +0.92), run 1 +0.04 above the released start (−0.58 to +0.63), and run 3 +0.38 above it (−0.03 to +0.78): every interval holds zero. Reading

The secondary endpoints are where the rule shows. Run 2 claims far less background than its start, 15.39 points less of the false-positive rate (interval 13.69 to 17.01), with a higher union IoU (+3.79) and foreground ARI (+1.94), twice the AP100 and fewer regions per picture, at the price of 2.58 points of recall. Run 1 moved the same way on background (−4.35 points) with a lower foreground ARI (−1.74); run 3, the head version, moved none of them. Reading

The probe ranks the same tokens differently. It puts the converted DINO last, 14.41 mIoU, and the released SelfPatch model 9.39 points above it, while the decoder reads DINO above the released model; run 1's tokens became more linearly readable than its start's by 1.35 points where the decoder did not move; and run 2 reads 4.04 points below run 1 on the probe, while the decoder ranks it above. Reading

What the table does not say: that a run is better than its start, since every primary interval holds zero and each run has one seed; that these numbers compare with published ones, since the decoder keeps at most five regions and the probe uses its own labels, resolution and schedule; or that the panel is fresh, since it has been read many times. Reading

Two reference points help to place the numbers. Perfect masks under the five-region cap would reach a Hungarian IoU of 0.7957, as a fraction, on this panel, so every reading sits far below the ceiling; and a decoder that keeps at most five regions finds, among its guesses, an AR100 of 6.17 on the zero-step start, against 12.7 to 27.1 for detector pipelines that propose many more masks. Fact

An amendment of 12 October 2026 adds a second seed of the two strongest pairs, four more runs, so that a later reading can say whether a pattern repeats in both seeds or in one of the two, and nothing more than that. Fact

Glossary of this page

Affinity: In the decoder, the table of link weights between every pair of patches of one picture: a full link of one where the cosine of two keys is above τ, and a hair-thin link otherwise.
Average precision: A detection score, AP. The detections are matched to the truth objects picture by picture, a match counting when its IoU reaches a threshold; then, going down all the detections from the most confident, the best precision reached at each level of recall is averaged over the levels from 0 to 1. AP50 uses the threshold 0.5; AP100 averages over the ten thresholds from 0.5 to 0.95 and allows up to 100 detections per image. Our readings ignore categories and pool all the panel's images.
Average recall: The share of the truth objects that the detections find, averaged over the ten IoU thresholds from 0.5 to 0.95. AR100 allows up to 100 detections per image.
Background false-positive rate: The share of the truth background's pixels that some predicted region covers. It is a cost: lower is better.
Bias-free keys: A key rule that subtracts the last block's key bias from every key before the decoder compares them. It leaves any other shared part of the keys in place.
Bootstrap interval: How far a difference between two checkpoints could move with other photographs: the panel's images are drawn again with replacement, the same draw for both, and the difference is computed on each draw. Our readings keep the middle 95 % of 2,000 such draws, 500 for AP. It covers the sampling of images only.
Component: A group of grid cells that can all be reached from one another by steps up, down, left or right without leaving the group. The decoder's mask is the component of the kept side that holds the seed patch.
Decoder: The fixed recipe that turns a trained network's per-patch vectors into regions. Ours is MaskCut with frozen settings; it has no weights and learns nothing.
Defined image: A picture of the panel that holds at least one retained truth object. A reading's pixel scores, their intervals and its regions per image are means over the defined images only; AP and AR pool every image of the panel.
Degenerate cut: A normalised cut that cannot be trusted, because the second and third eigenvalues are equal or nearly so, or the spectrum is broken. The decoder then stops and keeps the masks it already has.
Degree: In the decoder's graph, the sum of one patch's link weights. The degrees sit on the diagonal of the matrix D.
Eigenvalue: The number λ that goes with one solution v of an equation such as (D − A) v = λ D v. The decoder reads the second-smallest, λ₂, and compares it with the third, λ₃.
Fiedler vector: The solution of the normalised cut's equation that goes with the second-smallest eigenvalue: one number per patch, similar for patches joined by strong links. The decoder splits the patches at its mean.
Foreground ARI: The adjusted Rand index over the truth objects' pixels. For every pair of these pixels it checks whether the prediction groups the two the way the truth does, scored so that a perfect grouping gets one and a random one about zero.
Foreground union IoU: The IoU of all predicted regions taken together with all truth objects taken together. It ignores which region covers which object.
Graph density: The share of all pairs of patches that the affinity links, each patch paired with itself included. At a density of one every patch is linked to every other, and the cut has nothing left to separate.
Heat map: A picture of a table of numbers in which each cell is coloured by its value.
Hungarian matched IoU: The primary endpoint of our readings: truth objects and predicted regions are paired one to one so that the summed IoU is as large as possible, and the score is the mean over the truth objects, an object without a region counting zero.
IoU: Intersection over union: the area two masks share divided by the area they cover together. It is one for a perfect match and zero when they do not overlap.
Key: One of the three vectors that an attention block makes from each token, the one that the other tokens' queries are compared with. The decoder reads the keys of the last block, 384 numbers per patch.
Key bias: The fixed vector that the last attention block adds to every key. The network ignores it, because softmax cancels a shift that all scores share, but a decoder that compares raw keys does not.
Key offset: The part that every key of a picture shares, such as the key bias plus the effect of the LayerNorm's shift. Attention cancels it, yet it tilts all the keys towards one direction and raises every cosine between them.
Key rule: The label-free way the decoder prepares keys before comparing them: raw keys, bias-free keys or per-image mean subtraction. It must be the same for every model compared; the owner chose mean subtraction.
Linear probe: One linear layer trained with labels on top of a frozen network's tokens. How well it reads the classes off the tokens measures how linearly the tokens separate them; the network itself does not change.
Logit: A raw score for one class, before softmax turns the scores of all classes into probabilities. The largest logit is the prediction.
MaskCut: A published label-free method that cuts several objects out of one picture by splitting a graph of patch features with a normalised cut, again and again. Our decoder follows its rules.
mBO-i: Mean best overlap over instances: the mean over the truth objects of each one's best IoU with any predicted region. One region may serve several objects.
Mean subtraction: The key rule our decoder uses: subtract the average of one picture's keys from each of its keys. It removes any part that every key shares, the key bias included.
mIoU: Mean intersection over union: for each class, the tokens correctly given that class over the union of the tokens that are that class and those predicted as it, averaged over the classes present.
ncut score: The decoder's confidence score for a mask: one minus the second eigenvalue of its cut, clipped to lie between zero and one.
Normalised cut: A way to split a graph in two so that the links broken between the two groups are weak compared with all the links inside each group.
Recall: The share of the truth objects' pixels that some predicted region covers. It is a credit: higher is better.
Region: A group of patches that the decoder puts together, its guess at one object. Our decoder makes at most five per picture.
Seed patch: The patch where the decoder's mask starts: the largest entry of the Fiedler vector on the side that is kept, or of its negative when the sides were swapped. MaskCut keeps the connected part of that side that contains it. It differs from the patch with the largest entry ignoring its sign only when the corners swap the sides although that patch lay on the chosen side.
Test panel: A separate set of photographs whose labels are sealed and opened only once, at the very end, by one evaluator run. Our contrastive study never opens one.
Truth mask: The outline that a person drew around one object of a photograph, used only after training: by the evaluator, after the predictions are sealed, and by the linear probe.
Zero-region rule: The evaluator's rule that a prediction with no region on a picture that holds an object gets no credit: zero on every credit and the worst value, one, on the background false-positive rate.