feat: difference-based detection + full self-contained pipeline (supersedes #3)

[vickysharma-prog]

Supersedes #3. PR #3 was the classification module on its own. This PR is the whole self-contained pipeline: find the dialog, register the clean original, recover dots by subtracting it, and classify each dot against the per-image legend. It works from the image alone. Survey counts are never an input; the CSV is only for checking the results.

Status: detection 8.40× → 1.24× median · alignment 96.7% · classification agreement 0.26 → 0.36 · 166 tests passing.

Both halves improved, and they are not equally far along. Detection is close to done: over-detection drops from 8.40× to 1.24× and the worst band, sparse, drops from 63.51× to 2.13×. Classification improved but is still weak: agreement rises from 0.26 to 0.36, better on 27 of 41 frames and worse on 14. The rest of this describes how, and what is still wrong.

---

The pipeline: two questions per image

Every screenshot raises two separate questions, each handled by different code:

	Question	Modules
Detection	where are the dots, and how many?	align.py → subtract.py
Classification	which class is each dot?	legend.py → classify.py

legend.py (find dialog + parse legend)  →  align.py (register original)
        →  subtract.py (dots = image difference)  →  classify.py (assign class)

---

Why 63 images, not 4

The old four-image set is how the colour thresholds overfitted, and its ground truth was the wrong number. Reading the counting tool's own Total Count field settled it: the dot count is category_sum (the sum of the per-dot-type columns), not total_birds. total_birds excludes chicks, so it undercounts by up to 57% (mean error 0.25 vs 14.75). The benchmark was rebuilt as 63 stratified pairs: 7 years × 3 density bands × 3 images, across 40 colonies and 16 regions, with dot counts from 7 to 2,037.

Density band here means how many annotation dots are on the image, not how many colours or classes. The cut-offs are in scripts/build_benchmark.py: sparse 5–50 dots, medium 51–300, dense 301 and above. Sampling by band keeps the dense tail in the benchmark, since that is where detection has always been weakest, and it stops the median hiding a failure that only shows up at one end.

---

Detection: from colour thresholds to image difference

The clean high-res original still exists, so we can recover annotations from what changed between the two images instead of what falls in a colour range. A fixed colour range overfits one survey year and over-detects on the next. align.py registers the original onto the screenshot (SIFT + RANSAC, and it returns a failure rather than a bad warp); subtract.py subtracts the two.

Same frame, two methods. Colour thresholds mark every leaf and water glint (3,144 detections against a true count of 64); subtraction returns 76:

Detection ratio by band (detected ÷ true, median; closer to 1.0 is better):

Band	OLD colour	NEW subtraction
sparse	63.51×	2.13×
medium	9.15×	1.24×
dense	3.56×	1.14×
overall median	8.40×	1.24×
symmetric |log₂|	3.07	0.53

Dense colonies are the hardest case, and they are now close to solved (1,149 found vs 1,050 true). A distance transform splits the merged blob back into single markers:

The sparse-frame fix

On near-empty frames, the leftover noise (water glint, mudflat texture, and the grey box behind a label) outnumbers the real dots. That noise distorted the marker-size estimate the detector uses to split merged dots. Real dots are coloured; the noise is not. So the detector now drops any blob whose colour is too weak (low saturation) before it estimates size. This needs no legend, so it still works when the legend fails to parse. Sparse over-detection fell from 2.96× to 2.13×, medium from 1.59× to 1.24×, and dense stayed the same.

A finding: the ground truth itself needs checking

One dense frame looked like a bad regression (450 down to 9). I checked it by eye, not by count:

It is a "No photo coverage for this area" frame: no dots on it at all, just survey polygons and labels. Its category_sum of 450 is the estimate written in the text box. The old detector scored 288 by counting the polygon lines; the saturation floor correctly returns near zero. So this is a ground-truth error, not a regression. It is a good reminder to check frames by eye, not by count alone.

---

Classification: which class each dot is

classify.py separates classes that share a colour (BRPE wbn vs bird vs chick, all red). This is the hard part. Three changes, all measured:

1. Colour → each dot is matched to the colours the dialog actually uses in that image, in Lab space, not to fixed global bins. A yellow dot whose colour has shifted is matched to the nearest colour a class uses, not labelled orange when no class is orange.
2. Background removal → the glyph is masked by its own hue before templating, so the grey around the marker is not included in the match.
3. Shape → NCC (TM_CCOEFF_NORMED) on the actual patch. An ablation chose NCC over binary-mask cosine and also removed the +0.35 shape-name boost, which was lowering accuracy.

Two measurements, and they disagree. Stating both.

Legend self-recovery: D 55→76%, A 56→83%. Each legend glyph is degraded to aerial scale and pushed back through matching to see whether it recovers its own class. Counts cannot game it, but it flatters the method: the template and the test glyph come from the same pixels.

On real aerial dots the same method scores lower. Comparing per-class assigned counts against the counts read from the dialog, over the 41 cached frames whose legend yields at least two classes, with detection held constant so only matching varies:

Band	Previous matching	Current matching
sparse (7 frames)	0.223	0.485
medium (15)	0.273	0.352
dense (13)	0.266	0.382
overall mean	0.263	0.357
median	0.250	0.339

Better on 27 frames, worse on 14. So the changes help, and 0.36 is still low. Reproduce with the env toggles in classify.py: COLOR_ANCHOR=0 BG_REMOVAL=0 SHAPE_MATCH=cosine SHAPE_BOOST=1 runs the previous configuration. Per-frame numbers are in results/classify_ab/.

Read the two measurements together: self-recovery says 76–83%, real aerial dots say 36%. The gap is the method being tested against glyphs cut from the same pixels as its own templates. Both numbers are here so neither is mistaken for the other.

The figure below is generated from the live path by scripts/make_classify_figure.py, so it always shows current behaviour. Per class it gives the dialog marker, the 24×24 template cut from it, and six aerial patches drawn at random from everything assigned to that class, with a white ring on the matched point. Random rather than best-scoring, so the rows are typical rather than the strongest.

The frame is image D, the same one shown in the earlier review, so the rows can be compared directly. It is a legacy fixture with no clean original beside it, so detection here is the colour path, not subtraction. That holds detection constant between the two figures and leaves the matching changes as the only difference.

The top rows improved. BRPE chick (red plus) now matches pluses; it previously matched circles. LAGU bird returns clean green pluses and BRPE bird clean red pluses, where both were mixed before. Per class: BRPE wbn 169 expected / 181 assigned, BRPE chick 98 / 96, BRPE bird 116 / 86, LAGU bird 156 / 94.

The bottom three rows still fail, and they are the rare classes: BRPE empty (15 dots), BRPE brood (8) and BRPE pbn (4). Their patches are red map text and bare vegetation, so these are detection false positives reaching the matcher, not matching errors. The colour detector proposes them; subtraction removes most of this class of error, but image D has no clean original to subtract.

A specific failure worth naming. Where two classes in one colour also carry the same shape name, the templates score almost level and the winner is close to arbitrary. On one benchmark frame WHIB site (232 expected) and WHIB bird (86) are both parsed as circles, scoring 0.548 against 0.540 on the same dot; the pipeline recovered group sizes of 89 and 254, so the split is roughly right and the labels are swapped. Counts from the legend cap the group sizes but do not reassign, so this is not fixed by select_by_count.

The two halves are joined

classify.detect_dots_subtract takes the dot positions from subtraction and reads colour and shape at each one, using the same code the colour detector uses. assign_classes then sees the same input either way, so the class matching did not change. If alignment is refused, it falls back to the colour detector rather than returning nothing.

Detection stays a pure position finder here: the legend palette is not used to reject ink, so the subtraction path returns exactly what dot_candidates found. A test asserts that count parity, which keeps an unmeasured colour filter from creeping in.

---

What changed since PR #3

	PR #3 (baseline)	This PR
Question	which class	+ where / how many
Modules	legend.py · classify.py	+ align.py · subtract.py
Detection	colour thresholds (8.40× over)	subtraction (1.24×)
Class matching	global bins + binary-mask shape	Lab palette + NCC + bg removal
Measured on	4 images · total_birds (wrong GT)	63 pairs · category_sum (verified)
Tests	143	166

Repo map: what each module does (and the legacy ones)

Live pipeline: legend.py → align.py → subtract.py → classify.py

• align.py (new): register the screenshot against the clean original; a quality gate rejects bad warps.
• subtract.py (new): annotation ink as image difference; dot_candidates turns the ink into dots.
• classify.py: Lab colour anchoring, then NCC shape matching within a colour. detect_dots_subtract is the primary detector; detect_dots (colour) is the fallback.
• legend.py: find the dialog as a box, parse each row into (marker, colour, shape, template, class name, count).

Legacy (kept for reference and CI, not used by the live pipeline):

• decompose.py: the original Stage-1 splitter. legend.locate_dialog replaced it; the old ~50%-width split cut off half the aerial, which was the boundary bug raised in review.
• detect.py: the original Stage-2 colour detector. It needs CSV counts, which breaks the "image alone" rule. The self-contained colour baseline and fallback is classify.detect_dots, not this.

---

Limitations & next steps

• Sparse frames still over-detect at about 2.13×. Two separate things cause this. Tree and vegetation texture is one; a colour filter from the legend should help here, since leaves are not marker-coloured. Red label text and transect lines are the other; they are the same red as the markers, so colour cannot separate them. That second one is a limit on a handful of frames, and I ruled out depth, peak-count, and elongation as ways to tell them apart.
• Colour is not used to reject ink during detection. Dropping any blob whose colour is not in the legend palette would clean up sparse frames, but it costs too much elsewhere: on one sparse frame it cut 129 detections to 1 against a true count of 9, and to 0 whenever the legend failed to parse. It stays out until it can be made safe.
• Classification is the weak half. 0.36 agreement on real aerial dots against 76–83% self-recovery, and it went down on 14 of 41 frames. Two classes sharing a colour and a shape name are separated by near-level template scores, which is close to arbitrary. Detection is close to done; this is not.
• Ground truth is counts, not locations. To measure per-dot placement accuracy, and to improve classification against a real target rather than a proxy, we would need a small hand-labelled set. That is the prerequisite for further classification work.

Reproduce

python scripts/build_manifest.py && python scripts/build_benchmark.py --per-cell 3
python scripts/eval_detection.py     # the gate: old vs new vs category_sum
python scripts/eval_alignment.py     # registration success rate
pytest tests/ -q                     # 166 passing

Benchmark selection is in data/cache/benchmark.csv; image pairs cache under data/fixtures/pairs/ (gitignored, regenerable).

Docs

The README described the pipeline this PR replaces: 70.8% detection scored against total_birds, the survey CSV drawn as a pipeline input, and OCR listed under "what didn't work". Rewritten around the current modules and the measured numbers. The old figures are kept rather than deleted, in a clearly scoped earlier prototype section that states what they were measured on and against, because stages 3 to 7 still live in the notebook.

Two files needed restoring: 02ec833 had deleted docs/learnings.md and docs/training_analysis.md while the README linked to them 7 times, so merging would have removed both from main and left dead links.

results/figures/fig2_marker_to_class.png was stale, showing many unread class names where the current parser resolves 80 of 82 rows. It now comes from scripts/make_legend_figure.py, which runs the live path, matching how fig_classify.jpg is produced. Every figure in the README ships from a generator that runs current code.

[vickysharma-prog]

Hi @jveitchmichaelis @bw4sz @ethanwhite @henrykironde @cronosnull, opening this up for review when you get a chance 🙏

This is the full self-contained pipeline (supersedes #3): it works from the image alone, survey counts are validation only.

Detection (new: align.py + subtract.py). Register the clean high-res original (SIFT+RANSAC, 96.7%, refuses bad warps) and recover dots by image difference instead of colour thresholds. Detection error 8.40× → 1.24× median on a 63-pair stratified benchmark (vs category_sum); dense colonies effectively solved (1.14×). @cronosnull, this is the subtraction idea from discussion #6, now measured.

Classification (classify.py, legend.py). Per-image Lab colour anchoring to the dialog's own palette + glyph background removal + NCC shape matching. Within-colour separability D 55→76%, A 56→83%. @jveitchmichaelis, this implements the three asks from our thread.

Ground truth. Dot count is category_sum (verified against the tool's own Total Count), not total_birds; benchmark rebuilt as 63 stratified pairs (7 years × 3 density bands).

164 tests passing. Figures + full walkthrough are in the PR description. The one remaining soft spot is sparse frames (~2.13×), detailed at the bottom. Feedback very welcome!

[vickysharma-prog]

Merging this so the following work builds on top of it, and handling the remaining review points as separate PRs rather than growing this one further.

Fixed vs derived templates

Measured this rather than guessing. A fixed template library only works if the shape is named first, so I checked how often naming is right: 29 of 63 legend rows (46%) are named correctly, and the same 46% comes out of two independent namers, the contour heuristic and NCC against ideal glyphs. A further 15 rows are squares that no ideal glyph represents. A fixed library inherits that ceiling; templates cut from each image's own dialog skip naming entirely. So the pipeline stays on derived templates, and shape naming is the thing worth fixing.

Where the project stands

Detection is done. The pipeline works from the image alone: find the dialog, register the clean original, recover dots by subtracting it. Over-detection is down from 8.40x to 1.24x median, registration succeeds on 96.7% of pairs at 0.38px median error, 166 tests pass. Survey counts are never an input.

Classification is the weak half. Per-class count agreement went from 0.26 to 0.36 across 41 frames, so the colour palette, background removal and NCC changes did help, but 0.36 is low. Same-colour sub-classes are where it breaks, which is what the bottom rows of the figure show.

**Next, in order **

1. Shape naming, since that is what caps the fixed-template route and also separates same-colour classes.
2. Hand-label roughly 100 aerial dots. Classification is currently scored against counts, and counts cannot distinguish a correct labelling from one where two classes swapped labels wholesale. On one frame WHIB site and WHIB bird scored 0.548 against 0.540 on the same dot and the labels swapped, while the group sizes stayed close to right. Until there is per-dot truth there is no real target to optimise against.
3. The downstream stages, which still live in the notebook: map dots onto the original photograph, validate, export the DeepForest CSV, and train.

Closing #3, since its commits are contained in this branch.