feat: per-image legend pipeline marker→class recovery (83 tests + CI)

[vickysharma-prog]

Per-image legend pipeline recover which marker → which class, from the screenshot alone

This PR adds the legend-based annotation recovery pipeline: for each historical survey screenshot it figures out, without the annotation CSV, which marker (shape + colour) maps to which class, and then classifies every dot baked into the aerial photo accordingly.

This is the mentor's core ask. Colour alone is ambiguous a red ● can be BRPE WBN while a red + is BRPE bird, and the convention differs per image. So the mapping is recovered per image and dots are separated by shape, which colour-only detection cannot do.

It also fixes the boundary-cutting bug: the old stage assumed the dialog was a full-height right panel and split each screenshot at ~50% width, discarding ~half the aerial and its birds. The dialog is actually a floating window that sits in a different place in every image.

---

New modules

File	What it does
src/legend.py	locate_dialog() finds the floating "Manual Point Count" dialog as a box anywhere in the frame (works on 14/14 test images). parse_screenshot / parse_legend extract each legend row's colour, best-effort shape, and a canonical 24×24 glyph template. attach_class_names() reads the class text + Count via Tesseract OCR, fuzzy-matched to the 98 species codes (corrects e.g. BAPE→BRPE). Scale-adaptive: works at thumbnail and full-res.
src/classify.py	detect_dots() finds aerial dots and splits merged clusters via distance transform (dense colonies otherwise undercount). assign_classes() matches colour first, then shape (template correlation) to separate same-colour classes. select_by_count() is a count-guided top-N precision filter.
scripts/run_legend.py	Runs the whole pipeline over a folder of screenshots.

Full automatic pipeline: parse_screenshot → attach_class_names → detect_dots(exclude=bbox) → assign_classes → select_by_count.

---

Figures

1. Dialog localization the boundary fix

The "Manual Point Count" dialog auto-located as a box (red outline) across six screenshots. It floats in a different place each time (top-right / right / bottom-right). The new locator keeps the entire aerial available for detection, instead of cutting it at 50%.

2. Per-image marker → class mapping

For each study image: the dialog + the recovered colour/shape → SPECIES category mapping, read entirely from the screenshot (shape analysis + OCR). Same-colour markers are kept distinct by shape. The mapping is recovered per image the shape→category convention differs between images.

3. Aerial dots classified by recovered class

Each aerial with every detected dot coloured by its recovered class (dialog region dimmed). Overlapping dots in dense colonies are split via distance transform. Note the dots span the whole frame, including regions the old stage used to cut off.

4. Recovered vs ground-truth counts

Left: image D, recovered count vs the dialog's legend count per class zero-count categories (empty/pbn/brood) correctly stay near zero; same-colour classes are separated. Right: total recovered vs dialog total across the 4 images, with recall % (A 36%, B 77%, C 90%, D 69%).

5. Why colour alone is not enough

A synthetic scene of same-colour, different-shape markers. Colour-only collapses them into 2 classes; colour + shape separates them into 5 correct classes the simplest illustration of why the shape step exists.

---

Config centralization

All of legend.py's tunables live in the legend: section of config.yaml one place for every module's config. The values were reconciled to the module's actual numbers; detection output is byte-for-byte unchanged (verified by snapshotting all 4 fixtures before/after).

Supporting changes

• decompose.py low-confidence safe-boundary fallback; exposes the 3 raw boundary candidates (used by figure 1).
• detect.py optional per-category count metadata.

Tests & CI

• tests/test_legend.py (40) + tests/test_classify.py synthetic-glyph shape classification, colour/template/OCR-parsing units, and real-screenshot integration over the 4 study images (skip cleanly if fixtures are absent).
• CI (tests.yml) rewritten as UTF-8, now installs scipy and runs all four test files (previously only test_decompose.py).
• Full suite: 143 passed.

Honest status

• Solid / generalizes: dialog localization (100% on 14 images), recall via cluster splitting, end-to-end automation, config centralization.
• Open limitations: count-OCR (~60–65% on ~10px digits) and within-colour aerial shape split (~60–70%, hardest on dense colonies like A). Both improved; both incremental from here.

[vickysharma-prog]

cc @jveitchmichaelis @bw4sz @henrykironde for review

[jveitchmichaelis]

I see that in many cases in (2) the marker shape is incorrect, or duplicated?

Can you share a full example of:

1. A processed dialog box
2. The class/marker assignments
3. The templates that are derived. Are they taken from the dialog or generated?
4. Locations of each template match in the image

[vickysharma-prog]

@jveitchmichaelis Full worked example attached (image D, raccoon 2011); I can generate the same for any image. Answering each point:

1. Processed dialog box panel (1):— The located dialog with each detected legend row marked.
2. Class/marker assignments panel (2):— Per row, colour / shape → class (count), read from the dialog itself (species+category via OCR, count from the Count column).
3. Are the templates taken from the dialog or generated?:— Taken from the dialog. For each legend row I segment the marker glyph from the dialog cell (_read_glyph), then crop, centre and scale-normalise it to a 24×24 template (canonical_template). Panel (3) shows [dialog marker crop] → [derived 24×24 template] for every class nothing is synthetic.
4. Location of each template match panel (4):— Every classified aerial dot at its location, coloured by assigned class (637 dots here).

On the incorrect / duplicated shapes correct, and worth clarifying the architecture:

• The shape field is a coarse best-effort label (circle/plus/star/ring/square/triangle). At these marker sizes an asterisk (✳) often collapses to "plus", which is why 3 red classes (BRPE bird +, empty ✳, chick ✳) all read "plus" the duplication you saw.
• But the shape name is not the primary matching signal. Within a colour group, dots are matched by cosine similarity of the 24×24 templates (each glyph is centred + scale-normalised first, so it's a single normalised comparison rather than a sliding convolution); the shape-name agreement is only a secondary +0.35 boost. In panel (3) the "duplicated-plus" templates are actually distinct (clean + vs asterisk-with-diagonals), so the template can still separate them
• Where it genuinely degrades is the aerial side: dots are ~5–11px; in dense colonies (~5px) the glyph carries little shape signal, so both the label and the similarity weaken. That's the honest hard case (felicity).
• I've since fixed the aerial shape gate (it was rejecting ~⅔ of aerial dots as "unknown" before classification); within-colour classification recall vs hand-read ground truth improved (northdeer 71→82%, gaillard 85→89%).