[Correctness] Preprint SPLiT-seq whitelist filters are bypassed by Hamming fast path

[rob-p]

@elanfisher

Summary

The seqproc/antisequence versions used for the preprint do not apply the three
SPLiT-seq filter_within_dist whitelist filters. All three barcode lengths
(BC3=8, BC2=8, BC1=6) select antisequence's precomputed Hamming lookup. That
fast path does not populate the generic pattern attribute on which seqproc's
filter retention predicate depends. The result is that the predicate evaluates
to “retain” for both whitelist hits and misses.

This is reproducible with the exact preprint checkout. It materially affects
the SPLiT-seq recovery/accuracy evidence and requires regenerated paper results.

Affected versions

• seqproc: 03cb76335887ccc1f5191f458a4837debe424a31
• antisequence, pinned by seqproc's Cargo.lock:
  cd20194c9151b8679bba56411b231b88ac51526f
• analysis/configuration snapshot used around the preprint:
  4005c74dd18e340352cc6b35421e33de0763d786
• rebuilt preprint binary SHA-256:
  7a2f94542f6a7096acaa4049c611a1862d7c2d57c7d465e9cd0be82d6194fa32

Code-path diagnosis

1. Seqproc loads a filter whitelist as
   Patterns::from_strs(contents).with_pattern_name(FILTER):
   https://github.com/COMBINE-lab/seqproc/blob/03cb76335887ccc1f5191f458a4837debe424a31/src/processors/mod.rs#L173-L192

2. FilterWithinDist runs MatchAnyOp and then retains according to
   not(attr_exists(<label>.FILTER)):
   https://github.com/COMBINE-lab/seqproc/blob/03cb76335887ccc1f5191f458a4837debe424a31/src/geometry/interpret.rs#L655-L668

3. For all-literal patterns of uniform length <=8 and mismatch threshold <=2,
   antisequence enables HammingLookup. Its hit path hard-codes only sub and
   ambig; its miss path does the same. Neither path consults
   Patterns::pattern_name() or writes FILTER:
   https://github.com/COMBINE-lab/ANTISEQUENCE/blob/cd20194c9151b8679bba56411b231b88ac51526f/src/graph/ops/match_any_op.rs#L375-L424

4. Therefore <label>.FILTER is absent after every fast-path lookup.
   attr_exists(...) is false, its negation is true, and every read survives.

This differs from the slower general matching path, which does populate the
configured pattern attribute. It is consequently a fast/slow-path semantic
divergence, not merely a performance issue.

Reproduction

I rebuilt the exact seqproc checkout with its locked dependencies and ran
cumulative SPLiT-seq geometries on the first 1,000,000 paired fragments from
SRR6750041_10M_R1.fastq and SRR6750041_10M_R2.fastq. The geometry used UMI
length 8 so that the previously identified UMI-10 coordinate problem did not
confound this test.

Input provenance:

Input	SHA-256
1M R1 prefix	00a33679a953044eac49efe892abca80331944e8e0349c3bdd94c6ad04e9d747
1M R2 prefix	d4e5ec68f98306af05620e88c7b7cb2b914e3674f115ef5e7943536d87afe3cb
BC2/3 whitelist	1a1d246c0b0017e9cd246775be80164ff0c6ec80aa8703867398e0422054c641
BC1 whitelist	d62002ac8469d79d45e8acda8f9b0fd3f194bec7e93345cb15a6ee1f3053279a

Observed cumulative retention:

Last completed stage	Preprint binary	Saved optimization-pass build	Preprint-only
BC3 whitelist	1,000,000	914,570	85,430
Linker 1 edit anchor	919,739	882,558	37,181
BC2 whitelist	919,669	848,916	70,753
Linker 2 edit anchor	841,998	792,284	49,714
BC1 whitelist	841,244	760,445	80,799

The decisive observation is the first row: the preprint binary retains all one
million pairs after BC3, including sequences with no whitelist neighbor within
distance 1. The subsequent preprint losses arise from linker matching, not the
first whitelist.

Conditioning each stage on the pair set shared immediately before it attributes
85,430 additional pairs to BC3, 33,590 to BC2, and 31,490 to BC1. The final
accepted set from the saved optimization build is a strict subset of the
preprint accepted set: 760,445 shared pairs plus 80,799 preprint-only pairs and
zero optimization-build-only pairs.

The 760,445 result should not be treated as ground truth. The saved build was
an intermediate uncommitted optimization state and also exhibited edit-anchor
false negatives. For example, SRR6750041.100042 has independently calculated
semi-global linker-1 edit distance 5 (threshold 6); the preprint accepts it and
that intermediate build rejects it. This issue is specifically about the
proven preprint whitelist-filter bypass.

Impact on the preprint analysis

• The SPLiT-seq barcode whitelist constraints described by the geometry were
  not enforced in the preprint executable.
• Reported SPLiT-seq recovery, nominal precision/recall, pairwise concordance,
  discordant-read results, runtime, and peak RSS may all change after repair.
• Output barcodes can include sequences farther than the declared distance from
  every whitelist entry.
• The plausible-looking final count (841,244) masks the problem because the two
  linker filters still reject reads.
• Other protocols using filter_within_dist with short literal patterns may be
  affected and should be audited.

Proposed correction

Avoid using attribute presence as an implicit match-status channel. A robust
contract would return or store explicit fields such as matched, distance,
pattern_idx, and ambiguous, with identical semantics in lookup and general
matching implementations. filter_within_dist should retain reads based on
matched == true; ambiguity/tie policy should be an explicit, separate choice.

The current working antisequence source retains the optimized lookup and repairs
the bypass: its hit and miss paths now propagate the configured pattern and
per-pattern attributes, matching the general path's ordering. In the existing
same-label filter transform, remapping clears temporary attributes on a hit,
whereas a miss receives the FILTER sentinel after remapping; seqproc therefore
keeps hits with not(attr_exists(FILTER)) and rejects misses. Focused regression
tests should preserve this fast/slow equivalence. An explicit boolean match
result would still make this contract less dependent on operation ordering in a
future API revision.

A fresh build of the current seqproc and antisequence working sources retains
914,570 of the same 1,000,000 pairs at the isolated BC3 stage, matching the
saved optimization build rather than the preprint's 1,000,000-pair bypass. A
new 8-versus-9-base dispatch-boundary test passes, as do all 300 antisequence
library tests, seqproc's focused end-to-end filter test, and its existing
differential CLI suite. These tests are currently working-tree changes and
should accompany the eventual fix commit/PR.

Acceptance criteria

• Exact hit, one-mismatch hit, miss, equal-distance tie, and non-ACGT cases
  have documented behavior.
• Differential tests show identical results for the fast Hamming lookup and
  a brute-force reference matcher.
• Tests cover 6 bp, 8 bp, and 9 bp patterns, crossing the optimized-path
  boundary.
• filter_within_dist keeps within-threshold reads and rejects
  outside-threshold reads without relying on attribute existence.
• Synthetic paired SPLiT-seq reads validate all three barcode stages and
  both linker stages against known truth.
• The 1M-pair stage diagnostic is rerun from a committed build; output-set
  provenance and checksums are archived.
• SPLiT-seq recovery, concordance, downstream results, runtime, and RSS are
  regenerated together from the repaired accepted-set artifact.
• Other paper geometries using short-pattern filter_within_dist are
  audited for the same bypass.