bug: extraction-engine audit — 20+ verified defects (table alignment, encoding, OCR, config) and a redaction PII leak across ~12 output fields

[tobocop2]

Description

This started with one document — a car owner's manual (2011 Ford Crown Victoria) whose fuse-chart tables extracted with their rows destroyed (#1213). That turned out to be a pdf_oxide-level defect, but investigating it prompted a systematic audit of the whole extraction engine to find out what else was wrong. This issue is the result of that audit.

10 review rounds covering every subsystem (PDF text/reading order, table detection, OCR, office formats, HTML/email/archives, chunking, output composition, caching, MIME routing, config plumbing, FFI/API), each round independently re-verified, and the highest-severity findings reproduced at runtime. The handful of table defects that started it turned out to be instances of engine-wide classes; a mechanical field-by-field enumeration of ExtractedDocument then showed the redaction PII leak spans ~12 output fields, not the two originally found.

Scope: this issue tracks the xberg-side defects, which are fixable in this repo and land as one PR. The original trigger — bordered stroke-width-ruled tables (the fuse charts) — is a pdf_oxide defect and is tracked separately in #1213, blocked on yfedoseev/pdf_oxide#814; it is not fixed here.

Evidence tags

• [E] — reproduced at runtime with a minimal input. Actual output shown; the runnable test is in the reproduction harness (see Steps to reproduce).
• [R] — confirmed by two independent code reads (one reviewer found it, a second tried and failed to refute it); not yet runtime-driven, usually because it needs an OCR runtime, an embedding model, or a binary Office fixture.

Runtime testing also removed two findings that passed code review (a claimed HTML→XML misroute — both probes detect text/html; and colspan header misalignment — simple colspan headers align, only rowspan misaligns), which is why the tags matter.

Steps to reproduce

Reproduction harness (11 defects driven at runtime as #[ignore] integration tests) and the reference PDF are in this gist:

• https://gist.github.com/tobocop2/0d30905e2375e09b153ce50296316f91 — audit_reproductions.rs, cv-manual.pdf

Each [E] finding below states the exact input and the actual output observed.

Relevant files and configuration

Default ExtractionConfig unless a finding names a config field. All code locations are crates/xberg/src/....

---

Security

1. Redaction leaves most output fields unredacted — PII leak [E]

Redaction rewrites an allowlist of ~7 fields and leaves ~12 text-bearing fields carrying the original PII, while redaction_report reports success. A mechanical walk of ExtractedDocument found the stale fields: pages[].content, tables[] cells, elements[], uris[], structured_output, form_fields[], image captions/descriptions, annotations[], revisions[], archive children[], document (structure tree), and metadata.

Actual (HTML with alice@example.com in a paragraph and a table, redaction on):

• content → Contact: [REDACTED] for details.
• tables[0].cells → ["Alice", "alice@example.com"]
• redaction_report → success

Expected: every text-bearing field masked, or the report reflects what was actually redacted.

Where: text/redaction/engine.rs:19-182 (rewrites only content, formatted_content, chunks, entities.text, summary, translation, page_classification labels — no branch for any other field).

Sharpest instances (all serialize on REST/FFI/bindings/MCP):

• elements[] under result_format = ElementBased [E] — renders from a snapshot cloned before any processing (core/pipeline/mod.rs:141-149). Actual: content redacted, elements[].text returns the raw email, report claims success.
• structured_output [R] — LLM schema extraction pulls {name, email, ssn} into concentrated form; no mutator walks it (types/extraction.rs:260-267).
• form_fields[] [R] — filled tax/medical/loan form values (value/default_value/tooltip); always-present Vec, never touched (types/form_field.rs:44-80).
• uris[] [R] — mailto: and email URLs returned verbatim even though Email is an always-on redaction category (types/uri.rs:15-26).

Fix: one exhaustive field-walk in the Late redaction stage, not per-field patches.

---

Data corruption

2. Merged table cells put values in the wrong column [E] (HTML, DOCX)

Actual (HTML):

<tr><td rowspan="2">Alpha</td><td>Alice</td><td>10</td></tr>
<tr><td>Bob</td><td>20</td></tr>

Row 2 → ["Bob", "20", ""] — "Bob" lands in the Group column.

Expected: ["", "Bob", "20"] — the rowspan reserves column 1.

Where: extraction/html/structure.rs:690-735, html.rs:141,425, and the DOCX grid builder which ignored gridSpan (extractors/docx.rs:163-181) — this is also the DOCX bug reported in #1212.

PPTX and XLSX checked and NOT affected. Both were flagged on the same suspicion (their table code never reads span attributes), but live extraction shows they already align correctly, so this finding does not apply to them:

• PPTX: OOXML keeps an hMerge/vMerge placeholder <a:tc> for each covered column, so iterating all cells preserves the grid. A gridSpan=2 header extracts as ["Fuse", "", "Circuit"] with the data rows lined up beneath.
• XLSX: calamine's worksheet_range returns a dense grid — the merge origin holds the value and covered cells are in-place blanks — so ["Fuse", "", "Circuit"] again, aligned.

Both produce the same aligned blank-continuation the HTML/DOCX fix settles on; regression tests were added to lock this in.

The PDF cell-text superscript reorder ("Revenue¹" → "¹ Revenue") and RTL reversal are a separate PDF-backend concern tracked in #1213, not part of this finding.

3. Encoding handled per-format — same bytes decode differently

Actual / Expected:

• .txt with Latin-1 bytes [E]: café naïve résumé → caf� na�ve r�sum�. Same bytes as .csv decode correctly. Expected: café naïve résumé. (extractors/text.rs:88 — from_utf8_lossy, no detection.)
• XML declaring ISO-8859-1 [E]: <name>café</name> → caf�. Expected: honor the declaration. (extractors/xml.rs:46-125.)
• Excel CSV with UTF-8 BOM [E]: first header → \u{FEFF}Name. Expected: Name. (csv.rs:278-281.)
• Chinese .doc [R]: a ">25% CJK must be mis-flagged" heuristic re-decodes correct UTF-16 as CP1252 → mojibake (extraction/doc/mod.rs:302-322).
• Every PDF [R]: control char U+0003 after a letter → "ft" (a one-document hack), so of␃ce → offtce, corrupting the more common fi/fl ligatures (pdf/text.rs:77-91).
• Phone photos [R]: EXIF orientation stored but never applied → sideways OCR, transposed dimensions (extraction/image.rs:410-422).

4. Chunk/entity byte offsets point at the wrong text after post-processing [E]

Offsets are computed, then later stages mutate content without resyncing.

Actual (long markdown table, RepeatHeader chunking): 6 of 8 chunks report a 797-byte span for 823 bytes of content — off by exactly the injected header, so source[byte_start..byte_end] returns wrong text.

Expected: byte_end - byte_start == content.len() for every chunk.

Where: chunking/core.rs:178,384. Same class [R]: redaction preserve_offsets (default) never adjusts byte_start and shifts byte_end by the wrong delta (redaction/engine.rs:74-131); translation rewrites chunk.content to the target language while offsets stay source (translation/llm.rs:65-108); NFC changes byte lengths under the chunks (pipeline/mod.rs:239); NER entities[].start/end never resynced and entity.text masked without offset bookkeeping (ner.rs:70, entity.rs:16-21); semantic groups get interpolated offsets (chunking/semantic/merge.rs:137-146).

---

Silent failures

5. Encrypted PDFs return empty as success — passwords is dead code [E]

Actual (encrypted PDF with real text, correct passwords: ["secret123"]): Ok, content = "" — identical with and without the password. No error.

Expected: decrypt and extract, or return an error.

Where: PdfConfig.passwords has zero read sites (extractors/pdf/extraction.rs:50, pdf/oxide/mod.rs:63-68).

6. Token reduction computed then discarded for Markdown/HTML output [E]

Actual (343-char text, aggressive reduction): plain output 63 chars; markdown output (default) 344 chars — full size, work wasted.

Expected: reduction applies regardless of output format.

Where: apply_output_format overwrites content with pre-reduction rendered text (core/pipeline/mod.rs:235,273, format.rs:44).

7. OCR returns garbage as success [R]

Actual / Expected (each from code):

• OCR backend errors → already-failed native text returned as success; a scanned invoice with no tessdata yields empty content, no error (extractors/pdf/mod.rs:428-434). Expected: an error.
• All backends below quality threshold → best garbage returned with a log line (ocr.rs:1587-1605).
• OCR returns empty on a mixed page → native text erased by the empty string (ocr.rs:648-651).
• OCR always renders at 150 DPI; configured DPI ignored (ocr.rs:426, render.rs:25-37).
• Non-Latin scan → OCR'd as English (script detection disabled), gibberish accepted (ocr/tesseract_backend.rs:80).

8. Real tables rejected by shape heuristics [R]

Actual / Expected (each from code):

• 40-row Account | Amount ledger → rejected (>20 rows, ≤3 cols, >80% filled: pdf/table_reconstruct.rs:476-487). Expected: kept.
• First | Last | City roster → rejected (rows >80% alphabetic: table_reconstruct.rs:604-637).
• 16-column quarterly table → rejected outright (hard cap 12/15 cols, no truncation: oxide/table.rs:240).
• Page with a ruled + an unruled table → the ruled hit suppresses all other detection for the whole page; the unruled one is lost (extractors/pdf/extraction.rs:108-124).
• More than 20 candidate regions → bottom-of-page tables dropped (oxide/table.rs:381-388).

Status after investigation. The dense ledger case is fixed: the density guard now keeps a dense grid whose cells are short values (gated by a words-per-cell prose check), recovering 4 real dense reference tables across the test corpus with no regressions. The all-text roster case was investigated and deliberately left as-is: a roster and a scanned two-column prose page are indistinguishable to the row-coherence check, and relaxing the alpha-ratio guard reintroduced hundreds of false positives (bibliographies, scanned prose), so the conservative guard is kept — precision over recall for the ambiguous all-text case. The column cap, ruled-suppresses-unruled, and >20-region items are separate and remain open.

9. Silent content drops, non-PDF [R]

• PPTX SmartArt/charts/diagrams → None, all their text vanishes (extraction/pptx/parser.rs:164-182).
• PPTX slides ordered by filename, not sldIdLst → reordered deck extracts in wrong order (parser.rs:504).
• Archives: non-UTF-8 members dropped, one corrupt header aborts the whole archive, text capture allowlisted to 9 extensions, member order nondeterministic (archive/zip.rs, tar.rs, mod.rs:52, archive.rs:88).
• Email: unsniffable attachments discarded with no warning (email.rs:393-407).
• .doc: non-BMP characters (emoji, rare CJK) dropped via unpaired surrogates (doc/mod.rs:294-299).

---

Correctness

10. Same input, different result per entry point [R]

• DOCX misnamed report.pdf → extract_file trusts the extension (PDF extractor, garbage); extract_bytes sniffs magic (DOCX, correct) (core/mime.rs:635-640 vs :758-772).
• foo.py with #!/usr/bin/env node → path entry says Python, bytes entry says JavaScript (extractors/code.rs:107-114 vs :163-201).
• Sync vs async pipelines run language detection on opposite sides of chunking (core/pipeline/mod.rs:206-213 vs :406-413).
• One slow item in a batch with a caller cancel token cancels every other item (core/extractor/batch.rs:107,228,333).

11. Stale cache hits [R]

#[serde(skip)] fields are invisible to the cache key but change output. Extract with English tessdata_bytes, re-extract with German under the same language label → English result served (core/config/ocr.rs:496, core/extractor/file.rs:225-241). Same for source_name: identical bytes named x.py vs x.rb share one cached language attribution (core/config/extraction/core.rs:384).

12. Config fields that do nothing [R] and PDF structure defects [R]

• Inert config: top/bottom_margin_fraction, include_bbox, ocr_coverage_threshold — documented, plumbed through bindings, never read (core/config/pdf.rs:42-118).
• Zero-height glyph boxes → paragraph gap threshold 0 → every line its own paragraph (pdf/oxide/text.rs:471-478).
• 7pt footnote marker in 11pt body → sentence split into 3 paragraphs (pipeline.rs:519,544).
• well-known across a line break → wellknown (pipeline.rs:1948-2029).
• Repeated warnings/instructions deduplicated as furniture — repeated content deleted (pipeline.rs:2101-2120).
• Short numeric fragments become headings — ### 1,000)* (pipeline.rs:706-715).

13. Language detection, diff, embeddings [E] where tagged

• Language detection [E]: min_confidence: 0.99 in multi-language mode → still returns 0.35-confidence languages (silently capped); ties broken by HashMap order, so the primary language flips between runs (language_detection/mod.rs:84-102).
• Diff [E]: a table that gains a column → diff entry with zero cell changes, in neither added nor removed. And compare() never reads formatted_content, so rendered-output regressions read as "no change" (diff/mod.rs:60-197).
• Embeddings [R]: truncation hardcoded at 512 tokens, no config; long chunks embed only their prefix while storing full text (embeddings/mod.rs:674).

---

Removed by runtime testing (not defects)

• Table-parented captions dropped — caption machinery never invoked in production.
• Hairline stroke expansion harming checkbox strokes — excluded by the >5pt gate.
• HTML bytes routed to the XML extractor — both probes detect text/html.
• Colspan header misalignment — headers align; only rowspan misaligns (finding 2 narrowed).

Upstream (pdf_oxide) — tracked separately, NOT fixed in this issue's PR

These are pdf_oxide-level defects. The bordered stroke-width table class (the original trigger) is tracked in #1213.

• [Bug]: Tables with visible grid lines aren't detected in some print-era PDFs yfedoseev/pdf_oxide#812 — path bboxes ignore stroke width (root cause of the fuse-chart miss; PR chore(deps-dev): update types-psutil requirement from >=7.1.3.20251202 to >=7.2.2.20260408 #814). Blocks bug: bordered stroke-width-ruled tables not detected (fuse charts) — blocked on pdf_oxide #814 #1213.
• [Bug]: Extracted words come apart in the middle ("module" becomes "m odu le") yfedoseev/pdf_oxide#811 — words split inside single string literals ("module" → "m odu le").
• [Bug]: Pages with sideways (rotated) text extract as scrambled text, and words don't report their rotation yfedoseev/pdf_oxide#813 — rotated pages extract in portrait order, no rotation metadata.

Highest-leverage fixes

These findings share a handful of root causes; fixing those beats point patches:

1. One exhaustive field-walk in the redaction stage (fixes finding 1 across all ~12 fields).
2. One span-aware grid→rows helper for every table flattener (finding 2, four formats).
3. One charset-detection entry point for all text extractors (finding 3).
4. A pipeline rule: no content mutation after derive/chunking, or make the swap and chunks/entities merge-aware (findings 4, 6).
5. Cache keys from effective config, not serde output (finding 11).
6. A test that fails when a public config field has zero read sites (findings 5, 12).
7. Error-not-silence policy for OCR/archive/attachment failure paths (findings 7, 9).

The corpus A/B runner used during this audit caught two bad fix iterations before they shipped and is the natural regression harness for the above.