Extraction writes doubled carriage returns: every saved tab ends lines with CR CR LF

[djdarcy]

Extraction writes doubled carriage returns (data munging on every save)

Problem

Every tab file the extractor saves has doubled carriage returns: lines end in \r\r\n instead of \r\n. The tool is munging user data on its primary output path, and has been since v0.1.x.

Root cause (notepad_cleanup/saver.py:94):

filepath.write_text(text, encoding="utf-8")

write_text opens the file in text mode with default newline translation. On Windows that converts every \n to \r\n at write time. The text arriving from extraction (WM_GETTEXT on RichEditD2DPT, discovery.py:read_richedit_text) already uses \r\n line endings, so each one becomes \r + (\n -> \r\n) = \r\r\n.

Evidence -- a controlled specimen. The same Notepad tab, captured two ways: once by our extractor, once by manually saving the tab from Notepad itself. The files should be byte-identical. Instead:

	extractor output	Notepad's own save
bytes	5,180	5,054 (delta = exactly 1 CR per line)
line endings	\r\r\n x126	\r\n x126
after collapsing \r\r\n -> \r\n	character-identical	(ground truth)

Zero other differences -- no encoding damage, em-dashes and all content intact. The corruption is purely the CR doubling.

Spread. Sampling raw window*/tab*.txt files across three sessions spanning v0.1.x through v0.2.x: 20/24 contain \r\r\n (the clean minority are files with no line breaks to corrupt). This has shipped in every release.

User impact:

1. Files render double-spaced in any editor that treats a lone \r as a line break (modern Notepad, VS Code, most tools) -- users must clean up every extracted note.
2. Round-trip identity is broken. Re-saving the same content from Notepad produces a "different" file, as the specimen shows.
3. Dedup is silently defeated for mixed pairs. normalize_text() turns x\r\r\n into x\n\n -- the phantom blank lines survive into the hash. Two extracted copies still match each other (both corrupted identically), which is why compare has appeared to work -- but an extracted copy never exact-matches a clean copy of the same text, and with one phantom newline per line the char diff (126 in the specimen) far exceeds the near-match threshold (~54 at that size), so fuzzy misses it too. The tool cannot recognize its own output as a duplicate of the real file.
4. The AI organize step reads doubled text -- inflated size hints in the manifest (chars counts include the phantom CRs) and double-spaced content in every file Claude reads.

Proposed solution

Normalize line endings at the extraction boundary, and make the disk format explicit at the write boundary.

Option A (recommended): internal \n, explicit CRLF on disk.

• In extraction (both Phase 1 read_richedit_text and the Phase 2 UIA path), normalize immediately: text.replace("\r\n", "\n").replace("\r", "\n"). One internal representation everywhere -- the same one normalize_text() already assumes.
• In saver.py, write with explicit newline serialization: filepath.write_text(text, encoding="utf-8", newline="\r\n") -- output files then match what Notepad itself would save.
• Manifest chars counts become CR-free and stable across platforms.

Option B: pass-through. Keep the control's \r\n and write with newline="" (no translation). Fewer lines changed, but the internal text keeps \r\n, which leaks into labels, char counts, and any future platform (the Sublime adaptation extracts \n-only text -- Option A gives both platforms the same internal form for free).

Option A is preferred: platform-specific line endings stay at the platform boundary, which is exactly the seam the multi-editor architecture wants.

Remediation for existing archives

Every historical session on disk is corrupted, and after the writer fix, new extractions will be clean -- so cross-era dedup (clean new file vs corrupted archive) breaks everywhere until archives are repaired. Proposal:

• notepad-cleanup repair --line-endings <folder> with --dry-run: collapse \r\r\n -> \r\n in window*/tab*.txt and organized/ text files. The transform is unambiguous (a legitimate \r\r\n sequence cannot be produced by Notepad).
• Rewriting changes mtime, which correctly invalidates the hash cache (it revalidates by mtime+size); hashes then change, so stale hashes in old _compare_results.json are re-derived on the next compare. Link targets are path-based and unaffected.
• Alternative considered: hash-time tolerance (collapse \r\r\n inside normalize_text). Rejected as the primary fix -- it hides the corruption instead of removing it, and the files stay double-spaced for users. Possibly worth adding temporarily so unrepaired archives still dedup, but the repair command is the real remedy.

Acceptance criteria

• Extraction round-trip test: text containing \r\n line endings, saved by the saver, produces a file with \r\n only (no \r\r\n anywhere)
• Byte-identity test: saver output for given content equals a reference CRLF encoding of that content
• Both extraction phases (silent WM_GETTEXT and UIA tab-select) normalize to internal \n before returning text
• Manifest chars counts exclude CR characters
• repair --line-endings command with --dry-run, reporting per-file changes; no file without \r\r\n is touched
• Repair is verified against a copy of a real corrupted session (before/after byte comparison)
• Regression test pinning that an extracted file and a Notepad re-save of the same content hash identically after the fix
• Docs note the historical corruption and the repair procedure

Related issues

• Refs Pre-organize deduplication: diff new notes against previous sessions and link duplicates #9 -- the corruption silently defeats cross-session dedup for any pair mixing extracted and clean copies of the same content
• Refs Cross-platform support: macOS TextEdit and Linux text editors #2 -- the internal-\n convention (Option A) is the same boundary the cross-platform work needs

Analysis

Specimen pair (project-private, for maintainers): get-the-beep-outta-here-blues-shuffle-lyrics.txt (extractor output) vs get-the-beep-outta-here-blues-shuffle-lyrics v2.txt (Notepad's own save of the same tab).

[djdarcy]

Fix implemented in v0.2.6 (commit 4c25c76). Status against the acceptance criteria:

Done:

• Extraction round-trip test (tests/test_extraction_newlines.py::test_saver_writes_crlf_never_crcrlf)
• Byte-identity test (test_saver_output_matches_notepad_encoding)
• Both extraction phases normalize to internal \n -- both route through read_richedit_text, which now normalizes at the boundary; the saver serializes explicit \r\n
• Manifest chars counts exclude CR characters (test_manifest_chars_exclude_cr)
• Regression test extracted-vs-Notepad-save (test_normalize_unifies_clean_encodings) -- and verified stronger than the criterion asked: a live extraction of a real tab is byte-identical to Notepad's own save of the same content (5,054/5,054 bytes), with a full 64-tab session census showing zero \r\r\n and zero lone CRs
• Docs (docs/archive-repair.md + CHANGELOG [0.2.6])

Remaining (split out to #18):

• repair --line-endings command with --dry-run
• Repair verified against a copy of a real corrupted session

One design decision recorded during implementation: there is deliberately no hashing tolerance for the old \r\r\n pattern -- corrupted archives do not match clean extractions, and the remedy is repairing the files (#18), not special-case code. Pinned by test_no_compat_shim_for_prefix_corruption.

Leaving this issue open until #18 lands the repair half.