Pre-organize deduplication: diff new notes against previous sessions and link duplicates

[github-actions]

Summary

Before the AI organize step categorizes files into organized/, run a deduplication pre-step that diffs new notes against all previous notepad-cleanup-* session folders. When a note already exists in a previous session, create a link to it in the correct location instead of copying it again. Only genuinely new notes are passed to the AI for categorization.

Motivation

Users who run notepad-cleanup regularly accumulate many session folders (e.g., notepad-cleanup-2026-01-01_..., notepad-cleanup-2026-02-14_...). Without deduplication:

• The same note (e.g., a persistent to-do list or reference snippet) gets re-copied into organized/ on every session, wasting disk space and polluting category folders.
• The AI agent spends time re-categorizing notes it has seen before, increasing cost and latency.
• The organized/ folder layout grows inconsistently across sessions because the AI may assign the same file to different categories on different runs.

Proposed Workflow

notepad-cleanup extract
    └──> dedup pre-step (NEW)
            ├── Find all previous notepad-cleanup-* session folders
            ├── Hash / diff extracted files against previously organized files
            ├── For each duplicate: create a DazzleLink (symlink / hardlink / .dazzlelink)
            │       pointing to the original in the previous session
            └── Pass only NEW (non-duplicate) files to the AI organize step
notepad-cleanup organize (AI sees only new files + link stubs for context)

Implementation Notes

Deduplication heuristics (ordered by cost)

1. Exact hash match -- SHA-256 of file content; instant, no false positives.
2. Near-duplicate / fuzzy match -- e.g., difflib similarity >= 95 %; catches minor edits of the same note.
3. Filename + size match -- lightweight secondary signal.

Linking strategy

• DazzleLink (https://github.com/DazzleTools/dazzlelink/) -- create .dazzlelink descriptor files (or OS-native symlinks) from the new session into the original file in the old session. This preserves the file graph across sessions and lets any tool follow the provenance chain.
• DazzlePreserve (https://github.com/DazzleTools/preserve/) -- record file movement / copy events so the provenance graph is queryable; useful for auditing which session introduced a given note.

Folder context for AI

When the AI organize step runs, it should receive:

• The list of new files (content to categorize).
• The list of linked files (filename + target path only) so the AI can see the existing folder layout and keep its new categories consistent with prior sessions.

This gives the AI an implicit schema of the organized/ tree without it having to re-read thousands of old files.

Proposed Changes

• Add dedup.py (or deduplicator.py) module to notepad_cleanup/ with:
  • find_previous_sessions(base_dir) -- locate sibling notepad-cleanup-* folders
  • hash_organized_files(session_dir) -- build a {sha256: Path} index of a session's organized/ tree
  • find_duplicate(file_path, index) -- return the canonical path if the file matches, else None
  • create_link(src, target, strategy) -- create a DazzleLink / symlink / hardlink
• Add --dedup / --no-dedup flag to organize and run commands in cli.py (default: --dedup when previous sessions exist)
• Pass linked_files context list to generate_prompt() in organizer.py so the AI sees what already exists
• Update execute_plan() in organizer.py to write a link instead of copying when the plan entry is flagged as a duplicate
• Update README with a description of the deduplication step and DazzleLink/DazzlePreserve integration

Tasks

• Research DazzleLink API / CLI for creating .dazzlelink descriptor files programmatically
• Research DazzlePreserve API for recording file movement events
• Decide on fuzzy-match threshold (default 95 %?) and make it configurable via --dedup-threshold
• Implement find_previous_sessions() and hash_organized_files() in new dedup.py
• Implement find_duplicate() with exact + optional fuzzy matching
• Implement create_link() supporting symlink, hardlink, and DazzleLink strategies
• Add --dedup / --no-dedup flags to organize and run commands in cli.py
• Update generate_prompt() to include linked-file context for the AI
• Update execute_plan() to write links instead of copies for duplicate entries
• Add unit tests for dedup.py (exact match, fuzzy match, no-match cases)
• Update README with deduplication workflow and DazzleLink/DazzlePreserve setup instructions

[djdarcy]

Implementation Analysis Complete

A detailed DEV WORKFLOW PROCESS analysis has been completed for this issue, covering:

Library Ecosystem Audit

Explored all available DazzleTools libraries for reuse:

• dazzle-filekit (dazzle_filekit.verification) -- calculate_file_hash() with SHA-256/MD5/SHA1/SHA512 support, calculate_directory_hashes(), cross-platform path handling
• DazzleTreeLib -- Universal tree traversal with async support, caching, and filtering (evaluated but not recommended for Phase 1 -- directory structure is too shallow to benefit)
• preservelib -- Manifest v3 with DAG lineage, DestinationScanResult for identical/conflict classification, dazzlelink integration (evaluated for Phase 3 provenance tracking)
• dazzlelink -- Cross-platform .dazzlelink JSON descriptors, native symlink creation with privilege-escalation fallback, batch operations
• wtf-restarted/lib -- core_lib has Plan/Action/ConflictResolution pattern; log_lib has THAC0 verbosity. No file/hash functionality (the "sha feature in file_lib" actually lives in dazzle-filekit)

Recommended Architecture: dazzle-filekit + dazzlelink

New module: notepad_cleanup/dedup.py with:

• find_session_dirs(search_dirs) -- scan configured directories for historical notepad-cleanup-* folders
• build_hash_index(session_dirs, cache_path) -- SHA-256 index with mtime-based cache invalidation
• find_duplicates(new_dir, hash_index, threshold) -- exact hash match + fuzzy character-diff matching
• create_links(matches, strategy) -- dazzlelink (default) / symlink / hardlink / none
• DedupMatch and DedupResult dataclasses for structured results

CLI Parameters Proposed

--dedup / --no-dedup              (default: enabled)
--dedup-dirs PATH [PATH ...]      (additional scan directories)
--dedup-threshold N               (max char differences for fuzzy match, default: 20)
--link-strategy CHOICE            (dazzlelink|symlink|hardlink|none, default: dazzlelink)
--dedup-cache / --no-dedup-cache  (hash index caching, default: enabled)

Key Design Decisions

1. Fuzzy matching: Dual threshold -- absolute char count (default 20) AND 10% size-relative cap to prevent false positives on tiny files
2. Default link strategy: .dazzlelink descriptors (no admin privileges required, cross-platform portable)
3. AI prompt splitting: New files sent as primary set for categorization; linked files sent as context-only (category names from previous sessions) to maintain naming consistency
4. Dependencies: Optional via pip install notepad-cleanup[dedup] -- falls back to stdlib if dazzle-filekit/dazzlelink not installed
5. Hash caching: JSON cache with mtime+size validation, auto-rebuild on corruption

Files Affected

File	Change
notepad_cleanup/dedup.py	NEW (~300 lines)
notepad_cleanup/saver.py	Add SHA-256 to manifest entries
notepad_cleanup/organizer.py	Prompt accepts linked_files context
notepad_cleanup/cli.py	Add --dedup* and --link-strategy options
notepad_cleanup/prompts/organize.md	Add "Previously Organized" section
setup.py	Add optional dedup extras group

Phased Rollout

• Phase 1: Core dedup with exact + fuzzy matching, dazzlelink/symlink linking, CLI options
• Phase 2: Hash cache optimization, DazzleTreeLib for large collections (>50 sessions)
• Phase 3: preservelib DAG provenance, notepad-cleanup history <file> command

Analysis

See 2026-03-15__06-30-08__dedup-pre-organize-implementation-plan.md for detailed analysis.

[djdarcy]

Implementation Status Update (v0.2.0 - v0.2.3)

The core of this issue has been implemented across v0.2.0 through v0.2.3. Here's what shipped:

Completed

Dedup engine (notepad_cleanup/dedup.py):

• find_session_dirs() scans configured search directories for historical notepad-cleanup-* sessions
• build_hash_index() with SHA-256 hashing, text normalization, mtime+size cache
• find_duplicates() with exact hash match + fuzzy near-match via difflib.SequenceMatcher
• Heuristic threshold curve: allowed = 1.396*ln(size)^2 - 6.75*ln(size) + 10.14 (3.5% average fit error across anchor points, documented in docs/fuzzy-matching.md)
• create_links() supporting symlink, hardlink, and DazzleLink strategies with auto-detection and fallback chain (symlink -> hardlink -> dazzlelink -> copy)
• DazzleLink descriptors use schema v1 with default_mode: "open" for native file opening
• load_link_manifest() / get_linked_paths() shared data layer
• Historical session indexing prefers organized/ over window*/ when both exist

Link-aware organize (notepad_cleanup/organizer.py):

• execute_plan() checks each file against _dedup_links.json -- creates symlinks to canonical provenance root instead of copying data
• generate_prompt() passes previous session category names as reference for naming consistency
• _organized_links.json tracks which organized/ files are links vs copies
• Fallback chain: symlink -> hardlink -> dazzlelink -> copy

CLI commands:

• compare command with --last, --fuzzy, --no-fuzzy, --link, -s/-ss/-nsp, --show-threshold, progress bar with per-file detail
• links separate/join command to split organized/ into real files vs linked files (reversible)
• diff command to launch generated diff script in Beyond Compare / WinMerge / etc.
• config command with unified folder registry, ... notation, MRU history

Tests: 33 passing (19 config + 14 dedup) covering hash, exact/near/no-match detection, threshold curve, linking, and config system.

Docs: docs/fuzzy-matching.md, docs/config.md, docs/parameters.md

Remaining

1. DazzlePreserve integration -- recording file movement events in a queryable provenance DAG. We have JSON manifests (_dedup_links.json, _organized_links.json, _compare_results.json) but not preservelib's DAG lineage system. This is a Phase 3 item from the original plan.

2. --dedup/--no-dedup flags on organize and run -- currently dedup is a separate compare step rather than an integrated flag. The workflow is compare --last --link auto then organize --last. Integrating into a single organize --dedup flag is tracked in Issue Composable command chaining: auto-pass folder between extract/compare/organize #11 (run --compare --pause).

3. Filename + size match as a lightweight secondary signal -- not implemented, relying on hash + fuzzy instead. Hash is fast enough with caching that this isn't needed.