feat: migrate ghcrawl to persistent vectorlite search and clustering

[huntharo]

Summary

• I replaced the in-memory exact vector path with a persistent vectorlite sidecar index for semantic search, neighbors, and clustering.
• I moved the active pipeline to one 1024-dimension vector per open thread and wired refresh to perform the one-time migration on the first rebuild.
• I kept LLM summaries optional: the default embedding basis is now title_original, while title_summary is an opt-in mode that summarizes before embedding.
• I added operator-facing controls and guidance for summary model, embedding basis, migration costs, and the TUI refresh flow.

What Changed

• Persistent vector backend:
  • vectorlite is now the default vector backend.
  • active vectors are stored per thread and backed by the persistent sidecar store.
  • semantic search, neighbors, and cluster use that persistent vector store.
• Migration behavior:
  • the first successful ghcrawl refresh owner/repo rebuild migrates the repo to the new vector path.
  • plain sync does not perform that migration.
  • after a successful rebuild, ghcrawl removes legacy document_embeddings, compacts stored vectors, checkpoints the WAL, and vacuums the DB.
• Summaries and costs:
  • default summary model is gpt-5-mini.
  • default embedding basis is title_original to avoid surprise one-time LLM spend.
  • title_summary is available as an opt-in mode through ghcrawl configure.
  • README and TUI copy now explain that summarizing roughly 18k open issues/PRs in openclaw/openclaw is typically about $15-$30 one time with gpt-5-mini, and later refreshes are usually much cheaper.
• TUI / UX:
  • the update popup now says whether LLM summaries are enabled or disabled and explains the tradeoff.
  • the detail pane now shows LLM Summary above the raw body instead of exposing the internal dedupe_summary label.

Clustering Results

On the live openclaw/openclaw dataset:

• with title_original, non-solo cluster membership is about 43.7%
• with title_summary, non-solo cluster membership improves by about 50% relative to title_original

That is why summaries are presented as an opt-in quality mode rather than the default.

Size / Migration Results

On the migrated local openclaw/openclaw data:

• main DB dropped from about 5.2 GB to about 867 MB
• total ~/.config/ghcrawl dropped to about 965 MB
• legacy document_embeddings for the repo were removed after migration

How I Tested

• pnpm --filter @ghcrawl/api-core test
• pnpm --filter ghcrawl test
• pnpm typecheck
• pnpm --filter ghcrawl cli cluster openclaw/openclaw
• pnpm --filter ghcrawl cli cluster openclaw/openclaw --threshold 0.78

I also verified the migration behavior against the real local openclaw/openclaw database, including:

• one-time cleanup of legacy embeddings
• DB compaction
• updated cluster coverage
• PR CI is green on this branch

[github-actions]

Cluster Performance

• Backend: exact
• Timing basis: cluster-only
• Status: PASS
• Fixture median (cluster-only): 339.5 ms (12 samples, 3 cluster rebuilds/sample)
• Fixture median (total run): 394.0 ms
• Fixture median load stage: 48.5 ms
• Fixture median setup stage: 0.0 ms
• Fixture median exact edge-build stage: 338.5 ms
• Fixture median vector index-build stage: 0.0 ms
• Fixture median vector query stage: 0.0 ms
• Fixture median cluster-assembly stage: 1.0 ms
• Median peak RSS: 223.3 MiB
• Median peak heap used: 72.3 MiB
• Fixture baseline: 450.6 ms
• Fixture delta: -111.1 ms (-24.7%)
• Projected openclaw/openclaw duration: 6m 20.7s
• Projected openclaw/openclaw baseline: 8m 25.3s
• Projected delta: -124586.8 ms (-24.7%)
• Regression threshold: +50.0%
• Fixture shape: 512 threads x 3 source kinds
• Sample durations: 341.0 ms, 345.0 ms, 338.0 ms, 338.0 ms, 340.0 ms, 337.0 ms, 341.0 ms, 339.0 ms, 345.0 ms, 338.0 ms, 337.0 ms, 347.0 ms
• Suggested baseline update: {"fixtureMedianMs":339.5,"projectedOpenclawMs":380713}

Run: workflow run for 3ffa673