perf: large-repo indexing pass (type refs, health, dynamic hints, dead code) + update-path fixes (#450)

* perf(ingestion): index defined symbol names once in type-ref resolution

Type-ref resolution scanned graph.successors() of every candidate file
for every type reference to find a type's defining file, costing
O(refs x candidates x symbols_per_file) on package-fan-out languages.
On hugo (896 Go files) this was 8s of an 11s graph build.

Build one node-to-defined-symbol-names map per resolve pass and answer
lookups by set membership; hoist per-file candidate sorting out of the
per-ref loop. Only file-to-file type_use edges are added while the
strategies run, so the upfront index matches live scanning. The full
graph snapshot on hugo is bit-identical under a fixed hash seed; graph
build drops from 10.8s to 2.7s.

* perf(health): precompute test-pair basenames once per analyze pass

_has_paired_test_file scanned every analyzed path for every evaluated
file, doing O(files x paths x candidates) string suffix checks. On hugo
(2,014 files) that was 16.5s of the health phase under cProfile.

A path matches endswith(/ + candidate) or equals the candidate
exactly when its /-basename equals the candidate, so one basename set
built per analyze pass answers every lookup by set membership. Health
full analysis on hugo drops from 18.3s to 12.9s isolated (warm
duplication cache). A regression test pins the new lookup to a verbatim
copy of the old scan over a corpus of tricky paths.

* fix(cli): record the workspace distill verdict before init fingerprints config

repowise update backfills a workspace member's distill.commands.enabled
verdict into .repowise/config.yaml at the start of every run. Init left
the verdict unwritten, so the first update of every workspace member saw
a config-fingerprint mismatch and silently replaced the incremental path
with a full health re-score of every file (47s instead of the partial
update on an ~900-file Go repo). Init now runs the same backfill before
computing the fingerprint, making the update-time backfill a no-op.

* perf(ingestion): share one walk snapshot across dynamic-hint extractors

The 18 dynamic-hint extractors issued ~40 iter_glob queries per run and
each walked the tree from disk (the C++ extractor alone walks once per
extension). WalkSnapshot in fs_walk replays a single pruned walk for all
of them, preserving iter_glob semantics and yield order, serving
subtree-rooted queries from the snapshot, and falling back to a live
walk outside the tree. extract_all on hugo drops from 2.9s to 0.8s with
a bit-identical edge list.

* fix(pipeline): converge the update-built graph with the init-built graph

The incremental rebuild skipped two edge passes the init pipeline runs:
dynamic hints never executed on update, and co_changes edges were
re-added only for the changed files on a graph rebuilt from scratch, so
unchanged files lost their co-change edges. Metrics persisted by an
update disagreed with init for identical code, and the first post-init
update could never hit the centrality cache, recomputing betweenness
(~11s on hugo) on every fresh-init-then-update sequence.

build_repo_graph now runs the dynamic-hint registry like init does, and
index_changed_files exposes the full per-file partner map its repo-wide
co-change walk already computes so rebuild_graph_and_git re-adds
co_changes edges for every file. A convergence test asserts node and
edge set equality plus centrality-cache signature equality between
init-shaped and update-shaped builds.

* refactor(git): derive blame ownership from the single porcelain pass

Files below the function-blame commit floor took a second blame pass
through gitpython repo.blame just for ownership, while denser files
already derived ownership from the porcelain BlameIndex parse. One
git blame --line-porcelain invocation now serves both signals for
every file; the commit floor only gates whether the BlameIndex is
retained for the function biomarkers, exactly as before.

Wall time is neutral (blame cost is the subprocess, measured on a
~900-file repo), but ownership is bit-identical across that corpus,
the GIL-holding gitpython object parse no longer competes with
concurrent ingestion, and the duplicate code path is gone.

* perf(analysis): memoize the dead-code never-flag regex match

The never-flag alternation compiles ~540 globs into one pattern, so a
single match costs tens of microseconds, and both node-scanning
detector passes ask about the same ~15k node ids. A module-level
lru_cache on the match (a pure function of the path; the pattern set
is a module constant) cuts the full dead-code pass from 3.1s to 1.8s
on hugo, 0.4s for repeat analyses, with identical findings.

Author: RaghavChamadiya

packages/cli/src/repowise/cli/commands/init_cmd/command.py

@@ -795,6 +795,15 @@ async def _index_with_resume() -> Any:
     )
     register_editor_clients(console, repo_path)
 
+    # Inherit the workspace's distill rewrite-hook verdict NOW, before the
+    # config fingerprint below is computed. `repowise update` runs the same
+    # backfill at its start; if init leaves the verdict unwritten, the first
+    # update writes it, sees a fingerprint mismatch, and silently replaces
+    # the incremental path with a full health re-score of every file.
+    from repowise.cli.commands.workspace_cmd import inherit_workspace_distill_verdict
+
+    inherit_workspace_distill_verdict(repo_path)
+
     # ---- State (always) ----
     # Even in index-only mode we persist `last_sync_commit` so that a
     # subsequent `repowise update` (e.g. fired by the post-commit hook) has

packages/core/src/repowise/core/analysis/dead_code/analyzer.py

@@ -339,6 +339,19 @@ def _never_flag_regex(patterns: tuple[str, ...]) -> re.Pattern[str]:
     return re.compile("|".join(fnmatch.translate(os.path.normcase(p)) for p in patterns))
 
 
+@lru_cache(maxsize=131072)
+def _never_flag_regex_match(path: str) -> bool:
+    """Memoized ``_never_flag_regex`` match for the default pattern set.
+
+    The alternation has ~540 branches, so one match costs tens of
+    microseconds, and the detector passes ask about the same node ids
+    repeatedly (every graph node is checked by both the unreachable-files
+    and unused-exports passes). Pure function of *path*: the pattern set is
+    a module constant, so process-wide memoization is sound.
+    """
+    return _never_flag_regex(_NEVER_FLAG_PATTERNS).match(os.path.normcase(path)) is not None
+
+
 class DeadCodeAnalyzer:
     """Detects unreachable files, unused exports, unused internals, and
     zombie packages using the dependency graph and git metadata.
@@ -1186,7 +1199,7 @@ def _should_never_flag(self, path: str, whitelist: set[str]) -> bool:
         """Return True if path should never be flagged as dead."""
         if path in whitelist:
             return True
-        if _never_flag_regex(_NEVER_FLAG_PATTERNS).match(os.path.normcase(path)):
+        if _never_flag_regex_match(path):
             return True
         # Workspace-driven never-flag — set by language warmups that read
         # the build manifest (Gradle non-``main`` source sets, Cargo

packages/core/src/repowise/core/analysis/health/engine.py

@@ -219,11 +219,24 @@ def _build_repo_commit_counts(git_meta_map: dict[str, dict]) -> dict[str, int]:
     return out
 
 
-def _has_paired_test_file(rel_path: str, all_paths: set[str]) -> bool:
+def _path_basenames(all_paths: set[str]) -> set[str]:
+    """Final path components of *all_paths*, split on ``/`` only.
+
+    A path matches ``other.endswith("/" + c) or other == c`` for a
+    slash-free candidate filename ``c`` exactly when its ``/``-basename
+    equals ``c``, so one precomputed basename set answers every
+    ``_has_paired_test_file`` lookup. Splitting on ``/`` only (not ``\\``)
+    preserves that equivalence for any non-POSIX path that slips in.
+    """
+    return {p.rsplit("/", 1)[-1] for p in all_paths}
+
+
+def _has_paired_test_file(rel_path: str, path_basenames: set[str]) -> bool:
     """Heuristic: does any other file look like a test for *rel_path*?
 
     Cheap and conservative — looks for common test-file naming
-    conventions paired with the same basename.
+    conventions paired with the same basename. *path_basenames* is the
+    precomputed ``_path_basenames`` set for the analyzed file list.
     """
     p = Path(rel_path)
     stem = p.stem
@@ -241,9 +254,7 @@ def _has_paired_test_file(rel_path: str, all_paths: set[str]) -> bool:
         f"{stem}.spec.cts",
         f"{stem}_test.go",
     }
-    return any(
-        any(other.endswith("/" + c) or other == c for c in candidates) for other in all_paths
-    )
+    return not candidates.isdisjoint(path_basenames)
 
 
 class HealthAnalyzer:
@@ -301,7 +312,7 @@ def analyze(
         # PageRank is optional — graph_builder.symbol_pagerank exists but
         # is symbol-level; we use file-level in-degree as the dependents
         # signal (cheap, deterministic, conservative).
-        all_paths = {pf.file_info.path for pf in self.parsed_files}
+        path_basenames = _path_basenames({pf.file_info.path for pf in self.parsed_files})
         repo_commit_counts = _build_repo_commit_counts(self.git_meta_map)
         graph_view: HasEdge | None = _ImportEdgeView(self.graph) if self.graph is not None else None
 
@@ -364,7 +375,7 @@ def analyze(
             file_metric, file_findings = self._evaluate_file(
                 pf,
                 fcx,
-                all_paths,
+                path_basenames,
                 disabled=file_disabled,
                 dup_report=dup_report,
                 graph_view=graph_view,
@@ -422,7 +433,7 @@ async def analyze_async(
         per_file_disabled: dict[str, set[str]] = cfg.get("per_file_disabled", {}) or {}
         changed_set: set[str] | None = set(changed_files) if changed_files is not None else None
 
-        all_paths = {pf.file_info.path for pf in self.parsed_files}
+        path_basenames = _path_basenames({pf.file_info.path for pf in self.parsed_files})
         repo_commit_counts = _build_repo_commit_counts(self.git_meta_map)
         graph_view: HasEdge | None = _ImportEdgeView(self.graph) if self.graph is not None else None
 
@@ -504,7 +515,7 @@ async def _one(pf: Any) -> tuple[Any, FileComplexity]:
             file_metric, file_findings = self._evaluate_file(
                 pf,
                 fcx,
-                all_paths,
+                path_basenames,
                 disabled=file_disabled,
                 dup_report=dup_report,
                 graph_view=graph_view,
@@ -581,7 +592,7 @@ def _evaluate_file(
         self,
         pf: Any,
         fcx: FileComplexity,
-        all_paths: set[str],
+        path_basenames: set[str],
         *,
         disabled: list[str],
         dup_report: DuplicationReport,
@@ -627,7 +638,7 @@ def _evaluate_file(
             file_path=file_path,
             language=pf.file_info.language,
             nloc=nloc,
-            has_test_file=_has_paired_test_file(file_path, all_paths)
+            has_test_file=_has_paired_test_file(file_path, path_basenames)
             or _is_test_file(file_path)
             or _coverage_is_test_file(file_path),
             module=module,

packages/core/src/repowise/core/fs_walk.py

@@ -41,7 +41,7 @@
 
 log = structlog.get_logger(__name__)
 
-__all__ = ["PRUNED_DIRS", "PRUNED_DIRS_DERIVED", "iter_glob", "walk_repo"]
+__all__ = ["PRUNED_DIRS", "PRUNED_DIRS_DERIVED", "WalkSnapshot", "iter_glob", "walk_repo"]
 
 # Directory basenames that can NEVER hold first-party source or manifests:
 # VCS metadata, package/venv caches, and tool caches. Pruned at every level
@@ -196,6 +196,83 @@ def _matches(rel_posix: str, basename: str, pattern: str) -> bool:
     return fnmatch.fnmatch(rel_posix, pattern) or fnmatch.fnmatch(rel_posix, "*/" + pattern)
 
 
+class WalkSnapshot:
+    """One pruned walk, replayed for many :func:`iter_glob`-style queries.
+
+    Callers that issue several ``iter_glob`` queries against the same tree
+    (the dynamic-hint extractors run ~40 of them) pay one filesystem walk
+    here and answer every query from memory. Replay preserves
+    :func:`iter_glob` semantics and yield order exactly: entries are stored
+    in walk order, files before directories per directory, and ``/``-tail
+    patterns match against paths relative to the *query* root.
+
+    Queries rooted below the snapshot root are served from the snapshot's
+    subtree (os.walk pre-order keeps a subtree's entries in the same
+    relative order a direct walk of that subtree would produce). A query
+    root outside the snapshot tree falls back to a live walk.
+    """
+
+    __slots__ = ("_entries", "_prune_dirs", "_prune_nested_git", "root")
+
+    def __init__(
+        self,
+        root: Path | str,
+        *,
+        prune_dirs: frozenset[str] = PRUNED_DIRS,
+        prune_nested_git: bool = True,
+    ) -> None:
+        self.root = Path(root)
+        self._prune_dirs = prune_dirs
+        self._prune_nested_git = prune_nested_git
+        # (dirpath, root-relative posix dir ('' for the root), dirnames, filenames)
+        self._entries: list[tuple[Path, str, list[str], list[str]]] = []
+        for dirpath, dirnames, filenames in walk_repo(
+            self.root, prune_dirs=prune_dirs, prune_nested_git=prune_nested_git
+        ):
+            rel = dirpath.relative_to(self.root).as_posix()
+            self._entries.append(
+                (dirpath, "" if rel == "." else rel, list(dirnames), list(filenames))
+            )
+
+    def iter_glob(self, root: Path | str, patterns: str | Iterable[str]) -> Iterator[Path]:
+        """Replay of ``iter_glob(root, patterns)`` against the snapshot."""
+        query_root = Path(root)
+        pats = (patterns,) if isinstance(patterns, str) else tuple(patterns)
+
+        sub_rel: str | None
+        if query_root == self.root:
+            sub_rel = None
+        else:
+            try:
+                sub_rel = query_root.relative_to(self.root).as_posix()
+            except ValueError:
+                # Outside the snapshot tree: serve live with the same pruning.
+                yield from iter_glob(
+                    query_root,
+                    pats,
+                    prune_dirs=self._prune_dirs,
+                    prune_nested_git=self._prune_nested_git,
+                )
+                return
+
+        for dirpath, rel_dir, dirnames, filenames in self._entries:
+            if sub_rel is None:
+                q_rel = rel_dir
+            elif rel_dir == sub_rel:
+                q_rel = ""
+            elif rel_dir.startswith(sub_rel + "/"):
+                q_rel = rel_dir[len(sub_rel) + 1 :]
+            else:
+                continue
+            prefix = "" if q_rel == "" else q_rel + "/"
+            for name in filenames:
+                if any(_matches(prefix + name, name, p) for p in pats):
+                    yield dirpath / name
+            for name in dirnames:
+                if any(_matches(prefix + name, name, p) for p in pats):
+                    yield dirpath / name
+
+
 def iter_glob(
     root: Path | str,
     patterns: str | Iterable[str],

packages/core/src/repowise/core/ingestion/dynamic_hints/base.py

@@ -20,15 +20,24 @@ class DynamicEdge:
 class DynamicHintExtractor(ABC):
     name: str
 
+    # Shared walk snapshot, attached by HintRegistry.extract_all so the ~40
+    # _rglob queries the extractor fleet issues replay ONE filesystem walk
+    # instead of each walking the tree again. None -> live walk (direct
+    # extractor use in tests keeps working unchanged).
+    _walk_snapshot = None
+
     @abstractmethod
     def extract(self, repo_root: Path) -> list[DynamicEdge]: ...
 
-    @staticmethod
-    def _rglob(root: Path, pattern: str) -> Iterator[Path]:
+    def _rglob(self, root: Path, pattern: str) -> Iterator[Path]:
         """Pruned replacement for :py:meth:`pathlib.Path.rglob`.
 
         Skips ``node_modules``, ``.venv``, ``.next``, etc. so large
         polyrepos don't tank the dynamic-hints phase. See
-        :mod:`dynamic_hints._walk` for the full prune list.
+        :mod:`dynamic_hints._walk` for the full prune list. Served from
+        the registry's shared :class:`~repowise.core.fs_walk.WalkSnapshot`
+        when one is attached.
         """
+        if self._walk_snapshot is not None:
+            return self._walk_snapshot.iter_glob(root, pattern)
         return iter_glob(root, pattern)

packages/core/src/repowise/core/ingestion/dynamic_hints/registry.py

@@ -89,17 +89,38 @@ def extract_all(self, repo_root: Path) -> list[DynamicEdge]:
         if not self._extractors:
             return edges
 
-        with ThreadPoolExecutor(max_workers=self._max_workers) as pool:
-            futures = {pool.submit(self._run_one, ex, repo_root): ex for ex in self._extractors}
-            for future in as_completed(futures):
-                ex = futures[future]
-                try:
-                    got = future.result()
-                except Exception as e:
-                    log.warning("dynamic_hints_failed", extractor=ex.name, error=str(e))
-                    continue
-                edges.extend(got)
-                log.debug("dynamic_hints", extractor=ex.name, count=len(got))
+        # One pruned walk shared by every extractor: the fleet issues ~40
+        # _rglob queries, and each used to re-walk the tree. Snapshot
+        # construction failure falls back to per-extractor live walks.
+        snapshot = None
+        try:
+            from repowise.core.fs_walk import WalkSnapshot
+
+            from ._walk import PRUNED_DIRS
+
+            snapshot = WalkSnapshot(repo_root, prune_dirs=PRUNED_DIRS)
+        except Exception as e:  # pragma: no cover - snapshot is best-effort
+            log.warning("dynamic_hints_snapshot_failed", error=str(e))
+        for ex in self._extractors:
+            ex._walk_snapshot = snapshot
+
+        try:
+            with ThreadPoolExecutor(max_workers=self._max_workers) as pool:
+                futures = {
+                    pool.submit(self._run_one, ex, repo_root): ex for ex in self._extractors
+                }
+                for future in as_completed(futures):
+                    ex = futures[future]
+                    try:
+                        got = future.result()
+                    except Exception as e:
+                        log.warning("dynamic_hints_failed", extractor=ex.name, error=str(e))
+                        continue
+                    edges.extend(got)
+                    log.debug("dynamic_hints", extractor=ex.name, count=len(got))
+        finally:
+            for ex in self._extractors:
+                ex._walk_snapshot = None
         # Thread-completion order varies run-to-run; sort so downstream graph
         # construction (and therefore the exported KG) stays deterministic.
         edges.sort(key=lambda e: (e.source, e.target, e.edge_type, e.hint_source, e.weight))

packages/core/src/repowise/core/ingestion/dynamic_hints/xaml.py

@@ -95,7 +95,7 @@ class XamlDynamicHints(DynamicHintExtractor):
 
     def extract(self, repo_root: Path) -> list[DynamicEdge]:
         # Cheap pre-flight: any XAML in the tree at all?
-        xaml_files = list(_iter_xaml_files(repo_root))
+        xaml_files = list(_iter_xaml_files(repo_root, rglob=self._rglob))
         if not xaml_files:
             return []
 
@@ -172,10 +172,13 @@ def extract(self, repo_root: Path) -> list[DynamicEdge]:
 # Helpers (module-level so they're easy to unit-test in isolation)
 # ---------------------------------------------------------------------------
 
-def _iter_xaml_files(repo_root: Path):
-    from ._walk import iter_glob as _iter_glob
+def _iter_xaml_files(repo_root: Path, rglob=None):
+    if rglob is None:
+        from ._walk import iter_glob as _iter_glob
+
+        rglob = _iter_glob
     for ext in _XAML_EXTS:
-        for path in _iter_glob(repo_root, f"*{ext}"):
+        for path in rglob(repo_root, f"*{ext}"):
             try:
                 rel = path.resolve().relative_to(repo_root.resolve())
             except ValueError:

packages/core/src/repowise/core/ingestion/git_indexer/file_history.py

@@ -30,8 +30,12 @@
     _PR_NUMBER_RE,
     HOTSPOT_HALFLIFE_DAYS,
 )
-from .enrich import detect_original_path, get_blame_ownership, is_significant_commit
-from .function_blame import build_blame_index, ownership_from_blame
+from .enrich import detect_original_path, is_significant_commit
+from .function_blame import (
+    _MIN_COMMITS_FOR_BLAME,
+    build_blame_index,
+    ownership_from_blame,
+)
 from .records import (
     _LOG_FORMAT,
     _RECORD_SEP,
@@ -384,6 +388,14 @@ def index_file(
         # primary-owner signal and the in-memory ``BlameIndex`` consumed by
         # ``function_hotspot`` / ``code_age_volatility``. Skipped for large
         # files — git blame is O(lines) and can block the executor thread.
+        #
+        # Files below the function-blame commit floor used to take a second
+        # blame pass through gitpython's ``repo.blame`` just for ownership.
+        # That object-building parse is pure Python, holds the GIL across the
+        # 20-way thread fan-out, and on a large repo MOST files sit below the
+        # floor, so it dominated the FULL git phase. The porcelain parse now
+        # serves ownership for every file; the floor only gates whether the
+        # BlameIndex is retained for the function biomarkers, as before.
         if include_blame:
             try:
                 file_size = (repo_path / file_path).stat().st_size
@@ -392,25 +404,15 @@ def index_file(
                         repo,
                         file_path,
                         repo_path=repo_path,
-                        commit_count_total=meta["commit_count_total"],
                     )
                     if blame_idx.lines:
-                        meta["blame_index"] = blame_idx
+                        if meta["commit_count_total"] >= _MIN_COMMITS_FOR_BLAME:
+                            meta["blame_index"] = blame_idx
                         blame_name, blame_email, blame_pct = ownership_from_blame(blame_idx)
                         if blame_name:
                             meta["primary_owner_name"] = blame_name
                             meta["primary_owner_email"] = blame_email
                             meta["primary_owner_commit_pct"] = blame_pct
-                    else:
-                        # Below the in-blame commit-count floor — fall back
-                        # to the legacy gitpython ownership computation so we
-                        # don't lose owner data on small files that the
-                        # function biomarkers don't need anyway.
-                        blame_name, blame_email, blame_pct = get_blame_ownership(repo, file_path)
-                        if blame_name:
-                            meta["primary_owner_name"] = blame_name
-                            meta["primary_owner_email"] = blame_email
-                            meta["primary_owner_commit_pct"] = blame_pct
             except Exception:
                 pass  # blame is best-effort