Lower memory usage for lazily-loaded files and faster

go-to-implementation

Author: AJenbo

docs/CHANGELOG.md

@@ -52,6 +52,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 - **Diagnostic delivery model.** Editors that support pull diagnostics now get diagnostics on first file open without waiting for a debounce timer. Updates from external tools no longer re-run the entire native diagnostic pipeline.
 - **Virtual member resolution.** Mixins and virtual accessors are now resolved completely on every class, eliminating cases where they were missing after edits.
 - **Diagnostic code identifiers.** All diagnostic codes now use a consistent `snake_case` noun-phrase scheme: `unknown_variable`, `type_mismatch_argument`, `argument_count_mismatch`, `deprecated_usage`, `missing_implementation`. Users with editor filters matching on these codes will need to update them.
+- **Lower memory usage for lazily-loaded files.** Vendor and stub files no longer store per-file import tables and namespace maps after parsing, and go-to-implementation uses a dedicated reverse-inheritance index instead of scanning all parsed files.
 - **Lower memory usage for variable type tracking.**
 - **Updated embedded phpstorm-stubs.**
 

docs/todo.md

@@ -25,8 +25,6 @@ within the same impact tier.
 
 | #   | Item                                                                                                                    | Impact      | Effort      |
 | --- | ----------------------------------------------------------------------------------------------------------------------- | ----------- | ------ |
-| P13 | [Tiered storage: drop per-file maps for non-open files](todo/performance.md#p13-tiered-storage-drop-per-file-maps-for-non-open-files) | Medium-High | Medium-High |
-| P10 | [Redundant `parse_and_cache_file` from multiple threads](todo/performance.md#p10-redundant-parse_and_cache_file-from-multiple-threads)                                      | Medium      | Low         |
 | D10 | [PHPMD diagnostic proxy](todo/diagnostics.md#d10-phpmd-diagnostic-proxy)                                                | Low         | Medium      |
 |     | **Release 0.8.0**                                                                                                       |             |             |
 
@@ -165,7 +163,6 @@ unlikely to move the needle for most users.
 | E7  | [Stub-based framework patches](todo/external-stubs.md#e7-stub-based-framework-patches)                                                                                      | Medium      | Medium      |
 |     | **[Performance](todo/performance.md) / [Eager Resolution](todo/eager-resolution.md)**                                                                                       |             |             |
 | ER5 | [Mago-style separated metadata](todo/eager-resolution.md#er5--mago-style-separated-metadata)                                                                                | High        | High        |
-| P13 | [Tiered storage: drop per-file maps for non-open files](todo/performance.md#p13-tiered-storage-drop-per-file-maps-for-non-open-files)                                       | Medium-High | Medium-High |
 | P14 | [Eager docblock parsing into structured fields](todo/performance.md#p14-eager-docblock-parsing-into-structured-fields)                                                      | Medium      | Medium      |
 | P9  | [`resolved_class_cache` generic-arg specialisation](todo/performance.md#p9-resolved_class_cache-generic-arg-specialisation)                                                 | Medium      | Medium      |
 | P11 | [Uncached base-resolution in `build_scope_methods_for_builder`](todo/performance.md#p11-uncached-base-resolution-in-build_scope_methods_for_builder)                        | Low-Medium  | Low         |

docs/todo/performance.md

@@ -263,83 +263,6 @@ and eliminates the blocking fallback during interactive use.
 
 ---
 
-## P13. Tiered storage: drop per-file maps for non-open files
-
-**Impact: Medium-High · Effort: Medium-High**
-
-> **Note.** This item needs refinement when we work on it. The
-> codebase and feature set may change significantly before then.
-
-Not every file needs the same data at runtime. Storage should be
-split into three tiers based on how the file is used:
-
-| Data              | Open files | Closed user files   | Vendor files        |
-| ----------------- | ---------- | ------------------- | ------------------- |
-| ClassInfo (full)  | keep       | keep (via fqn_index)| keep (via fqn_index)|
-| SymbolMap         | keep       | drop (on-demand for find-refs) | never     |
-| use_map           | keep       | drop after index    | drop after index    |
-| namespace_map     | keep       | drop after index    | drop after index    |
-| parse_errors      | keep       | never               | never               |
-| ast_map entry     | keep       | drop (redundant with fqn_index) | drop     |
-| fqn_index         | keep       | keep                | keep                |
-| class_index       | keep       | keep                | keep                |
-| GTI index (new)   | keep       | keep                | keep                |
-
-**Key observations:**
-
-- **SymbolMap is the biggest win.** Each SymbolMap stores a
-  SymbolSpan for every symbol reference in the file, plus
-  VarDefSite, CallSite, and scope data. A typical file with
-  100-500 symbols is several KB. Across thousands of files this
-  adds up to tens or hundreds of MB.
-
-- **ast_map entries are redundant with fqn_index** once indexing
-  is complete. The slow linear fallback in `find_class_in_ast_map`
-  should not fire when fqn_index is fully populated. Go-to-definition
-  can re-parse on demand using the file path from class_index.
-
-- **Vendor files are rarely edited but can be diagnosed.** Users
-  working in monorepos or with `--prefer-source` packages edit
-  vendor files directly, and diagnostics run on any file open in
-  the editor. Tiered storage must still keep enough data to
-  support diagnostics for open vendor files, but non-open vendor
-  files only need ClassInfo for type resolution and class_index
-  for go-to-definition file lookup.
-
-- **Go-to-implementation currently scans all ast_map entries.**
-  A dedicated GTI index (parent FQN to list of child FQNs, built
-  during indexing) would decouple it from ast_map and allow
-  ast_map entries for non-open files to be dropped without
-  breaking implementation search. GTI needs vendor data (to find
-  chains through vendor classes) but only the parent/child
-  relationship, not the full per-file maps.
-
-- **Find-references only needs SymbolMaps for user code.** These
-  could be built on demand (parse, scan, drop) rather than kept
-  resident.
-
-- **Analyse mode benefits from laziness.** It never loads vendor
-  files that are not referenced by any user chain. LSP mode with
-  full vendor indexing would load everything since it cannot
-  predict what the user will type next. This makes the tiered
-  cleanup more important for LSP than for analyse.
-
-### Implementation sketch
-
-1. Track which URIs are "open" (already done via `open_files`).
-2. On `did_close`, drop the SymbolMap, use_map, namespace_map,
-   parse_errors, and ast_map entries for that URI. The fqn_index
-   entry (Arc\<ClassInfo\>) stays.
-3. For vendor files, use `parse_and_cache_content` (not
-   `update_ast`) so SymbolMaps are never created. After indexing,
-   sweep vendor URIs out of ast_map/use_map/namespace_map.
-4. Build a dedicated GTI index during indexing so that
-   `find_implementors` does not need ast_map.
-5. For find-references, build SymbolMaps on demand by re-parsing
-   from disk.
-
----
-
 ## P14. Eager docblock parsing into structured fields
 
 **Impact: Medium · Effort: Medium**
@@ -439,7 +362,7 @@ a `PhpVersion` parameter or build the filtered maps inline.
 Low priority. The current `RwLock` overhead is unmeasurable in
 practice (~10-20 ns per completion request). Worth revisiting if
 the stub indexes grow significantly or if `Backend` construction
-is restructured for other reasons (e.g. P13 tiered storage).
+is restructured for other reasons.
 
 ---
 

src/definition/implementation.rs

@@ -625,28 +625,49 @@ impl Backend {
         // Track by FQN to avoid short-name collisions across namespaces.
         let mut seen_fqns: HashSet<String> = HashSet::new();
 
-        // ── Phase 1: scan ast_map ───────────────────────────────────────
-        // Collect all candidate classes first, then drop the lock before
-        // calling class_loader (which may re-lock ast_map).
-        let ast_candidates: Vec<ClassInfo> = {
-            let map = self.ast_map.read();
-            map.values()
-                .flat_map(|classes| classes.iter().map(|c| ClassInfo::clone(c)))
-                .collect()
+        // ── Phase 1: GTI index lookup ───────────────────────────────────
+        // Use the reverse inheritance index for O(1) lookup of classes
+        // that directly extend/implement/use the target.  Then
+        // recursively collect transitive children.
+        let gti_candidates: Vec<String> = {
+            let gti = self.gti_index.read();
+            if direct_only {
+                gti.get(target_fqn).cloned().unwrap_or_default()
+            } else {
+                // Transitive: BFS collecting all descendants.
+                let mut all_children: Vec<String> = Vec::new();
+                let mut queue: Vec<String> = vec![target_fqn.to_string()];
+                let mut visited: HashSet<String> = HashSet::new();
+                visited.insert(target_fqn.to_string());
+                while let Some(parent) = queue.pop() {
+                    if let Some(children) = gti.get(&parent) {
+                        for child in children {
+                            if visited.insert(child.clone()) {
+                                all_children.push(child.clone());
+                                queue.push(child.clone());
+                            }
+                        }
+                    }
+                }
+                all_children
+            }
         };
 
-        for cls in &ast_candidates {
-            let cls_fqn = crate::util::build_fqn(&cls.name, cls.file_namespace.as_deref());
-            if self.class_implements_or_extends(
-                cls,
-                target_short,
-                target_fqn,
-                class_loader,
-                include_abstract,
-                direct_only,
-            ) && seen_fqns.insert(cls_fqn)
-            {
-                result.push(cls.clone());
+        for child_fqn in &gti_candidates {
+            if seen_fqns.contains(child_fqn) {
+                continue;
+            }
+            if let Some(cls) = class_loader(child_fqn) {
+                if !direct_only {
+                    if cls.kind == ClassLikeKind::Interface {
+                        continue;
+                    }
+                    if cls.is_abstract && !include_abstract {
+                        continue;
+                    }
+                }
+                seen_fqns.insert(child_fqn.clone());
+                result.push(Arc::unwrap_or_clone(cls));
             }
         }
 

src/lib.rs

@@ -376,6 +376,16 @@ pub struct Backend {
     /// inside `ClassInfo.methods`; future phases will make this the
     /// authoritative source and shrink `ClassInfo.methods` to just names.
     pub(crate) method_store: types::MethodStore,
+    /// Reverse inheritance index: parent FQN → list of child FQNs.
+    ///
+    /// For each class/interface/trait, maps the FQNs of its parents
+    /// (parent_class, interfaces, used_traits) to the child's FQN.
+    /// Used by `find_implementors` for O(1) lookup of direct children
+    /// instead of scanning all `ast_map` entries.
+    ///
+    /// Populated incrementally in `update_ast_inner` and
+    /// `parse_and_cache_content_versioned` as files are parsed.
+    pub(crate) gti_index: Arc<RwLock<HashMap<String, Vec<String>>>>,
     /// Embedded PHP stubs for built-in functions (e.g. `array_map`,
     /// `str_contains`, …).  Maps function name → raw PHP source code.
     ///
@@ -651,6 +661,7 @@ impl Backend {
             stub_constant_index: RwLock::new(stubs::build_stub_constant_index()),
             resolved_class_cache: virtual_members::new_resolved_class_cache(),
             method_store: Arc::new(RwLock::new(HashMap::new())),
+            gti_index: Arc::new(RwLock::new(HashMap::new())),
             php_version: Mutex::new(types::PhpVersion::default()),
             diag_version: Arc::new(AtomicU64::new(0)),
             diag_notify: Arc::new(tokio::sync::Notify::new()),
@@ -727,6 +738,7 @@ impl Backend {
             stub_constant_index: RwLock::new(HashMap::new()),
             resolved_class_cache: virtual_members::new_resolved_class_cache(),
             method_store: Arc::new(RwLock::new(HashMap::new())),
+            gti_index: Arc::new(RwLock::new(HashMap::new())),
             php_version: Mutex::new(types::PhpVersion::default()),
             diag_version: Arc::new(AtomicU64::new(0)),
             diag_notify: Arc::new(tokio::sync::Notify::new()),
@@ -977,6 +989,57 @@ impl Backend {
         }
     }
 
+    /// Populate the GTI (go-to-implementation) reverse inheritance index
+    /// for the given classes.  For each class, inserts the class's FQN
+    /// into the child list of every parent (parent_class, interfaces,
+    /// used_traits).
+    pub(crate) fn populate_gti_index(&self, classes: &[Arc<ClassInfo>]) {
+        let mut gti = self.gti_index.write();
+        for cls in classes {
+            if cls.name.starts_with("__anonymous@") {
+                continue;
+            }
+            let child_fqn = cls.fqn().to_string();
+
+            if let Some(ref parent) = cls.parent_class {
+                let parent_str = parent.to_string();
+                let children = gti.entry(parent_str).or_default();
+                if !children.contains(&child_fqn) {
+                    children.push(child_fqn.clone());
+                }
+            }
+            for iface in &cls.interfaces {
+                let iface_str = iface.to_string();
+                let children = gti.entry(iface_str).or_default();
+                if !children.contains(&child_fqn) {
+                    children.push(child_fqn.clone());
+                }
+            }
+            for tr in &cls.used_traits {
+                let tr_str = tr.to_string();
+                let children = gti.entry(tr_str).or_default();
+                if !children.contains(&child_fqn) {
+                    children.push(child_fqn.clone());
+                }
+            }
+        }
+    }
+
+    /// Remove all GTI entries where `child_fqn` appears as a child.
+    /// Called before re-populating when a file is re-parsed.
+    pub(crate) fn evict_gti_for_fqns(&self, fqns: &[String]) {
+        if fqns.is_empty() {
+            return;
+        }
+        let fqn_set: HashSet<&str> = fqns.iter().map(|s| s.as_str()).collect();
+        let mut gti = self.gti_index.write();
+        for children in gti.values_mut() {
+            children.retain(|child| !fqn_set.contains(child.as_str()));
+        }
+        // Remove empty entries to avoid unbounded growth.
+        gti.retain(|_, v| !v.is_empty());
+    }
+
     /// Create a shallow clone of this `Backend` that shares every
     /// `Arc`-wrapped field with the original.
     ///
@@ -1023,6 +1086,7 @@ impl Backend {
             stub_index: RwLock::new(self.stub_index.read().clone()),
             resolved_class_cache: Arc::clone(&self.resolved_class_cache),
             method_store: Arc::clone(&self.method_store),
+            gti_index: Arc::clone(&self.gti_index),
             stub_function_index: RwLock::new(self.stub_function_index.read().clone()),
             stub_constant_index: RwLock::new(self.stub_constant_index.read().clone()),
             php_version: Mutex::new(self.php_version()),

src/parser/ast_update.rs

@@ -519,8 +519,10 @@ impl Backend {
 
         // Populate the global method store for O(1) method lookup.
         self.evict_methods_for_fqns(&old_fqns);
+        self.evict_gti_for_fqns(&old_fqns);
         if let Some(arc_classes) = self.ast_map.read().get(&uri_string) {
             self.populate_method_store(arc_classes);
+            self.populate_gti_index(arc_classes);
         }
 
         self.symbol_maps

src/resolution.rs

@@ -443,15 +443,11 @@ impl Backend {
         self.ast_map
             .write()
             .insert(uri.to_owned(), arc_classes.clone());
-        self.use_map.write().insert(uri.to_owned(), file_use_map);
-        self.namespace_map.write().insert(
-            uri.to_owned(),
-            vec![crate::types::NamespaceSpan {
-                namespace: file_namespace.clone(),
-                start: 0,
-                end: content.len() as u32,
-            }],
-        );
+        // NOTE: use_map and namespace_map are intentionally NOT stored
+        // for lazily-loaded files (vendor, stubs, PSR-4).  These maps
+        // are only needed for files open in the editor (populated by
+        // update_ast_inner).  Skipping them reduces memory usage across
+        // thousands of vendor files.  See P13 (tiered storage).
 
         // Populate the fqn_index so that `find_class_in_ast_map` can
         // resolve these classes via O(1) hash lookup.
@@ -476,6 +472,7 @@ impl Backend {
             self.evict_methods_for_fqns(&fqns);
         }
         self.populate_method_store(&arc_classes);
+        self.populate_gti_index(&arc_classes);
 
         // Remove newly-discovered FQNs from the negative-result cache.
         {