Add cursor-based code navigation via LSP graph edges#1331
Add cursor-based code navigation via LSP graph edges#1331Asher- wants to merge 15 commits intooraios:mainfrom
Conversation
…kit-lsp sourcekit-lsp was launched with no arguments, giving it no location to store its background index. Without --scratch-path, textDocument/references always returns empty because there is no index store for cross-file symbol resolution. - Pass --scratch-path <repo>/.build/sourcekit-lsp when launching - Increase local indexing delay from 5s to 10s (real projects need more time) - Add retry logic for local runs when references are empty, not just CI Fixes root cause of issue oraios#876. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
On macOS, bare sourcekit-lsp resolves to Command Line Tools version which has limited indexing capabilities. xcrun without DEVELOPER_DIR also resolves to CLT. Setting DEVELOPER_DIR=/Applications/Xcode.app/Contents/Developer gives the full Xcode sourcekit-lsp with proper background indexing support. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Implements stateful cursor navigation that lets agents explore code structure incrementally along LSP relationships: containment, references, call hierarchy, and type hierarchy. The cursor tracks visited positions as a trail. New files: - src/serena/cursor.py: CursorState, CursorManager, EdgeType, neighbor resolution - src/serena/tools/cursor_tools.py: 6 tools (start, move, look, configure, history, close) Modified: - src/solidlsp/ls.py: 4 new high-level methods on SolidLanguageServer for call hierarchy (incoming/outgoing) and type hierarchy (supertypes/subtypes) - src/serena/agent.py: lazy-init CursorManager via get_cursor_manager() - src/serena/tools/__init__.py: register cursor_tools module Tools are marked ToolMarkerOptional so they must be explicitly enabled. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
- 39 unit tests covering CursorManager lifecycle, all 7 edge types (mocked LSP), edge type filtering, formatting, and error handling - 14 integration tests using the Python test repo with a live LSP server: start/look, contains edges, multi-step navigation with trail tracking, edge type configuration, format output, type hierarchy graceful handling, and multiple independent cursors - Documentation page covering concepts, all 6 tools with parameters and example output, usage patterns, and configuration Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
The prior commit test_cursor.py uses mocks for LSP and test_cursor_integration.py exercises CursorManager directly. This adds 32 tests that exercise the full MCP tool chain (CursorStartTool, CursorMoveTool, CursorLookTool, CursorConfigureTool, CursorHistoryTool, CursorCloseTool) through SerenaAgent.get_tool() against a live Python language server, covering: - Tool lifecycle (start, look, move, close) - Edge type configuration and invalid edge type error handling - Symbol body inclusion via cursor_configure - Multi-step navigation with trail verification - Multiple independent cursors - Type hierarchy edge graceful handling - Nested class navigation - NeighborSymbol formatting edge cases Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…d, fix test cleanup - Narrow resolve_neighbors except clauses from bare Exception to SolidLSPException so programming errors (KeyError, AttributeError) surface instead of being silently swallowed - Add summary warning log when edge types fail to resolve, making partial failures visible at default log level - Add debug log in _symbol_name_at fallback path for diagnosability - Guard start_cursor against duplicate cursor IDs (raises ValueError) - Add autouse cleanup fixture in test_cursor_navigation.py to close cursors between tests, preventing state accumulation - Fix section header from 'Unit Tests' to 'Integration Tests' - Fix misleading docstring in test_cursor.py test_move_cursor_to_neighbor - Update mock test to use SolidLSPException instead of bare Exception Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
…unit tests Mock tests used hardcoded /tmp paths and file:///tmp/... URIs which fail on Windows (path is on mount \\tmp, start on mount D:). Now uses tempfile.gettempdir() + PathUtils.path_to_uri() for cross-platform compatibility. Also fixes path separator in format assertion. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Cursor navigation is tested and stable — promote from optional to default-enabled. Users can still disable via excluded_tools in project config. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
… type - _make_symbol: accept None for rel_path, line, col (used in incomplete location test) - _cleanup_cursors: return Iterator[None] instead of None (generator fixture) Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
Hi @Asher- . Would you be up for a call to discuss the problem you are trying to solve here? There are many possible approaches to simplify the "map a repository" task, the one you propose is only one of them. |
|
Absolutely— I sent you an email to coordinate a chat. You're certainly right about there being many ways to map a repository; but the point here is not to create a map, rather to treat the existing information like a map. Claude can only see what makes it into Claude's context window. When MCPs operate through a query interface where all activity happens between query (send) and response (receive), everything in-between is hidden from Claude. The goal here is to let Claude be in control of the navigation through the information according to the terms in which the information is used. It gives Claude a geographical location in the LSP and lets it relate to the LSP content like moving through a labyrinth. Any given location is a node in a graph and permits Claude to 1. investigate what is at that node and where it connects along various axes of determination 2. to move along any of these axes to follow where it leads 3. to maintain multiple locations in the code space and navigate their relations as they interact. Of course all of this can be achieved from a top-down summary approach, but that relies on presenting "everything" at once and then having Claude sort through what matters and what to do with it as well as keeping track of it in the context. The cursor makes the context tracking a part of the history of the cursor that can also be investigated and recalled and that naturally includes the direction of inquiry because it is structured according to taking steps through the inquiry. It also functions to narrow down the reporting criterion, allowing any particular report to 1. report less 2. format more coherently and with less structure 3. imply options for next steps automatically 4. utilize far fewer cumulative tokens. |
|
@Asher- @MischaPanch please invite me to the call as well. |
…nsert_after, rename Extend the cursor — the stateful LSP-positioned navigator — to be the full MCP surface for symbol-level activity. New primitives: - cursor_find: pattern / substring search (multi-match variant of cursor_start, which requires a unique match). Auto-starts a cursor when the match is unique; otherwise returns the candidate list. - cursor_overview: list top-level symbols in a file (cursor-first replacement for get_symbols_overview). - cursor_replace_body / cursor_insert_before / cursor_insert_after: edit at the cursor's current position, re-anchoring after line changes. - cursor_rename: rename the symbol at the cursor throughout the codebase, re-anchoring to the renamed symbol. All new tools delegate to the existing LanguageServerSymbolRetriever / LanguageServerCodeEditor — no LSP logic is duplicated. The read primitives are ToolMarkerSymbolicRead; the edit primitives are ToolMarkerSymbolicEdit. None are ToolMarkerOptional — the cursor is the default MCP surface. CursorManager gains find_symbols / register_cursor_at_symbol / reanchor_cursor helpers to support cursor_find and post-edit re-anchoring.
The cursor is now the complete MCP interface for LSP-addressable activity: navigation, search, overview, read, and edit. The seven old symbol tools remain as Python classes for direct CLI / API use and as the cursor's internal delegate, but are hidden from the default MCP surface. Tools marked ToolMarkerOptional: - GetSymbolsOverviewTool (superseded by cursor_overview) - FindSymbolTool (superseded by cursor_find / cursor_start) - FindReferencingSymbolsTool (superseded by cursor_configure referenced-by + cursor_look) - ReplaceSymbolBodyTool (superseded by cursor_replace_body) - InsertAfterSymbolTool (superseded by cursor_insert_after) - InsertBeforeSymbolTool (superseded by cursor_insert_before) - RenameSymbolTool (superseded by cursor_rename) Each tool retains its existing ToolMarkerSymbolicRead / ToolMarkerSymbolicEdit marker. This is the symmetric move to d77ff6d, which made cursor tools default-enabled by removing ToolMarkerOptional.
- editing.yml: rewrite the 'Symbolic editing' section so agents guide users through cursor_start / cursor_find + cursor_replace_body / cursor_insert_* / cursor_rename instead of the old symbol tools. - planning.yml / onboarding.yml: add rename_symbol and the cursor edit tools to excluded_tools (read-only modes were already excluding the old edit tools). - internal_modes/jetbrains.yml: explicitly exclude all cursor tools (the cursor depends on LSP graph edges which the JetBrains backend doesn't provide). - system_prompt.yml: branch on cursor_start availability so the prompt walks agents through the cursor primitives when they are exposed, falling back to the old find_symbol / get_symbols_overview narrative otherwise. - docs/02-usage/042_cursor_navigation.md: expand to cover the new read / edit primitives, include a migration table from old symbol tools to cursor equivalents, and document that the old tools remain available via included_optional_tools.
test_cursor.py (unit, mocked LSP): - TestFindAndAnchor: find_symbols delegates to retriever; register_cursor_at_symbol handles auto and explicit IDs; reanchor_cursor refreshes after edits and handles renamed symbols. test_cursor_navigation.py (SerenaAgent integration, live Python LSP): - TestCursorFindAndOverview: cursor_find unique/multiple/empty paths, substring matching, cursor_overview happy path + missing-file error. - TestCursorEditTools: cursor_replace_body / cursor_insert_before / cursor_insert_after / cursor_rename exercised against a throwaway file that is cleaned up in the fixture. Error path: cursor without a relative_path refuses to edit. test_serena_agent.py: - Switch test_find_symbol_name_path, test_find_symbol_overloaded_function and the no-match variant from .apply_ex() to .apply(). FindSymbolTool is now ToolMarkerOptional so it is inactive in the default agent context; .apply() still exercises the underlying tool logic since these tests are about the tool's implementation, not the active-set gating.
|
The red X on this run ( Ubuntu (2 FAILED tests, both haxe) Root cause: Windows (setup failure, 0 tests run) Root cause: macOS: PASS. Comparison: The previous Tests run on this branch ( Local verification: All 101 cursor tests ( No FAILED test in the CI output touches code this PR modifies. A rerun of the failed jobs should clear it — I don't have permission to trigger that on this repo. @opcode81 / @MischaPanch, could one of you rerun, or let me know if you'd prefer I push a no-op trigger commit instead? |
3 participants
Motivation
Serena's existing symbol tools (
find_symbol,find_referencing_symbols,get_symbol_body) support point lookups — you ask about a specific symbol and get an answer. But an agent exploring an unfamiliar codebase needs something more: a map. It needs to land on a symbol, see what's around it, pick a direction, move there, and keep going — building up a mental model of the code's structure as it navigates, rather than teleporting between disconnected lookups.Today this requires the agent to repeatedly invoke
find_symbol+find_referencing_symbolsin separate tool calls, manually tracking what it's already visited and which direction to go next. The cursor changes this by making the code graph navigable: the agent places a cursor on a symbol and sees its neighborhood — what it contains, what references it, what it calls, what inherits from it. Eachcursor_moveshifts the viewport to an adjacent symbol, and the trail records everywhere the cursor has been. The agent uses Serena's code graph the way you'd use a map: orient, explore a direction, backtrack, try another path.The design uses pure LSP queries (no persistent index, no external dependencies) so it works for all Serena users out of the box. Every neighborhood resolution is a live LSP request, so the cursor always reflects current code state.
Summary
src/serena/cursor.py): Stateful cursors with trail tracking, 7 configurable edge types (contains, references, referenced-by, calls, called-by, inherits, inherited-by), multi-cursor supportsrc/serena/tools/cursor_tools.py):cursor_start,cursor_move,cursor_look,cursor_configure,cursor_history,cursor_close— default-enabled (users can disable viaexcluded_tools)src/solidlsp/ls.py):request_call_hierarchy_incoming,request_call_hierarchy_outgoing,request_type_hierarchy_supertypes,request_type_hierarchy_subtypesxcrunwithDEVELOPER_DIR, add--scratch-pathforfind_referencing_symbolsdocs/02-usage/042_cursor_navigation.md— concepts, all 6 tools with example output, usage patternsAudit-driven hardening
A 6-lens code audit (error propagation, exception safety, state machine, type system, test architecture, resource lifecycle) was run against the implementation. Key fixes applied:
except Exceptiontoexcept SolidLSPExceptioninresolve_neighborsso programming errors (KeyError, AttributeError from malformed LSP responses) surface instead of being silently swallowedstart_cursor— raisesValueErrorinstead of silently overwritingTest plan
test/serena/test_cursor.py)test/serena/test_cursor_integration.py)SerenaAgent.get_tool()(test/serena/test_cursor_navigation.py)🤖 Generated with Claude Code