Extract reusable tree-sitter LSP framework from wat-lsp

[EmNudge]

Motivation

The wat-lsp project has matured with solid native/WASM dual-build support. To enable supporting additional tree-sitter languages in the future, we should split the project into a reusable LSP framework crate and a WAT language consumer crate. This maximizes code sharing so future languages get document management, debouncing, incremental parsing, protocol handling, and WASM support for free.

The current codebase already has good layering (~30% is framework-ready), but ~70% is WAT-specific and deeply entangled with WAT grammar node kinds, instruction semantics, and symbol structures.

Proposed Architecture: 3-Crate Workspace

wat-lsp/
├── Cargo.toml                    # [workspace] root
├── crates/
│   ├── ts-lsp/                   # Reusable tree-sitter LSP framework
│   │   ├── Cargo.toml
│   │   └── src/
│   │       ├── lib.rs
│   │       ├── language.rs       # Language trait (the key abstraction)
│   │       ├── server.rs         # LspServer<L> (generic tower-lsp Backend)
│   │       ├── wasm_host.rs      # WasmLspHost<L> (generic WASM wrapper)
│   │       ├── core/
│   │       │   ├── mod.rs
│   │       │   └── types.rs      # Position, Range, Diagnostic, HoverResult, CompletionItem, etc.
│   │       ├── ts_facade.rs      # Tree-sitter native/WASM abstraction
│   │       └── native/
│   │           ├── mod.rs
│   │           └── adapters.rs   # Core type → tower-lsp type conversions
│   │
│   └── wat/                      # WAT language implementation
│       ├── Cargo.toml
│       ├── build.rs              # Doc gen + grammar compilation
│       └── src/
│           ├── lib.rs            # WatLanguage struct + Language impl
│           ├── symbols.rs
│           ├── parser.rs
│           ├── utils.rs
│           ├── instruction_metadata.rs
│           ├── docs.rs
│           ├── symbol_lookup.rs
│           ├── tree_sitter_bindings.rs
│           ├── wast_parser.rs
│           ├── test_utils.rs
│           ├── diagnostics_core/   # All WAT semantic analysis
│           ├── diagnostics/        # Native diagnostic wrappers
│           └── features/           # All WAT feature implementations
│
├── src/
│   ├── main.rs                   # Binary: LspServer<WatLanguage>::run()
│   └── wasm/
│       └── api.rs                # #[wasm_bindgen] WatLSP wrapping WasmLspHost<WatLanguage>
├── grammars/                     # Stays at root (referenced by wat crate's build.rs)
├── packages/                     # Stays at root
└── benches/                      # Stays at root

The Language Trait

Defined in crates/ts-lsp/src/language.rs. This is the contract between framework and consumer.

pub trait Language: Send + Sync + 'static {
    type SymbolTable: Default + Clone + Send + Sync;

    // Identity
    fn name(&self) -> &str;
    fn diagnostic_source(&self) -> &str;
    fn completion_trigger_characters(&self) -> Vec<String>;
    fn signature_help_trigger_characters(&self) -> Vec<String>;

    // Tree-sitter setup (native)
    #[cfg(feature = "native")]
    fn ts_language(&self) -> tree_sitter::Language;
    #[cfg(feature = "native")]
    fn create_parser(&self) -> tree_sitter::Parser;

    // Symbol extraction
    fn extract_symbols(&self, tree: &Tree, source: &str) -> Result<Self::SymbolTable, String>;

    // Features (all use protocol-independent core types)
    fn hover(&self, doc: &str, symbols: &Self::SymbolTable, tree: &Tree, pos: Position) -> Option<HoverResult>;
    fn completion(&self, doc: &str, symbols: &Self::SymbolTable, pos: Position) -> Vec<CompletionItem>;
    fn definition(&self, ...) -> Option<Range> { None }         // optional
    fn references(&self, ...) -> Vec<Range> { vec![] }          // optional
    fn document_symbols(&self, ...) -> Vec<DocumentSymbolInfo> { vec![] }
    fn folding_ranges(&self, ...) -> Vec<FoldingRangeResult> { vec![] }
    fn signature_help(&self, ...) -> Option<SignatureHelp> { None }

    // Diagnostics
    fn immediate_diagnostics(&self, tree: &Tree, source: &str, symbols: &Self::SymbolTable) -> Vec<Diagnostic>;
    fn debounced_diagnostics(&self, ...) -> Vec<Diagnostic> { vec![] }  // optional

    // Rename (optional)
    fn prepare_rename(&self, ...) -> Option<Range> { None }
    fn validate_rename(&self, new_name: &str) -> Result<(), String> { Ok(()) }
}

WAT implements every method by delegating to its existing _core functions.

Generic Server (LspServer<L: Language>)

crates/ts-lsp/src/server.rs — The current main.rs Backend generalized. Handles:

• DashMap<String, (String, L::SymbolTable, Tree)> document state
• Debounced validation with cancellation (watch::channel)
• Incremental text editing + tree-sitter InputEdit
• All LanguageServer trait methods delegating to self.language.hover(), etc.
• Core type → tower-lsp type conversion via existing From impls
• Rename flow: prepare_rename → validate_rename → references → WorkspaceEdit

The main.rs binary becomes ~5 lines:

fn main() { ts_lsp::run_server(wat::WatLanguage::new()) }

What Moves Where

To ts-lsp (framework):

• src/core/types.rs — as-is
• src/ts_facade.rs — minus wat_language()/create_parser() (those go to wat)
• src/native/adapters.rs — as-is
• src/main.rs Backend logic → generalized into server.rs
• src/wasm/api.rs generic parts → wasm_host.rs
• src/utils.rs generic fns: position_to_byte, apply_text_edit, node_to_range, get_line_at_position
• node_copy!/node_clone! macros (tree-sitter abstraction, not WAT-specific)

To wat (language):

• Everything else: symbols.rs, parser.rs, utils.rs (WAT parts), instruction_metadata.rs, docs.rs, symbol_lookup.rs, tree_sitter_bindings.rs, wast_parser.rs, diagnostics_core/, diagnostics/, all features/

Stays in root crate:

• src/main.rs (thin binary)
• src/wasm/api.rs (thin #[wasm_bindgen] concrete wrapper)
• src/bin/*.rs (CLI tools)
• benches/, tests/

Migration Order (Incremental)

Each phase should compile and pass tests before proceeding.

Phase 1: Workspace skeleton

• Convert Cargo.toml to [workspace] with members
• Create crates/ts-lsp/ and crates/wat/ with minimal Cargo.toml
• Root crate depends on both; all code stays in root initially
• Verify cargo build --features native and cargo test --features native pass

Phase 2: Extract core types to ts-lsp

• Move src/core/types.rs → crates/ts-lsp/src/core/types.rs
• Move src/native/adapters.rs → crates/ts-lsp/src/native/adapters.rs
• Root and wat crate re-import from ts_lsp::core::types
• Add FoldingRangeResult/FoldingRangeKind to core types (currently in features/folding)

Phase 3: Extract ts_facade to ts-lsp

• Move src/ts_facade.rs → crates/ts-lsp/src/ts_facade.rs
• Move generic utils (position_to_byte, apply_text_edit, node_to_range, node_copy!/node_clone!) to ts-lsp
• Remove WAT-specific functions (wat_language, create_parser) from facade

Phase 4: Define Language trait in ts-lsp

• Add crates/ts-lsp/src/language.rs
• Just the trait definition, no implementors yet

Phase 5: Move WAT code to wat crate

• Largest phase — move module by module, testing after each:
  1. symbols.rs
  2. instruction_metadata.rs + docs.rs + build.rs
  3. parser.rs + symbol_lookup.rs
  4. utils.rs (WAT-specific parts)
  5. tree_sitter_bindings.rs + wast_parser.rs
  6. diagnostics_core/
  7. diagnostics/
  8. features/ (all of them)
  9. test_utils.rs

Phase 6: Implement Language for WatLanguage

• Create WatLanguage struct in crates/wat/src/lib.rs
• Implement trait by delegating to existing functions

Phase 7: Extract generic server

• Move Backend logic from src/main.rs → crates/ts-lsp/src/server.rs as LspServer<L>
• Replace all direct WAT calls with self.language.*() dispatch
• Root src/main.rs becomes LspServer::<WatLanguage>::run()

Phase 8: Extract generic WASM host

• Move generic WASM logic → crates/ts-lsp/src/wasm_host.rs as WasmLspHost<L>
• Root src/wasm/api.rs becomes thin #[wasm_bindgen] wrapper over WasmLspHost<WatLanguage>

Feature Flags

Propagate through workspace:

• ts-lsp: native (tower-lsp, tokio, tree-sitter, dashmap), wasm (wasm-bindgen, web-tree-sitter-sg, js-sys)
• wat: native → ts-lsp/native + wast + regex, wasm → ts-lsp/wasm
• Root: native → wat/native + ts-lsp/native + clap, wasm → wat/wasm + ts-lsp/wasm

Verification

1. cargo build --features native — native build compiles
2. cargo test --features native — all existing tests pass
3. cargo build --features wasm --target wasm32-unknown-unknown — WASM build compiles
4. Start native LSP and exercise hover, completion, definition, references, rename, diagnostics
5. Load WASM module in browser and verify hover/completion/diagnostics work