Improve symbol coverage and bound search scoring; require v2 cases

Author: ktappdev

AGENTS.md

@@ -142,7 +142,7 @@ Default ignores: `.venv`, `site-packages`, `__pycache__`, `node_modules`, `.env*
 6. **No log levels** — `LogInfof`/`LogWarnf`/`LogErrorf` always print; use stderr for warnings/errors
 7. **Watch behavior** — Snapshot persistence is shutdown-only (in-memory updates during runtime, flush on graceful shutdown)
 8. **Symbol-feeding** — Symbols extracted before synonyms, passed to LLM for domain-accurate generation
-9. **Bench case format** — v2 supports categories, v1 still works, `LoadCasesAuto` handles both
+9. **Bench case format** — version 2 with categories only; `LoadEvalCases` is shared by bench and eval
 10. **Scoring** — Exact basename +15 (highest), confidence gap truncation reduces noise
 11. **Hybrid search** — Use `--hybrid` on `search-hints` to fill gaps in index results. Ripgrep scans file contents for query tokens and merges unmatched files at score=1. Useful when the index misses files whose content literally contains the query words but whose symbols/synonyms don't connect. Add `--memory=false` if using a snapshot (non-watch) index.
 12. **`--agent` flag** — Blocks `init`/`watch`/`sync`/`clean` for safety in automated flows

README.md

@@ -48,36 +48,37 @@ Remote LLM endpoints must use HTTPS; HTTP is accepted only on loopback.
 
 `ctxt init` walks the filesystem tree and builds `.ctxt/ctx_index.json` — a single recursive JSON tree where every entry has `full_path`, `type`, optional `symbols`, optional `synonyms`, and nested `children` for directories.
 
-**Symbols:** Every source file gets scanned with language-specific extractors that pull out exported symbols — functions, types, variables, classes, constants. These become a `symbols` array on the file node. The default extractor is `auto` (tree-sitter with regex fallback). Supported languages: Go (go/parser), Python/JavaScript/TypeScript/Rust/Svelte/Astro (tree-sitter), Vue/Ruby (regex fallback).
+**Symbols:** Every source file gets scanned with language-specific extractors that pull out named declarations, including Go receiver methods — functions, types, variables, classes, constants. These become a `symbols` array on the file node. The default extractor is `auto` (tree-sitter with regex fallback). Supported languages: Go (go/parser), Python/JavaScript/TypeScript/Rust/Svelte/Astro (tree-sitter), Vue/Ruby (regex fallback).
 
 **Synonyms:** Each node (file or directory) gets 5–12 LLM-generated synonyms via batched API calls to an OpenAI-compatible endpoint. For example, `Skeletons.tsx → ["skeletons", "loading", "placeholder", "animation"]`. Names are smart-batched (up to 60 per request, 10 parallel by default). The prompt includes extracted symbols and, for JS/TS files, ESM imports to generate contextual terms. Use `--offline` to guarantee that no LLM requests are made.
 
 **Bootstrap diff:** On subsequent runs, `watch` diffs the filesystem against the existing snapshot using file modification times and sizes. Timestamp-preserving edits with a different size are detected; edits preserving both size and timestamp require a full rebuild. Deleted files are removed, new files are added, modified files are re-extracted. Watch bootstrap reuses cached synonyms; run `ctxt sync` to fill gaps and refresh changed symbol/import context. Synonym cache keys use the full project-relative path (e.g., `billing/routes/index.ts`) so identically named files in separate features remain distinct.
 
 ### Search mechanics
 
-When you run `search-hints "product review rating"`, the query is lowercased and split into tokens. Each token is scored against every node in the tree using matchers that each contribute points:
+When you run `search-hints "product review rating"`, the query is lowercased and split into tokens. Each token is scored against every node in the tree using the matchers below. Each token receives points for its strongest path match, strongest synonym match, and up to three distinct symbol matches, so repeated mentions of one concept cannot inflate a file’s score:
 
 | Matcher | Points | Description |
 |---------|--------|-------------|
 | exact | +12 | Token exactly matches a directory basename |
 | basename | +7 | Token matches the file's basename |
-| exact basename | +15 | Token exactly matches the full filename (with or without extension) |
+| exact basename | +15 | Entire query exactly matches the full filename (with or without extension) |
 | path segment | +4 | Token matches any segment in the full path |
 | segment prefix | +5 | Token is a prefix of a path segment |
 | syn-exact | +8 | Token exactly matches a synonym |
 | syn overlap | +5 | Token partially overlaps a synonym |
-| sym exact | +8 | Token exactly matches a symbol name |
+| sym exact | +24 | Token exactly matches a symbol name |
 | sym contains | +5 | Token is contained in a symbol name |
 | sym token | +4 | Token matches a camelCase/PascalCase split token in a symbol |
 
 The `--explain` flag reveals this breakdown:
 
 ```
-basename contains +7: skeleton + syn exact +8: loading + sym contains +5: PageSkeleton = total 148
+syn exact +8: loading
+sym contains +5: PageSkeleton
 ```
 
-Results are ranked by total score. Low-signal short/common words are filtered from query and synonym matching to reduce noise. Results are truncated at confidence gaps — if there's a 50%+ score drop between consecutive results, lower-scoring results are omitted even if `--limit` would include them. Exact filename/symbol lookups can return just one strong match; conceptual queries retain up to three alternatives. Zero-score candidates are never returned.
+Matching additional distinct query terms adds +4 per term after the first; generated stems do not count as extra query terms. Exact symbol matches outweigh three partial symbol matches. Results are ranked by total score. Low-signal short/common words are filtered from query and synonym matching to reduce noise. Results are truncated at confidence gaps — if there's a 50%+ score drop between consecutive results, lower-scoring results are omitted even if `--limit` would include them. Exact filename/symbol lookups can return just one strong match; conceptual queries retain up to three alternatives. Zero-score candidates are never returned.
 
 ### Watch mode
 
@@ -178,7 +179,7 @@ Benchmark Hit@1/3/5 + MRR from manual query cases.
 ctxt eval --cases ctx_cases.json --json
 ```
 
-Input format (v2 with categories, backward compatible with v1):
+Input format (version 2 with categories):
 ```json
 {
   "version": 2,
@@ -193,12 +194,7 @@ Input format (v2 with categories, backward compatible with v1):
 }
 ```
 
-v1 format (bare array) is also supported:
-```json
-[
-  {"query": "auth middleware", "expect_any": ["internal/auth/middleware.go"]}
-]
-```
+Bare arrays and other case versions are rejected.
 
 ### `ctxt bench`
 
@@ -316,7 +312,7 @@ search-hints → load .ctxt/ctx_index.json (or query live memory) → score toke
 - **`.ctxt/ctx_cache.json`** — versioned project-relative synonyms and symbol/import context fingerprints
 - **`.ctxt/ctx_config.toml`** — config-driven defaults
 - **`.ctxt/ctx_runtime.json`** — live watch discovery for memory search
-- **Bench/eval case files** — v2 format with categories (path-intent, symbol-lookup, concept-synonym, exact-file, narrow-scope, vague-intent); v1 bare array format is backward compatible
+- **Bench/eval case files** — v2 format with categories (path-intent, symbol-lookup, concept-synonym, exact-file, narrow-scope, vague-intent); requires version 2
 
 ## Project size guard
 

bench_results/003_symbol_coverage_ranking_2026-10-10.md

@@ -0,0 +1,58 @@
+# Symbol coverage and bounded ranking — 2026-10-10
+
+## Method
+
+Compared commit b9a3942 with the new extractor and scoring changes, using the
+same archived repository source and the existing 45 cases in
+`docs/bench_cases.json`. The baseline index used the previous extractor; the
+final index was rebuilt with the new extractor. Both used offline lexical
+synonyms, the ctxt engine, limit 10, and minimum score 1. No LLM calls were made.
+These cases informed scoring changes, so this is a regression check rather than
+an independent evaluation on unseen projects.
+
+## Changes
+
+- Include Go receiver methods alongside package declarations.
+- Include indented Python methods and named default JS/TS exports in regex mode.
+- Retain camelCase boundaries before splitting symbol names.
+- Award each token its strongest path match and strongest synonym match.
+- Cap symbol evidence at three distinct matches per token; exact symbol names
+  receive +24 plus a +4 token bonus, so they outweigh three partial symbol matches (+9 each at most).
+- Reward additional distinct query terms with +4 each; generated stems do not
+  count as extra terms.
+
+## Results
+
+| Metric | Before | After |
+| --- | ---: | ---: |
+| Hit@1 | 66.67% | 71.11% |
+| Hit@3 | 82.22% | 84.44% |
+| Hit@5 | 86.67% | 86.67% |
+| Recall at limit 10 | 86.67% | 88.89% |
+| Average noise ratio | 69.89% | 68.74% |
+| Average results | 6.44 | 6.40 |
+| Average estimated tokens | 28.51 | 25.62 |
+
+| Category | Hit@1 before | Hit@1 after |
+| --- | ---: | ---: |
+| concept-synonym | 50.00% | 37.50% |
+| exact-file | 100.00% | 100.00% |
+| narrow-scope | 57.14% | 85.71% |
+| path-intent | 87.50% | 87.50% |
+| symbol-lookup | 85.71% | 100.00% |
+| vague-intent | 14.29% | 14.29% |
+
+The correct file ranks first in 32/45 cases (previously 30), appears in the top
+three in 38/45 (previously 37), and appears within the limit in 40/45 (previously
+39). Symbol lookups improve from 6/7 to 7/7. Concept-synonym Hit@1
+regresses on this offline fixture; vague-intent Hit@1 stays at 14.29%; no broad claim about conceptual or LLM
+synonym quality is supported by this run.
+
+## Validation
+
+Existing Go tests and `go vet ./...` passed. Temporary CLI fixtures checked Go
+receiver methods and exclusion of local variables, Python methods and named
+JS/TS default exports in both auto and regex modes, bounded repetition of
+symbols/synonyms, exact versus partial symbol matches, camelCase matching, and
+stem coverage. Bench and eval both reject v1 bare arrays. Existing compatibility
+tests were updated or removed; no new test files were added.

bench_test.go

@@ -12,14 +12,14 @@ func buildFixtureBench(t *testing.T) (string, *ContextIndex) {
 	t.Helper()
 	root := t.TempDir()
 	files := map[string]string{
-		"src/audio/batch.go":         "package audio\nfunc BatchProcess() {}",
-		"src/audio/stream.go":        "package audio\nfunc Stream() {}",
-		"src/search/scoring.go":      "package search\nfunc Score() {}",
-		"src/cmd/register.go":      "package cmd\nfunc RegisterCommand() {}",
-		"src/ignore/rules.go":        "package ignore\nfunc Match() {}",
-		"node_modules/lib/index.js":  "module.exports = {}\n", // should be ignored
-		".git/config":                "[core]\n",
-		"README.md":                  "# bench fixture\n",
+		"src/audio/batch.go":        "package audio\nfunc BatchProcess() {}",
+		"src/audio/stream.go":       "package audio\nfunc Stream() {}",
+		"src/search/scoring.go":     "package search\nfunc Score() {}",
+		"src/cmd/register.go":       "package cmd\nfunc RegisterCommand() {}",
+		"src/ignore/rules.go":       "package ignore\nfunc Match() {}",
+		"node_modules/lib/index.js": "module.exports = {}\n", // should be ignored
+		".git/config":               "[core]\n",
+		"README.md":                 "# bench fixture\n",
 	}
 	for path, content := range files {
 		full := filepath.Join(root, filepath.FromSlash(path))
@@ -43,7 +43,7 @@ func buildFixtureBench(t *testing.T) (string, *ContextIndex) {
 func writeCasesBench(t *testing.T, cases []EvalCase) string {
 	t.Helper()
 	path := filepath.Join(t.TempDir(), "cases.json")
-	data, err := json.Marshal(cases)
+	data, err := json.Marshal(CaseFile{Version: 2, Categories: map[string]CaseCategory{"bench": {Cases: cases}}})
 	if err != nil {
 		t.Fatalf("marshal cases: %v", err)
 	}
@@ -548,7 +548,7 @@ func TestCategoryReportFallback(t *testing.T) {
 	_, index := buildFixtureBench(t)
 	cases := []EvalCase{
 		{Query: "batch audio processing", ExpectAny: []string{"src/audio/batch.go"}}, // No category
-		{Query: "scoring", ExpectAny: []string{"src/search/scoring.go"}}, // No category
+		{Query: "scoring", ExpectAny: []string{"src/search/scoring.go"}},             // No category
 	}
 	out := runBench(BenchInput{
 		Index:        index,

cases_v2_test.go

@@ -6,7 +6,7 @@ import (
 	"testing"
 )
 
-func TestLoadCasesV1Array(t *testing.T) {
+func TestLoadCasesRejectsBareArray(t *testing.T) {
 	content := `[
 		{"query": "test query", "expect_any": ["file1.go"]},
 		{"query": "another query", "expect_any": ["file2.go"]}
@@ -17,21 +17,9 @@ func TestLoadCasesV1Array(t *testing.T) {
 		t.Fatalf("write test file: %v", err)
 	}
 
-	cases, err := LoadCasesAuto(path)
-	if err != nil {
-		t.Fatalf("LoadCasesAuto error: %v", err)
-	}
-	if len(cases) != 2 {
-		t.Fatalf("expected 2 cases, got %d", len(cases))
-	}
-	if cases[0].Query != "test query" {
-		t.Fatalf("unexpected query: %s", cases[0].Query)
-	}
-	if cases[0].Description != "" {
-		t.Fatalf("expected empty description for v1, got %s", cases[0].Description)
-	}
-	if cases[0].Category != "" {
-		t.Fatalf("expected empty category for v1, got %s", cases[0].Category)
+	_, err := LoadEvalCases(path)
+	if err == nil {
+		t.Fatal("expected legacy bare array to be rejected")
 	}
 }
 
@@ -60,9 +48,9 @@ func TestLoadCasesV2Object(t *testing.T) {
 		t.Fatalf("write test file: %v", err)
 	}
 
-	cases, err := LoadCasesAuto(path)
+	cases, err := LoadEvalCases(path)
 	if err != nil {
-		t.Fatalf("LoadCasesAuto error: %v", err)
+		t.Fatalf("LoadEvalCases error: %v", err)
 	}
 	if len(cases) != 3 {
 		t.Fatalf("expected 3 cases, got %d", len(cases))
@@ -105,7 +93,7 @@ func TestLoadCasesV2MissingVersion(t *testing.T) {
 		t.Fatalf("write test file: %v", err)
 	}
 
-	_, err := LoadCasesAuto(path)
+	_, err := LoadEvalCases(path)
 	if err == nil {
 		t.Fatalf("expected error for missing version, got nil")
 	}
@@ -122,33 +110,16 @@ func TestLoadCasesV2EmptyCategories(t *testing.T) {
 		t.Fatalf("write test file: %v", err)
 	}
 
-	cases, err := LoadCasesAuto(path)
+	cases, err := LoadEvalCases(path)
 	if err != nil {
-		t.Fatalf("LoadCasesAuto error: %v", err)
+		t.Fatalf("LoadEvalCases error: %v", err)
 	}
 	if len(cases) != 0 {
 		t.Fatalf("expected 0 cases, got %d", len(cases))
 	}
 }
 
-func TestLoadCasesAutoDetectsV1(t *testing.T) {
-	content := `[{"query": "test", "expect_any": ["file.go"]}]`
-	tmpDir := t.TempDir()
-	path := filepath.Join(tmpDir, "cases.json")
-	if err := os.WriteFile(path, []byte(content), 0644); err != nil {
-		t.Fatalf("write test file: %v", err)
-	}
-
-	cases, err := LoadCasesAuto(path)
-	if err != nil {
-		t.Fatalf("LoadCasesAuto error: %v", err)
-	}
-	if len(cases) != 1 {
-		t.Fatalf("expected 1 case, got %d", len(cases))
-	}
-}
-
-func TestLoadCasesAutoDetectsV2(t *testing.T) {
+func TestLoadEvalCasesVersion2(t *testing.T) {
 	content := `{
 		"version": 2,
 		"categories": {
@@ -166,40 +137,14 @@ func TestLoadCasesAutoDetectsV2(t *testing.T) {
 		t.Fatalf("write test file: %v", err)
 	}
 
-	cases, err := LoadCasesAuto(path)
-	if err != nil {
-		t.Fatalf("LoadCasesAuto error: %v", err)
-	}
-	if len(cases) != 1 {
-		t.Fatalf("expected 1 case, got %d", len(cases))
-	}
-	if cases[0].Category != "test" {
-		t.Fatalf("expected category 'test', got %s", cases[0].Category)
-	}
-}
-
-func TestEvalCaseV1UnmarshalIgnoresNewFields(t *testing.T) {
-	content := `[{"query": "test", "expect_any": ["file.go"]}]`
-	tmpDir := t.TempDir()
-	path := filepath.Join(tmpDir, "cases.json")
-	if err := os.WriteFile(path, []byte(content), 0644); err != nil {
-		t.Fatalf("write test file: %v", err)
-	}
-
 	cases, err := LoadEvalCases(path)
 	if err != nil {
 		t.Fatalf("LoadEvalCases error: %v", err)
 	}
 	if len(cases) != 1 {
 		t.Fatalf("expected 1 case, got %d", len(cases))
 	}
-	if cases[0].Query != "test" {
-		t.Fatalf("unexpected query: %s", cases[0].Query)
-	}
-	if cases[0].Description != "" {
-		t.Fatalf("expected empty description, got %s", cases[0].Description)
-	}
-	if cases[0].Category != "" {
-		t.Fatalf("expected empty category, got %s", cases[0].Category)
+	if cases[0].Category != "test" {
+		t.Fatalf("expected category 'test', got %s", cases[0].Category)
 	}
 }

command_bench.go

@@ -91,7 +91,7 @@ Examples:
 				return fmt.Errorf("--cases is required")
 			}
 
-			cases, err := LoadCasesAuto(casesPath)
+			cases, err := LoadEvalCases(casesPath)
 			if err != nil {
 				return err
 			}

command_eval.go

@@ -26,14 +26,14 @@ Examples:
   ctxt eval --cases cases.json --json              JSON output`,
 		RunE: func(cmd *cobra.Command, args []string) error {
 			var absConfigPath string
-		if configPath != "" {
-			var cfgErr error
-			absConfigPath, cfgErr = filepath.Abs(configPath)
-			if cfgErr != nil {
-				return fmt.Errorf("resolve config path: %w", cfgErr)
+			if configPath != "" {
+				var cfgErr error
+				absConfigPath, cfgErr = filepath.Abs(configPath)
+				if cfgErr != nil {
+					return fmt.Errorf("resolve config path: %w", cfgErr)
+				}
 			}
-		}
-		cfg, err := LoadContextingConfig(absConfigPath)
+			cfg, err := LoadContextingConfig(absConfigPath)
 			if err != nil {
 				return err
 			}
@@ -63,7 +63,7 @@ Examples:
 			if err != nil {
 				return err
 			}
-			cases, err := LoadCasesAuto(casesPath)
+			cases, err := LoadEvalCases(casesPath)
 			if err != nil {
 				return err
 			}

command_mcp.go

@@ -472,7 +472,7 @@ Setup: Add to your AI client's MCP config:
 	cmd.Flags().StringVar(&flags.Model, "llm-model", "", "LLM model used for synonym generation")
 	cmd.Flags().StringVar(&flags.APIKey, "api-key", "", "LLM API key (falls back to config api_key_env, LLM_API_KEY, OPENROUTER_API_KEY)")
 	cmd.Flags().StringVar(&flags.Endpoint, "llm-endpoint", "", "LLM API endpoint URL")
-	cmd.Flags().IntVar(&flags.BatchSize, "batch-size", 0, "Names per LLM request (0 = send all, legacy option)")
+	cmd.Flags().IntVar(&flags.BatchSize, "batch-size", 0, "Names per LLM request (0 = automatic, up to 60)")
 	cmd.Flags().IntVar(&maxBatchSize, "max-batch-size", 0, "Maximum names per LLM request (0 = send all at once, default)")
 	cmd.Flags().IntVar(&flags.SynonymsPerName, "synonyms", defaultSynonyms, "Desired synonyms per name")
 	cmd.Flags().IntVar(&flags.SynonymsMin, "synonyms-min", 0, "Min synonyms per name (0 = use synonyms value)")

command_search.go

@@ -31,7 +31,7 @@ func newSearchCommand() *cobra.Command {
 
 How it works:
   • Your query is tokenized and matched against filenames, symbols (functions, classes, types), and synonyms
-  • Each match is scored by type: synonym=20, symbol=10, path=1
+  • Path, synonym, and symbol matches contribute bounded scores; matching more query terms adds a coverage bonus
   • Results are ranked by total score and returned
 
 Requires an existing index (run 'ctxt init .' first).

command_watch.go

@@ -391,7 +391,7 @@ Examples:
 	cmd.Flags().StringVar(&flags.Model, "llm-model", "", "LLM model used for synonym generation")
 	cmd.Flags().StringVar(&flags.APIKey, "api-key", "", "LLM API key (falls back to config api_key_env, LLM_API_KEY, OPENROUTER_API_KEY)")
 	cmd.Flags().StringVar(&flags.Endpoint, "llm-endpoint", "", "LLM API endpoint URL")
-	cmd.Flags().IntVar(&flags.BatchSize, "batch-size", 0, "Names per LLM request (0 = send all, legacy option)")
+	cmd.Flags().IntVar(&flags.BatchSize, "batch-size", 0, "Names per LLM request (0 = automatic, up to 60)")
 	cmd.Flags().IntVar(&maxBatchSize, "max-batch-size", 0, "Maximum names per LLM request (0 = send all at once, default)")
 	cmd.Flags().IntVar(&flags.SynonymsPerName, "synonyms", defaultSynonyms, "Desired synonyms per name")
 	cmd.Flags().IntVar(&flags.SynonymsMin, "synonyms-min", 0, "Min synonyms per name (0 = use synonyms value)")