Skip to content

fix(tool): retry a rejected PCRE pattern as a literal in code_search - #1351

Open
LiRunGuo wants to merge 2 commits into
alibaba:mainfrom
LiRunGuo:fix/code-search-literal-fallback
Open

LiRunGuo wants to merge 2 commits into
alibaba:mainfrom
LiRunGuo:fix/code-search-literal-fallback

Conversation

@LiRunGuo

@LiRunGuo LiRunGuo commented Sep 17, 2026 •

Copy link
Copy Markdown

Description

code_search hands the model's search_text straight to git grep -P when use_perl_regexp is true. If the term is not valid PCRE, git refuses to compile the pattern and exits 128 without searching anything:

$ git grep -P -e 'Set("gorm:'
fatal: -e option, 'Set("gorm:': missing closing parenthesis

#1340 measured 31 of 348 and 11 of 236 code_search calls failing this way in single runs (about 9%). Each failure costs the model a round trip, and the model usually falls back to reading files in chunks — the term is almost always code text (Set("gorm:) that merely happens to be unbalanced as a regex.

This PR retries the same search as a fixed string when git rejects the pattern, keeping every other argument (case sensitivity, pathspec, ref, --untracked / --no-index), and opens the result with a note that repeats git's reason:

Note: the pattern is not valid PCRE syntax (missing closing parenthesis), so it was searched as a literal string instead; the matches below are literal, not regex, matches.
File: gorm.go
Match lines: 1
4|	db.Set("gorm:save_associations", false)

Three points about the design:

  • Detection is git's own marker, not a heuristic. builtin/grep.c registers the origin string "-e option" for a pattern passed with -e, and grep.c's compile_regexp_failed() renders the failure as <origin>, '<pattern>': <reason> from a literal format string that is not translated. I checked the marker against git source at v2.20.0, v2.30.0, v2.39.0, v2.45.0 and v2.50.0 — it is unchanged. It is also specific: git diagnoses an unresolvable ref (unable to resolve revision) and an invalid pathspec (Invalid pathspec magic) before it compiles patterns, so this marker only ever appears when the pattern itself is the problem.
  • No failure becomes a different failure. The literal retry replaces the rejected pattern only when it produced an answer — matches, or git's own "no matches" exit (status 1, no output). If the retry fails on its own (a timeout, say), the original diagnosis is returned unchanged. A pattern PCRE accepts is never retried.
  • No matches found stays byte-identical when no fallback happened, and gains the note when it did, so the model learns its term was treated literally.

Second commit: doc drift found while rebasing

Rebasing onto main (now at 189be5b, which includes #1306) was a small merge: #1306 moved the truncation note to after the parse loop, and this PR's literal note now sits directly above it, so both notes can appear in one result.

Reviewing that merge surfaced stale documentation. #1306 changed the limit from per-file --max-count 100 to a global 100-match cap with a --max-count 101 per-file probe, and reworded the note — but the tools guide in all five locales (pages/src/content/docs/*/tools.md) and the code_search description in internal/config/toolsconfig/tools.json still quoted the old per-file wording and the old note text. The second commit brings those six files in line with the merged behavior.

Type of Change

  • Bug fix (non-breaking change that fixes an issue)

How Has This Been Tested?

  • make test passes locally (race, all packages)
  • make check passes (license headers, English-only check, go mod tidy, gofmt -s, go vet)
  • make coverage passes: above the 90% threshold; the new helpers are at 100%
  • New tests exercise the fallback against real git grep in workspace mode, commit mode (ref + pathspec carried into the retry), and a non-git directory (where the rejection only appears on the --no-index retry), plus no-match, valid-regex-stays-regex, non-pattern-errors-still-error, and table tests for the marker and reason parsing
  • Upstream's TestGitGrep_ResultLimit from fix(tool): enforce the global code search result limit #1306 still passes on the rebased tree; the retry inherits its --max-count handling unchanged
  • Manual check of the tool output shown above

TestGitGrep_PerlRegexp_InvalidPattern_ReturnsError asserted the old behavior (invalid pattern → error) and is replaced by the fallback tests.

Checklist

  • My code follows the project's coding style (go fmt, go vet)
  • I have performed a self-review of my code
  • I have added tests that prove my fix is effective or my feature works
  • New and existing unit tests pass locally with my changes
  • I have updated the documentation accordingly (if applicable)
  • I have signed the CLA
  • I did not use AI/LLM to create this PR, or I disclosed the tool/model below and reviewed its output; I did not attribute commits to AI and will answer maintainer questions and review comments myself without AI/LLM.

AI disclosure: the implementation, tests and documentation edits were drafted with Codex (GPT-5, OpenAI) and reviewed line by line before pushing; no commit trailer attributes it to a model. The zh/ja/ko/ru wording in pages/src/content/docs/*/tools.md was drafted the same way and is technically accurate, but I am not a native speaker of those languages — corrections are welcome. The CLA is not signed yet; I will sign as soon as the bot posts the link.

Related Issues

Closes #1340

One thing I deliberately left out: a git built without PCRE2 (-P unsupported). That failure has a different message and different semantics, and I could not reproduce it here.

@github-actions

Copy link
Copy Markdown
Contributor

✅ OpenCodeReview: Review complete: 0 finding(s) across 2 selected item(s).

@CLAassistant

Copy link
Copy Markdown

CLA assistant check
Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
You have signed the CLA already but the status is still pending? Let us recheck it.

@lizhengfeng101

Copy link
Copy Markdown
Contributor

@LiRunGuo Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

@lizhengfeng101

Copy link
Copy Markdown
Contributor

#1306 is merged, please rebase main

git grep refuses to run when -P cannot compile the pattern, so a term
like `Set("gorm:` failed the whole call and cost the model a round trip.
Retry it as a fixed string, keep the rest of the arguments, and say in
the result that the matches are literal.
@lizhengfeng101

Copy link
Copy Markdown
Contributor

@LiRunGuo ping again

The note was reworded when the global result limit landed (alibaba#1306), but
the tools guide and the tool description still quoted the old per-file
wording. Describe the per-file max-count 101 probe and the global 100
match cap as they now behave.
@LiRunGuo
LiRunGuo force-pushed the fix/code-search-literal-fallback branch from 5b73802 to aa545d9 Compare September 18, 2026 13:25
@LiRunGuo

Copy link
Copy Markdown
Author

Sorry for the delay, and thanks for the ping. Rebased onto main (189be5b, includes #1306) and pushed — the branch is aa545d9 and the PR is mergeable again.

Resolving the #1306 conflict was small: it moved the truncation note to after the parse loop, and the literal note from this PR now sits directly above it, so both can appear in one result. The retry inherits the new --max-count 101 handling unchanged, and upstream's TestGitGrep_ResultLimit still passes.

While reviewing that merge I noticed the docs had drifted from #1306's merged behavior — the limit is now a global 100-match cap with a per-file 101 probe, and the note was reworded, but pages/src/content/docs/*/tools.md (all five locales) and the code_search description in internal/config/toolsconfig/tools.json still quoted the old per-file wording and the old note text. I fixed those six files in a second commit (aa545d9) so the guide matches what the tool actually prints.

Re-verified after the rebase: make test (race, all packages), make check, and make coverage all pass.

On the CLA: I will sign it as soon as the link is available to me. CI is waiting on maintainer approval for the fork run.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

code_search fails with exit 128 when a regex-mode search term contains an unbalanced parenthesis

4 participants