fix(json): reject raw unpaired surrogates in string lexing instead of aborting

[bobzhang]

Fixes #4049

The bug

@json.parse crashed the process (a guard! abort, not a catchable ParseError) on strings mixing a raw lone trailing surrogate with any escape sequence — minimal repro: the 5-code-unit text " U+DC00 \n ". Root cause: lex_string_slow's flush sliced the pending run with the checked ctx.input[start:end], which aborts when a slice boundary lands on a trailing surrogate. Meanwhile the escape-free fast path silently accepted raw lone surrogates, producing ill-formed strings.

The fix: reject raw unpaired surrogates

Per the project's design direction — "we should maintain String unicode safe, so unsafe_to_char is indeed unsafe" — this is robustness/defense-in-depth: under the well-formedness invariant such input should not exist (a well-formed String cannot contain a raw lone surrogate), but unsafe code can manufacture one, and a clean, documented error beats a process abort or an ill-formed parse result. The string lexer now rejects raw unpaired surrogates with ParseError (InvalidChar, sensible position) on both the fast path (previous silent acceptance removed) and the slow path, and flush slices with the bounds-check-only view(start_offset~, end_offset~), so no abort path remains.

Well-formed pairs — raw astral characters, including next to escapes — parse exactly as before.

Scope

Raw (unescaped) surrogates only. Unpaired \uXXXX escape sequences (which currently manufacture ill-formed strings from valid ASCII input) are tracked in #4062 and fixed in a separate PR; rejection there aligns with serde_json (Python and JS accept lone surrogates, Go substitutes U+FFFD).

Tests

Deterministic regressions in json/lex_string_test.mbt: escape-free raw lone surrogates (leading, trailing, reversed pair), raw lone surrogates next to escapes (the former abort), the exact ParseError shape, and still-accepted well-formed pairs with a stringify/parse roundtrip.

Verified: moon check clean; json suite green on wasm-gc, js, native; no .mbti changes. The adversarial QuickCheck suite that found the bug is #4045.

🤖 Generated with Claude Code

[Copilot]

Pull request overview

This pull request fixes two JSON lexer/parser correctness/robustness issues: (1) preventing a process abort when parsing strings containing lone surrogates combined with escapes, and (2) preserving IEEE-754 negative zero for the literal -0 so it matches other negative-zero spellings.

Changes:

• Update lex_string_slow to slice pending string runs using StringView::view(...) (bounds-check-only) to avoid aborting on surrogate slice boundaries.
• Update the safe-integer fast path in lex_number_end to apply the sign after converting to Double, preserving -0.0 for -0.
• Add deterministic regression tests for both issues.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.

File	Description
json/lex_string.mbt	Avoids abort on lone-surrogate boundaries in the slow string lexing path by using view(...) slicing.
json/lex_string_test.mbt	Adds regression coverage for lone surrogate + escape combinations to ensure parsing succeeds (no abort).
json/lex_number.mbt	Preserves the sign bit for -0 by applying negation after Int64 -> Double conversion in the safe-integer fast path.
json/lex_number_test.mbt	Adds regression coverage asserting sign-bit preservation across multiple negative-zero spellings and underflow-to-zero cases.

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

[coveralls]

Coverage Report for CI Build 6071

Warning

Build has drifted: This PR's base is out of sync with its target branch, so coverage data may include unrelated changes.
Quick fix: rebase this PR. Learn more →

Coverage increased (+0.007%) to 90.672%

Details

• Coverage increased (+0.007%) from the base build.
• Patch coverage: 16 of 16 lines across 1 file are fully covered (100%).
• No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.

---

Coverage Stats

Relevant Lines:	17657
Covered Lines:	16010
Line Coverage:	90.67%
Coverage Strength:	150734.89 hits per line

---

💛 - Coveralls

[bobzhang]

Independent review by OpenAI Codex CLI (codex-cli 0.147.0, read-only session; posted on its behalf):

---

No blocking findings.

• The string fix is correct: view preserves raw UTF-16 code units while still checking bounds. Lexer offsets always fall after the opening quote/escape or before an ASCII quote/backslash, so this cannot split a valid raw surrogate pair. Control-character and escape validation remain unchanged.
• Coverage includes both surrogate kinds before and after escapes, including the previously crashing lone-low-surrogate boundaries.
• Applying the sign after exact Int64→Double conversion correctly produces -0.0 and is equivalent for every nonzero integer in the safe range. Larger integers and decimal/exponent paths remain untouched.
• Negative-zero tests inspect the sign bit directly and cover integer, fractional, exponent, and underflow spellings. Existing tests cover the 2^53 slow-path boundary; adding explicit ±(2^53-1) cases would be optional rather than required.
• The changes are minimal, idiomatic, and introduce no API or representation changes.

Verdict: LGTM