Skip to content

Add Eisel-Lemire decimal parsing fast path - #3957

Merged
bobzhang merged 3 commits into
mainfrom
perf/eisel-lemire
Aug 20, 2026
Merged

Add Eisel-Lemire decimal parsing fast path#3957
bobzhang merged 3 commits into
mainfrom
perf/eisel-lemire

Conversation

@Yu-zh

@Yu-zh Yu-zh commented Jul 30, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • add a correctly rounded Eisel-Lemire fast path to internal/strconv
  • pass JSON scanner mantissa/exponent data directly to the fast path
  • retain the existing Clinger path and exact Decimal fallback for ambiguous, truncated, subnormal, and out-of-range inputs
  • add differential correctness tests, Canada-style coordinate cases, and parsing/JSON benchmarks
  • record the Go 1.26.2 source adaptation in NOTICE

Motivation

Canada coordinate values such as -65.613616999999977 have 17-19 digit mantissas that exceed the exact Clinger limit. Nearly every coordinate therefore fell back to arbitrary-precision Decimal conversion, making decimal parsing the dominant JSON decode cost.

The new path handles those mantissas with cached powers and explicit ambiguity checks. Inputs that cannot be certified by the fast path continue through the existing exact conversion.

Performance

Re-measured after the rebase onto main, per backend (Apple Silicon macOS, moon bench, mean of 10×N runs):

parse_double, long mantissas (4,096 parses):

Target Before After Improvement
native 779.48 µs 196.98 µs 3.96x
wasm-gc 1.07 ms 425.94 µs 2.51x
js 8.79 ms 2.26 ms 3.89x

JSON long-mantissa array (10,000 numbers):

Target Before After Improvement
native 2.46 ms 832.45 µs 2.95x
wasm-gc 3.86 ms 1.39 ms 2.78x
js 23.72 ms 6.35 ms 3.73x

Existing integer and short-float paths remained within benchmark noise on all three backends (fast-digit, underscore, JSON int-array, and JSON float-array benches all within a few percent).

Validation

  • moon test internal/strconv json --target all: 228/228 passed on native, JS, Wasm, and Wasm-GC
  • full native suite: 6,835/6,835 passed
  • full JS suite: 6,878/6,878 passed
  • full Wasm-GC suite: 6,926/6,926 passed
  • moon check --target all --warn-list +73
  • moon info and moon fmt
  • git diff --check
  • no generated .mbti changes

A full moon test --target all run reached an unrelated host-environment failure in the builtin Wasm tests because wasi_snapshot_preview1.random_get was not callable. The affected internal/strconv and json packages pass independently on Wasm.

@Yu-zh
Yu-zh force-pushed the perf/eisel-lemire branch from d55deac to 8dd3284 Compare August 14, 2026 03:45
@coveralls

coveralls commented Aug 14, 2026

Copy link
Copy Markdown
Collaborator

Coverage Report for CI Build 6249

Coverage increased (+0.02%) to 90.764%

Details

  • Coverage increased (+0.02%) from the base build.
  • Patch coverage: 32 of 32 lines across 3 files are fully covered (100%).
  • No coverage regressions found.

Uncovered Changes

No uncovered changes found.

Coverage Regressions

No coverage regressions found.


Coverage Stats

Coverage Status
Relevant Lines: 18027
Covered Lines: 16362
Line Coverage: 90.76%
Coverage Strength: 310113.89 hits per line

💛 - Coveralls

@bobzhang

Copy link
Copy Markdown
Contributor

Rebased onto main (was 110 commits behind; clean rebase, no conflicts) and added property-based tests in 87dcf24:

  • internal/strconv: two quickcheck properties asserting that every value try_eisel_lemire64 accepts is bit-for-bit equal to the exact Decimal conversion (reached through parse_double by zero-padding the spelling past the 19-digit fast-path limit). One samples mantissas across all magnitudes and exponents overshooting the table range on both sides; the other clusters mantissas around powers of two, where rounding carries and halfway ambiguity live.
  • json: a spelling-independence property — plain, fraction-padded, and pointed spellings of the same value must parse to identical bits, with the padded spelling forcing the exact fallback that anchors the fast routes.

Validation: moon check, moon fmt, moon info (no .mbti drift), and the affected packages pass on native, js, and wasm-gc (266/266).

Codex CLI review (ultra reasoning effort) — sign-off, no findings

The Eisel–Lemire implementation matches its Go reference, preserves exact fallbacks, and integrates correctly with both parsers. All-target checks and the full 7,473-test suite pass.

Codex also ran an independent differential fuzz of the ported algorithm against CPython's float parsing — 1,000,000 random (mantissa, exponent, sign) cases biased across bit lengths and the full exponent range, 834,771 accepted by the fast path, zero mismatches — and confirmed the parser state handed to the fast path is exact whenever many_digits is false, with the JSON scanner preserving the same invariant.

🤖 Generated with Claude Code

@bobzhang
bobzhang marked this pull request as ready for review August 20, 2026 15:13
Copilot AI lite review requested due to automatic review settings August 20, 2026 15:13
Yu-zh and others added 3 commits August 20, 2026 23:17
Whenever try_eisel_lemire64 accepts an input it must return the correctly
rounded double, so both new strconv properties compare accepted results
against the exact Decimal conversion, reached through parse_double by
zero-padding the mantissa spelling past the 19-digit fast-path limit. One
draws mantissas across every magnitude; the other clusters them around
powers of two where rounding carries and halfway ambiguity live.

The json property pins the scanner wiring: plain, fraction-padded, and
pointed spellings of the same value must parse to the same bits, with the
padded spelling forcing the exact fallback that anchors the other routes.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This pull request introduces a correctly-rounded Eisel–Lemire fast path for decimal-to-binary64 parsing in the internal/strconv standard-library package, and wires JSON number lexing to feed pre-scanned mantissa/exponent/sign data directly into that fast path to avoid expensive arbitrary-precision Decimal fallback for common long-mantissa inputs.

Changes:

  • Add an Eisel–Lemire conversion implementation plus a generated cached-power table to internal/strconv, keeping NaN as a private “reject” sentinel so the exact Decimal fallback remains authoritative for ambiguous/edge cases.
  • Extend parse_double to try Eisel–Lemire when Clinger’s fast path doesn’t apply and the scanned digits were not truncated.
  • Update JSON number lexing to attempt Eisel–Lemire using lexer-scanned mantissa/exponent data; add targeted regression tests, property tests, and benchmarks for long-mantissa cases.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated no comments.

Show a summary per file
File Description
NOTICE Records adaptation provenance for the new Eisel–Lemire implementation and table from Go 1.26.2 sources.
json/number_quickcheck_test.mbt Adds a property test ensuring JSON number parsing is invariant to equivalent spellings (fast paths vs exact fallback).
json/number_bench_test.mbt Adds a benchmark case focused on long-mantissa JSON number arrays (Canada-style coordinates).
json/lex_number.mbt Routes JSON number lexing through Eisel–Lemire when Clinger doesn’t apply and digits aren’t truncated.
json/lex_number_test.mbt Adds regression tests validating exact bit patterns for specific 17–19 digit mantissa cases.
internal/strconv/strconv_eisel_lemire.mbt Implements try_eisel_lemire64 with explicit rejection of uncertifiable rounding cases to preserve correctness via fallback.
internal/strconv/strconv_eisel_lemire_wbtest.mbt Adds whitebox tests comparing accepted Eisel–Lemire results against exact Decimal conversion and covering boundaries.
internal/strconv/strconv_eisel_lemire_table.mbt Adds the generated cached-power table required by Eisel–Lemire for efficient scaling.
internal/strconv/strconv_double.mbt Integrates the Eisel–Lemire attempt into parse_double after Clinger’s fast path and before Decimal fallback.
internal/strconv/parse_double_bench_test.mbt Adds a long-mantissa parsing benchmark for parse_double.
internal/strconv/eisel_lemire_quickcheck_test.mbt Adds property tests ensuring any Eisel–Lemire-accepted value matches exact Decimal rounding (bit-for-bit).

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

@bobzhang
bobzhang enabled auto-merge (squash) August 20, 2026 15:20
@bobzhang
bobzhang merged commit e50d33d into main Aug 20, 2026
16 checks passed
@bobzhang
bobzhang deleted the perf/eisel-lemire branch August 20, 2026 15:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants