decimal: string conversion keeps mantissa digits and overflowed exponents MySQL discards

[arthurschreiber]

Not looking for external contributions on this one. We're already working in this area and intend to fix it ourselves. Please don't open a PR against this issue — it's filed to track the problem and record the analysis, not as an invitation. Comments and additional repro cases are very welcome.

Overview of the Issue

decimal.NewFromString keeps two things MySQL's string-to-decimal conversion discards, so a converted value can differ from MySQL's at the extremes of precision and exponent. Raised by @GrahamCampbell while reviewing #20721 (#20721 (comment)).

Mantissa digits past MySQL's decimal buffer. MySQL limits the mantissa to its decimal buffer before applying the exponent, while Vitess reads every digit into a big integer and lets the exponent shift them back into significance:

-- MySQL 8.0.46
SELECT CAST(CONCAT('0.', REPEAT('0', 71), '1e+73') AS DECIMAL(20,6));  -- 10.000000
SELECT CAST(CONCAT('0.', REPEAT('0', 72), '1e+74') AS DECIMAL(20,6));  -- 0.000000

Vitess returns 10 for both: the 73rd fraction digit survives in the big integer and the exponent shifts it back. The first spelling is what #20721 fixed (it used to be 0 for the wrong reason); the second is a divergence.

Exponents past what a uint64 holds. 1e+18446744073709551615 converts to the largest decimal in both MySQL and Vitess. One past that, MySQL's exponent wraps and the result is 0, while Vitess's fastparse.ParseInt64 saturates, the value clamps to ExponentLimit, and the overflow error is dropped by evalengine — so Vitess still returns the largest value.

Why it is out of #20721's scope

Both divergences predate that PR and exist on main through the spellings that carry no written plus — CAST(CONCAT('0.', REPEAT('0', 72), '1e74') AS DECIMAL(20,6)) and CAST('1e18446744073709551616' AS DECIMAL(20,6)) are 0 in MySQL and not in Vitess at any recent head, and the exponent clamp code is unchanged there. Accepting the plus spellings widened which texts reach the divergence; it did not create it.

Analysis

The suggested direction from review: a MySQL-compatible string-to-decimal coercion that applies the mantissa limit before the exponent and distinguishes uint64 exponent overflow from saturation, covering both string branches in evalToDecimal (plain text and JSON strings), while preserving prefix behaviour such as 1e+5x converting to 100000.

Open questions before it's real:

• Where exactly MySQL's mantissa boundary sits for each shape of number (the probes above pin one shape; str2dec's buffer accounting decides the general rule), and whether the limit belongs in NewFromString itself or in a separate conversion the evalengine calls.
• Whether the wrap-to-zero exponent behaviour is worth reproducing bit-for-bit or is better served by an error, given that evalengine currently drops the conversion error either way.

Reproduction Steps

-- MySQL 8.0.46
SELECT CAST(CONCAT('0.', REPEAT('0', 72), '1e+74') AS DECIMAL(20,6));   -- 0.000000
SELECT CAST('1e+18446744073709551616' AS DECIMAL(20,6));                -- 0.000000

Vitess returns 10.000000 and 99999999999999.999999 for these.

Binary Version

arthur/decimal-exponent-sign @ 4a7808a210 (PR #20721)

Operating System and Environment details

Linux aarch64; not environment-specific

Log Fragments

n/a - no failure occurs; the conversions silently disagree with MySQL