feat: typed parameters, and adopt integer types the server reports - #4488
Conversation
Every JS number is bound as MYSQL_TYPE_DOUBLE. MySQL rejects a DOUBLE where the statement needs an integer, so `LIMIT ?` and `LIMIT ? OFFSET ?` fail with ER_WRONG_ARGUMENTS for the most natural call a user can write. Since MySQL 8.0.22 a prepared statement is no longer re-prepared when a later execution changes a parameter type, so the first type a cached statement sees decides how every later execution is read. Binding a number once fixes the position as DOUBLE, and an exact BIGINT string bound afterwards is silently rounded through it. These tests fail on MySQL 8.3 and 9.7 and pass on MariaDB, which accepts a DOUBLE for LIMIT and re-optimises each execution. Refs #1239, #1407
Two changes to how a bind parameter picks its MySQL type.
TypedParameter carries the type alongside the value and is accepted
anywhere execute() takes one:
conn.execute('SELECT * FROM t LIMIT ?', [mysql.TypedParameter.BIGINT(10)])
It expresses what a JavaScript value cannot: integer width, signedness, and
binary versus text. The declared type is written into COM_STMT_EXECUTE, so
the unsigned flag is now sent rather than hardcoded to zero, and a typed
null keeps its declared type instead of collapsing to MYSQL_TYPE_NULL.
Values that do not fit are rejected at the call site, including numbers that
already lost precision. Types no server accepts as a bind type (MEDIUMINT,
YEAR, ENUM, SET, and JSON on MariaDB) travel as the nearest accepted type.
For untyped values the driver now adopts the type COM_STMT_PREPARE reports,
but only when that type is an integer type and the value is already an
integer that fits it. That is enough to make 'LIMIT ?' accept a number, and
to stop a cached statement being fixed as DOUBLE by its first execution.
Every other combination keeps the type inferred from JavaScript, so a bare
'SELECT ?' still round-trips a number as a number and server-side string
coercions are left alone. MariaDB reports MYSQL_TYPE_NULL for every
parameter, which fails the same gate, so nothing changes there.
Closes #1239
Refs #1407, #548, #446, #1483, #1173
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #4488 +/- ##
==========================================
+ Coverage 92.09% 92.29% +0.19%
==========================================
Files 92 93 +1
Lines 15214 15671 +457
Branches 2086 2190 +104
==========================================
+ Hits 14012 14464 +452
- Misses 1202 1207 +5
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…oders MySQL 5.7 answers VAR_STRING with a zero length for every parameter, the same non-answer MariaDB gives as MYSQL_TYPE_NULL. Neither reaches the integer gate, so nothing is adopted on either and no version check is needed, but the test asserting what the server reports assumed every MySQL resolves types. It now detects that capability instead of naming versions. Also formats the new docs page with the website's own prettier config, adds the 5.7 and MariaDB rows to it, and covers the remaining parameter encoders (FLOAT, DATE, TIMESTAMP, TIME in all four forms, length coded text, and the unsupported-type path), taking lib/packets/typed_parameter.js from 92% to 99% statement coverage.
|
Pushed 7e4fc7e to fix the red CI.
MySQL 5.7 — the interesting one. 5.7 resolves no parameter types at all: it answers No production code changed for this. The placeholder fails gate 1 structurally, so nothing is adopted on 5.7 exactly as intended, and every behavioural test already passed there — only the test that documents what the server reports was wrong. It now detects that capability by preparing a known-integer statement rather than naming versions, which also covers proxies and forks. 5.7 does not need the adoption anyway: it pre-dates the 8.0.22 change and still re-prepares when a parameter type changes, so Also addressed the codecov report: added coverage for the remaining encoders (FLOAT, DATE, TIMESTAMP, TIME in all four input forms, length-coded text, and the unsupported-type path). Full suite now 231/231 on MySQL 5.7.44, 8.3.0 and MariaDB 12.3.2; 230/231 on 9.7 with the pre-existing 🤖 Addressed by Claude Code |
|
The implementation is mostly CC, but I reviewed it and it looks good. Would be good to have another review @wellwelwel when you have a chance, both in terms of surface API and internals. Surface API is probably even more important, once its in there is no way back |
Adds
TypedParameterfor stating a bind parameter's MySQL type explicitly, and makes the driver adopt the integer typesCOM_STMT_PREPAREalready reports.The first commit is the red baseline: tests reproducing the current failures, passing on MariaDB and failing on MySQL 8.3 and 9.7. The second makes them pass.
Background
Bind parameter types are inferred from the JavaScript value in
toParameter. That mapping arrived in #353 and #705, replacing an earlier design that sent everything as a string and let the server coerce.MySQL 8.0.22 then changed the server: a statement is prepared once at
PREPARErather than re-prepared per execution, so the parameter types of the first execution stick. This is #1239, open since October 2020 and the most-reported issue in this area. #1407 was an attempt to fix it by following the prepare response for every parameter; it stalled because the hint is not always meaningful, and the follow-up plan discussed there and in #1483 was a hybrid — explicit containers plus a safer default.Prior discussion: #1239, #1407, #353, #705, #548, #446, #1483, #1173, #1760, #348, and the closed duplicates #1789, #1623, #2793, #2302, #1521.
What is wrong today
LIMIT ?with a number is rejected. Every JS number is sent asDOUBLE, and MySQL requires an integer here.This is the single most reported symptom. Measured across every context I could find,
LIMIT/OFFSETis the only place aDOUBLEis refused outright — everywhere else the server coerces, which is why the remaining problems are silent.A cached statement is fixed by its first execution. Bind a number once and the position stays
DOUBLE, so an exactBIGINTbound later is read back through it:Comparing a number to an indexed string column loses the index, permanently. The server compares the column numerically, and the cached statement keeps that shape. On a 100k-row table:
Deterministically, via
Handler_read_nexton a 2000-row table: 1 row read per lookup before the number, 2000 after — for every later execution, including the correctly-typed ones. MariaDB re-optimises each execution so it recovers; MySQL does not.The unsigned flag is never sent — it is hardcoded to
0, so the top half ofBIGINT UNSIGNEDis unreachable — and binary versus text cannot be expressed, which is #1760 and the MariaDBVECTORcase.TypedParameterNamed after both the protocol types and their SQL spellings (
BIGINT/LONGLONG,INT/LONG,MEDIUMINT/INT24). Integer factories expose.unsigned. Values that do not fit are rejected at the call site:T.BIGINT(null)sends SQLNULLwhile keeping the declared type, so a position does not change type between null and non-null executions.Neither server accepts every type as a bind type — MySQL refuses
INT24 YEAR ENUM SET BIT GEOMETRY, MariaDB refusesJSON VECTOR— so declared types travel as the nearest type both accept (MEDIUMINT→LONG,YEAR→SHORT,ENUM/SET→STRING,VECTOR→BLOB,JSON→VAR_STRINGon MariaDB).BITandGEOMETRYget no factory, since neither has an unambiguous encoding; useT.BIGINT(mask)andT.BLOB(wkb).Integer types adopted from the prepare response
For untyped values the driver now consults the reported parameter type, but adopts it only when both hold:
TINY,SHORT,LONG,LONGLONG), andnumberthat is a safe integer,bigint, orboolean— that fits it.Anything else keeps today's inference. That is deliberately narrow, and each restriction is load-bearing:
SELECT ?andWHERE varchar_col = ?produce byte-identical hints (VAR_STRING, empty schema/table/orgTable, name?). AdoptingVAR_STRINGwould fix the index case but would also turnSELECT ?bound with42into the string"42", and the two are indistinguishable. That is the wall Change the way types for prepared statement parameters are calculated #1407 hit.1.5or'7abc'would turn a working call into a client-side error.MYSQL_TYPE_NULLfor every parameter (verified: 35/35 across 27 statements) and MySQL 5.7 answersVAR_STRINGwith a zero length for every parameter. Both fail gate 1 structurally, as would any other server or proxy that reports nothing. Neither needs the adoption anyway: both pre-date the 8.0.22 change and still re-prepare when a parameter type changes.This fixes
LIMIT ?/OFFSET ?with a number, and stops a cached statement being fixed asDOUBLEby an integer first execution — so theBIGINTrounding above no longer happens. It does not fix the index case, which still needs an explicitT.VARCHAR(...); that is a limit of what a provenance-free hint can safely support.Compatibility
Only MySQL 8.0 and later resolve parameter types at all:
VAR_STRING, length 0, charset 63 for every?MYSQL_TYPE_NULLfor every?What goes on the wire for each column type and value.
LONGLONGmarks where the hint is adopted; every other cell is unchanged from today.MySQL 9.7.2 (8.3.0 identical apart from
DATETIMEcharset)TINYINTSMALLINTMEDIUMINTINTBIGINTBIGINT UNSIGNEDYEARDECIMAL(20,2)FLOATDOUBLEVARCHAR(32)CHAR(8)TEXTVARBINARY(16)BLOBDATEDATETIME(6)TIME(6)TIMESTAMP(6)JSONENUMSETBIT(8)LIMIT ?OFFSET ?SELECT ?SELECT ? + ?WHERE i IN (?, ?)DATE_ADD(?, ...)Note
YEARandBIT: the server reports them but refuses them as bind types, so they are excluded from the adoptable set. IncludingYEARinitially causedINSERT INTO t (year_col) VALUES (?)to start failing withER_WRONG_ARGUMENTS; the test suite caught it.MariaDB 12.3.2 and MySQL 5.7.44 — no hint resolves, so every cell is inference, identical to today:
LIMIT ?)LIMIT ?)Tests
test/unit/packets/test-typed-parameter.test.mts— 71 assertions: wire encoding for every supported type, two's complement, the full unsigned 64-bit range, range and precision rejection, the wire-type mapping, typed null, and every gate of the hint policy including the placeholder types 5.7 and MariaDB report. 99% statement coverage oflib/packets/typed_parameter.js.test/integration/connection/test-execute-integer-parameters.test.mts— the red baseline from the first commit.test/integration/connection/test-typed-parameter.test.mts— round trips for exact/unsigned/negative 64-bit, binary, typed null, forced string.test/integration/connection/test-typed-parameter-plan.test.mts— index retention viaHandler_read_next, no timing.test/integration/connection/test-prepare-hint-fidelity.test.mts— what each server reports, the missing provenance, and the cases the driver deliberately ignores.Full suite: 231/231 on MySQL 5.7.44, 8.3.0 and MariaDB 12.3.2. On MySQL 9.7, 230/231 with one pre-existing failure (
test-execute-nocolumndef, which asserts 8.xEXPLAINmetadata and fails on master too).Notes for review
lib/constants/types.jsgains0xf2: 'VECTOR'in the reverse map. The named export andtypings/.../types.d.tsalready had it; only the number-to-name lookup was missing.TypedParameteralso givesmysql2an answer for Prepared statement integer arguments converted to string causes issues with BIT(n) #548, Problem usingexecuteand boolean parameters #446, If a BigInt/Int field exists on update, the record is not updated. #1483, JSON in prepared statement broken #1173 and Unable to query again using the geometry type obtained from the database #1760 at the call site.YEARis reported by MySQL but refused as a bind type, so including it in the adoptable set brokeINSERT INTO t (year_col) VALUES (?); it is now excluded. And MySQL 5.7 turned out to resolve no parameter types at all, which the capability check now handles without naming a version.