evalengine accepts malformed charset introducer literals that MySQL rejects with ER_INVALID_CHARACTER_STRING

[arthurschreiber]

Overview of the Issue

MySQL validates string literals with a charset introducer (_utf16 X'...', _utf8mb4 X'...', …) for well-formedness in that character set, and rejects the whole statement with ER_INVALID_CHARACTER_STRING (error 1300) when the bytes are malformed. Vitess performs no such validation: introducerCast in the evalengine attaches the collation to whatever bytes were supplied, and evaluation proceeds best-effort (? substitution during conversion, or NULL from a failed CONVERT).

Found by differential testing against MySQL 8.4.11 while working on #20721:

Expression	MySQL 8.4	Vitess
CAST(_utf16 X'D800' AS DECIMAL(20,6))	ERROR 1300: Invalid utf16 character string: 'D800'	0.000000
CAST(_utf8mb4 X'C2' AS DECIMAL(20,6))	ERROR 1300: Invalid utf8mb4 character string: 'C2'	0.000000
CAST(_utf32 X'00110000' AS DECIMAL(20,6))	ERROR 1300: Invalid utf32 character string: '001100'	0.000000
CONVERT(_utf16 X'D800' USING latin1)	ERROR 1300: Invalid utf16 character string: 'D800'	NULL

This does not appear to be an intentional design decision from the original charset/collations work: the building blocks already exist (charset.Validate, sqlerror.ERInvalidCharacterString), and the evalengine already uses charset.Validate where MySQL-visible semantics demand it (CHAR(... USING cs) returns NULL for invalid output). Introducer literals are simply not wired up to it.

Notes:

• Severity is low since the divergence is "statement succeeds with a substituted value" vs "statement fails". A related worse symptom — unpaired UTF-16 surrogates in these literals hanging the evaluation goroutine via a non-advancing DecodeRune — was fixed separately in decimal: read numeric text the way MySQL does #20721.
• The gap is not specific to CAST; any expression over a malformed introducer literal shows it (SELECT _utf16 X'D800' already errors in MySQL). Introducer literals are just the main way for malformed bytes to enter the evalengine, since column data arrives pre-validated by MySQL.
• A fix would likely validate in the evalengine's introducer translation/eval (and mirror it in the compiled path), returning error 1300 to match MySQL.

Reproduction Steps

Against vtgate (any keyspace):

SELECT CAST(_utf16 X'D800' AS DECIMAL(20,6));  -- returns 0.000000
SELECT CONVERT(_utf16 X'D800' USING latin1);   -- returns NULL

Against MySQL 8.4:

SELECT CAST(_utf16 X'D800' AS DECIMAL(20,6));
-- ERROR 1300 (HY000): Invalid utf16 character string: 'D800'

Binary Version

main (v25.0.0-SNAPSHOT), also reproducible on earlier releases.

Operating System and Environment details

Any.

Log Fragments

n/a