Pandas version checks
Reproducible Example
import pandas as pd
from io import BytesIO
# integer converter stops at the NUL and accepts the truncated prefix
pd.read_csv(BytesIO(b'a\n1\n"1\x00xyz"\n3\n'))["a"].tolist()
# c engine: [1, 1, 3] dtype int64
# python engine: ['1', '1\x00xyz', '3'] dtype str
# same for bool
pd.read_csv(BytesIO(b'a\nTrue\n"True\x00xyz"\nFalse\n'))["a"].tolist()
# c engine: [True, True, False] dtype bool
# python engine: ['True', 'True\x00xyz', 'False'] dtype str
# and for the nullable boolean path
pd.read_csv(BytesIO(b'a\n1\x00x\nTrue\n'), dtype="boolean")["a"].tolist()
# c engine: [True, True]
# python engine: raises ValueError: 1\x00x cannot be cast to bool
Issue Description
A field containing an embedded NUL byte is parsed only up to that NUL by the c engine's numeric and boolean converters, so "1\x00xyz" is accepted as the integer 1 and "True\x00xyz" as True. The trailing bytes are silently discarded and no error is raised, so a column of junk can be read as a clean numeric or boolean column.
engine="python" rejects these values, leaving the column as strings (or raising, for dtype="boolean"), so the two engines disagree.
The cause is that str_to_int64 / str_to_uint64 in pandas/_libs/src/parser/tokenizer.c finish on *endptr rather than checking that the parse consumed the whole token, to_boolean compares with strcasecmp, and _bool_numeric_literal in pandas/_libs/parsers.pyx makes the same NUL-terminated assumption. Each token's exact length is already available at all of these call sites via _token_len.
Note these are entangled and should be fixed together: to_boolean backs the default bool path while _bool_numeric_literal backs dtype="boolean", so fixing only one leaves the two disagreeing with each other. In particular _bool_numeric_literal lives in parsers.pyx, not tokenizer.c, so a fix scoped to the C tokenizer alone would miss it.
Not affected: the float path. "1.5\x00xyz" reads as 1.5 on both engines, because lib.maybe_convert_numeric truncates the same way, so there is no divergence to fix there.
This is a sibling of GH-19886 and GH-34551 (same root cause: values handled as NUL-terminated C strings). It was deliberately left out of GH-66511 / GH-66512 / GH-66513 to keep those focused.
Expected Behavior
"1\x00xyz" should not parse as an integer and "True\x00xyz" should not parse as True. The c engine should agree with engine="python".
Installed Versions
Details
Reproduced on main at 7986b42596f (3.1.0.dev0).
Pandas version checks
Reproducible Example
Issue Description
A field containing an embedded NUL byte is parsed only up to that NUL by the
cengine's numeric and boolean converters, so"1\x00xyz"is accepted as the integer1and"True\x00xyz"asTrue. The trailing bytes are silently discarded and no error is raised, so a column of junk can be read as a clean numeric or boolean column.engine="python"rejects these values, leaving the column as strings (or raising, fordtype="boolean"), so the two engines disagree.The cause is that
str_to_int64/str_to_uint64inpandas/_libs/src/parser/tokenizer.cfinish on*endptrrather than checking that the parse consumed the whole token,to_booleancompares withstrcasecmp, and_bool_numeric_literalinpandas/_libs/parsers.pyxmakes the same NUL-terminated assumption. Each token's exact length is already available at all of these call sites via_token_len.Note these are entangled and should be fixed together:
to_booleanbacks the defaultboolpath while_bool_numeric_literalbacksdtype="boolean", so fixing only one leaves the two disagreeing with each other. In particular_bool_numeric_literallives inparsers.pyx, nottokenizer.c, so a fix scoped to the C tokenizer alone would miss it.Not affected: the float path.
"1.5\x00xyz"reads as1.5on both engines, becauselib.maybe_convert_numerictruncates the same way, so there is no divergence to fix there.This is a sibling of GH-19886 and GH-34551 (same root cause: values handled as NUL-terminated C strings). It was deliberately left out of GH-66511 / GH-66512 / GH-66513 to keep those focused.
Expected Behavior
"1\x00xyz"should not parse as an integer and"True\x00xyz"should not parse asTrue. Thecengine should agree withengine="python".Installed Versions
Details
Reproduced on main at
7986b42596f(3.1.0.dev0).