Skip to content

BUG: convert_dtypes raises OverflowError for integers outside the int64 range #66517

Description

@jbrockmendel

Pandas version checks

  • I have checked that this issue has not already been reported.

  • I have confirmed this bug exists on the latest version of pandas.

  • I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd

pd.Series([2**64], dtype=object).convert_dtypes()
# OverflowError: Python int too large to convert to C long

pd.Series([-(2**63) - 1], dtype=object).convert_dtypes()
# OverflowError: Python int too large to convert to C long

Issue Description

Series construction correctly infers object for an integer that no NumPy/nullable integer
dtype can hold:

>>> pd.Series([2**64]).dtype
dtype('O')

but convert_dtypes then tries to force that column to Int64 regardless, and overflows:

  File "pandas/core/internals/blocks.py", line 582, in convert_dtypes
    rbs.append(b.astype(dtype=dtype, squeeze=b.ndim != 1))
  File "pandas/core/dtypes/astype.py", line 82, in _astype_nansafe
    return dtype.construct_array_type()._from_sequence(arr, dtype=dtype, copy=copy)
  File "pandas/core/arrays/numeric.py", line 248, in _coerce_to_data_and_mask
    values = dtype_cls._safe_cast(values, dtype, copy=False)
  File "pandas/core/arrays/integer.py", line 65, in _safe_cast
    casted = values.astype(dtype, copy=copy)
OverflowError: Python int too large to convert to C long

lib.infer_dtype reports "integer" for these values, and the object-block conversion maps
that straight to Int64 without checking that the values actually fit. IntegerArray._safe_cast
first attempts astype(..., casting="safe"), which raises TypeError for object input, and the
fallback values.astype(dtype) is what raises OverflowError.

Note this is a plain OverflowError escaping a public API rather than a pandas error type, so
it is also not catchable via anything in pandas.errors.

Expected Behavior

convert_dtypes should leave the column as object when the values do not fit in the target
integer dtype, matching what construction already infers:

>>> pd.Series([2**64], dtype=object).convert_dtypes().dtype
dtype('O')

Raising a clear pandas-level error would also be defensible, but silently attempting Int64 and
surfacing a raw OverflowError is not.

Installed Versions

Details
INSTALLED VERSIONS
------------------
commit                : e68db09ecf6427d1b62e565bacf17f2e525a3032
python                : 3.13.11
python-bits           : 64
OS                    : Darwin
OS-release            : 23.3.0
pandas                : 3.0.5
numpy                 : 2.5.1
dateutil              : 2.9.0.post0

Also reproduced on main at 7986b42.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions