Skip to content

BUG: remaining NaT-sentinel and out-of-bounds gaps in datetimelike arithmetic (GH-66510 follow-ups) #66549

Description

@jbrockmendel

Follow-ups found while fixing GH-66510 for the vectorized offset paths. Each is pre-existing on
main (verified at 082e7739) and out of scope for that PR, but they are the same family of bug:
arithmetic on the raw int64 datetime64 values either lands on the NaT sentinel or leaves the
representable range without being reported as such.

Filing as one issue because (1) and (2) each need their two halves fixed together, and the whole
set shares a root cause. Happy to split if that's easier to triage.

1. Adding a timedelta can silently produce NaT

NPY_NAT is INT64_MIN, so a result that lands exactly there is indistinguishable from NaT
downstream. add_overflowsafe checks for int64 overflow but not for this, and the scalar
_Timestamp.__add__ does not check either:

>>> pd.Timestamp.min + pd.Timedelta(-1, "ns")
NaT
>>> pd.DatetimeIndex([pd.Timestamp.min]) + pd.Timedelta(-1, "ns")
DatetimeIndex(['NaT'], dtype='datetime64[ns]', freq=None)
>>> pd.TimedeltaIndex([pd.Timedelta.min]) - pd.Timedelta(1, "ns")
TimedeltaIndex(['NaT'], dtype='timedelta64[ns]', freq=None)
>>> pd.PeriodIndex([pd.Period(ordinal=-2**63 + 1, freq="D")]) - 1
PeriodIndex(['NaT'], dtype='period[D]')

All four should raise OutOfBoundsDatetime / OutOfBoundsTimedelta. Both the scalar and the
array side need to change together — fixing only one would replace this with a scalar/array
disagreement.

2. Out-of-bounds month and quarter shifts raise OverflowError from the vendored C

shift_months, shift_quarters and SemiMonthOffset._apply_array build their results with
npy_datetimestruct_to_datetime, whose overflow check surfaces as a bare OverflowError naming
a C source line, where the scalar path raises OutOfBoundsDatetime:

>>> pd.DatetimeIndex([pd.Timestamp.max]) + pd.DateOffset(months=1)
OverflowError: Overflow occurred at .../pandas/_libs/src/vendored/numpy/datetime/np_datetime.c:512
>>> pd.Timestamp.max + pd.DateOffset(months=1)
OutOfBoundsDatetime: Cannot cast 2262-05-11 23:47:16.854775 to unit='ns' without overflow.

Same for MonthEnd, QuarterEnd, SemiMonthEnd, YearEnd.

Relatedly, because the month shift is materialized before the timedelta component is applied, a
composed offset whose final result is representable is rejected on the array path:

>>> pd.Timestamp.max + pd.DateOffset(months=1, days=-31)
Timestamp('2262-04-10 23:47:16.854775807')
>>> pd.DatetimeIndex([pd.Timestamp.max]) + pd.DateOffset(months=1, days=-31)
OverflowError: Overflow occurred at .../np_datetime.c:512

Fixing that means applying the two components in one pass so the out-of-range intermediate is
never stored.

3. SemiMonthOffset and shift_quarters truncate n to 32 bits

n is held in a C int, so a multiple of 2**32 silently becomes a no-op:

>>> dr = pd.date_range("2000-01-03", periods=1, unit="s")
>>> dr + pd.offsets.SemiMonthEnd(2**32)
DatetimeIndex(['2000-01-15'], dtype='datetime64[s]', freq=None)   # == SemiMonthEnd(0)
>>> dr + pd.offsets.QuarterEnd(2**32)
DatetimeIndex(['2000-03-31'], dtype='datetime64[s]', freq=None)   # == QuarterEnd(0)

BusinessDay and Week had the same bug; widening those to int64_t is what the GH-66510
follow-up PR does, and the same treatment applies here.

4. ccalendar truncates years above 2**31

dayofweek(int y, int m, int d), year_add_months and month_add_months take the year as a C
int, but npy_datetimestruct.year is int64_t. Reachable only at second resolution, where the
representable range runs to year ~2.9e11:

>>> big = pd.DatetimeIndex(np.array([np.iinfo(np.int64).max], dtype="M8[s]"))
>>> big[0]
Timestamp('292277026596-12-04 15:30:07')
>>> big + pd.DateOffset(months=1)
DatetimeIndex(['219250469-01-04 15:30:07'], dtype='datetime64[s]', freq=None)   # wrapped
>>> big[0].weekday()
OverflowError: value too large to convert to int

The weekday truncation also silently gives BusinessDay the wrong anchor for such dates, rather
than raising.

5. Minor: duplicated checked_* externs

portable.h's checked_add / checked_sub / checked_mul are declared in a cdef extern block
in tzconversion.pyx and again in offsets.pyx, with different signatures (int vs bint, with
and without noexcept nogil). Worth hoisting into a shared .pxd so they cannot drift.

Installed versions

main @ 082e7739 (3.1.0.dev0)

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions