Follow-ups found while fixing GH-66510 for the vectorized offset paths. Each is pre-existing on
main (verified at 082e7739) and out of scope for that PR, but they are the same family of bug:
arithmetic on the raw int64 datetime64 values either lands on the NaT sentinel or leaves the
representable range without being reported as such.
Filing as one issue because (1) and (2) each need their two halves fixed together, and the whole
set shares a root cause. Happy to split if that's easier to triage.
1. Adding a timedelta can silently produce NaT
NPY_NAT is INT64_MIN, so a result that lands exactly there is indistinguishable from NaT
downstream. add_overflowsafe checks for int64 overflow but not for this, and the scalar
_Timestamp.__add__ does not check either:
>>> pd.Timestamp.min + pd.Timedelta(-1, "ns")
NaT
>>> pd.DatetimeIndex([pd.Timestamp.min]) + pd.Timedelta(-1, "ns")
DatetimeIndex(['NaT'], dtype='datetime64[ns]', freq=None)
>>> pd.TimedeltaIndex([pd.Timedelta.min]) - pd.Timedelta(1, "ns")
TimedeltaIndex(['NaT'], dtype='timedelta64[ns]', freq=None)
>>> pd.PeriodIndex([pd.Period(ordinal=-2**63 + 1, freq="D")]) - 1
PeriodIndex(['NaT'], dtype='period[D]')
All four should raise OutOfBoundsDatetime / OutOfBoundsTimedelta. Both the scalar and the
array side need to change together — fixing only one would replace this with a scalar/array
disagreement.
2. Out-of-bounds month and quarter shifts raise OverflowError from the vendored C
shift_months, shift_quarters and SemiMonthOffset._apply_array build their results with
npy_datetimestruct_to_datetime, whose overflow check surfaces as a bare OverflowError naming
a C source line, where the scalar path raises OutOfBoundsDatetime:
>>> pd.DatetimeIndex([pd.Timestamp.max]) + pd.DateOffset(months=1)
OverflowError: Overflow occurred at .../pandas/_libs/src/vendored/numpy/datetime/np_datetime.c:512
>>> pd.Timestamp.max + pd.DateOffset(months=1)
OutOfBoundsDatetime: Cannot cast 2262-05-11 23:47:16.854775 to unit='ns' without overflow.
Same for MonthEnd, QuarterEnd, SemiMonthEnd, YearEnd.
Relatedly, because the month shift is materialized before the timedelta component is applied, a
composed offset whose final result is representable is rejected on the array path:
>>> pd.Timestamp.max + pd.DateOffset(months=1, days=-31)
Timestamp('2262-04-10 23:47:16.854775807')
>>> pd.DatetimeIndex([pd.Timestamp.max]) + pd.DateOffset(months=1, days=-31)
OverflowError: Overflow occurred at .../np_datetime.c:512
Fixing that means applying the two components in one pass so the out-of-range intermediate is
never stored.
3. SemiMonthOffset and shift_quarters truncate n to 32 bits
n is held in a C int, so a multiple of 2**32 silently becomes a no-op:
>>> dr = pd.date_range("2000-01-03", periods=1, unit="s")
>>> dr + pd.offsets.SemiMonthEnd(2**32)
DatetimeIndex(['2000-01-15'], dtype='datetime64[s]', freq=None) # == SemiMonthEnd(0)
>>> dr + pd.offsets.QuarterEnd(2**32)
DatetimeIndex(['2000-03-31'], dtype='datetime64[s]', freq=None) # == QuarterEnd(0)
BusinessDay and Week had the same bug; widening those to int64_t is what the GH-66510
follow-up PR does, and the same treatment applies here.
4. ccalendar truncates years above 2**31
dayofweek(int y, int m, int d), year_add_months and month_add_months take the year as a C
int, but npy_datetimestruct.year is int64_t. Reachable only at second resolution, where the
representable range runs to year ~2.9e11:
>>> big = pd.DatetimeIndex(np.array([np.iinfo(np.int64).max], dtype="M8[s]"))
>>> big[0]
Timestamp('292277026596-12-04 15:30:07')
>>> big + pd.DateOffset(months=1)
DatetimeIndex(['219250469-01-04 15:30:07'], dtype='datetime64[s]', freq=None) # wrapped
>>> big[0].weekday()
OverflowError: value too large to convert to int
The weekday truncation also silently gives BusinessDay the wrong anchor for such dates, rather
than raising.
5. Minor: duplicated checked_* externs
portable.h's checked_add / checked_sub / checked_mul are declared in a cdef extern block
in tzconversion.pyx and again in offsets.pyx, with different signatures (int vs bint, with
and without noexcept nogil). Worth hoisting into a shared .pxd so they cannot drift.
Installed versions
main @ 082e7739 (3.1.0.dev0)
Follow-ups found while fixing GH-66510 for the vectorized offset paths. Each is pre-existing on
main(verified at082e7739) and out of scope for that PR, but they are the same family of bug:arithmetic on the raw int64 datetime64 values either lands on the
NaTsentinel or leaves therepresentable range without being reported as such.
Filing as one issue because (1) and (2) each need their two halves fixed together, and the whole
set shares a root cause. Happy to split if that's easier to triage.
1. Adding a timedelta can silently produce
NaTNPY_NATisINT64_MIN, so a result that lands exactly there is indistinguishable fromNaTdownstream.
add_overflowsafechecks for int64 overflow but not for this, and the scalar_Timestamp.__add__does not check either:All four should raise
OutOfBoundsDatetime/OutOfBoundsTimedelta. Both the scalar and thearray side need to change together — fixing only one would replace this with a scalar/array
disagreement.
2. Out-of-bounds month and quarter shifts raise
OverflowErrorfrom the vendored Cshift_months,shift_quartersandSemiMonthOffset._apply_arraybuild their results withnpy_datetimestruct_to_datetime, whose overflow check surfaces as a bareOverflowErrornaminga C source line, where the scalar path raises
OutOfBoundsDatetime:Same for
MonthEnd,QuarterEnd,SemiMonthEnd,YearEnd.Relatedly, because the month shift is materialized before the timedelta component is applied, a
composed offset whose final result is representable is rejected on the array path:
Fixing that means applying the two components in one pass so the out-of-range intermediate is
never stored.
3.
SemiMonthOffsetandshift_quarterstruncatento 32 bitsnis held in a Cint, so a multiple of2**32silently becomes a no-op:BusinessDayandWeekhad the same bug; widening those toint64_tis what the GH-66510follow-up PR does, and the same treatment applies here.
4.
ccalendartruncates years above2**31dayofweek(int y, int m, int d),year_add_monthsandmonth_add_monthstake the year as a Cint, butnpy_datetimestruct.yearisint64_t. Reachable only at second resolution, where therepresentable range runs to year ~2.9e11:
The weekday truncation also silently gives
BusinessDaythe wrong anchor for such dates, ratherthan raising.
5. Minor: duplicated
checked_*externsportable.h'schecked_add/checked_sub/checked_mulare declared in acdef externblockin
tzconversion.pyxand again inoffsets.pyx, with different signatures (intvsbint, withand without
noexcept nogil). Worth hoisting into a shared.pxdso they cannot drift.Installed versions
main@082e7739(3.1.0.dev0)