BUG: datetimelike arithmetic landing on the NaT sentinel silently returns NaT

[jbrockmendel]

Summary

NaT is stored as INT64_MIN, so any datetimelike arithmetic whose result lands exactly on
INT64_MIN is indistinguishable from a missing value once stored. @cython.overflowcheck(True)
catches a sum that goes below INT64_MIN but not one that lands on it, so there is a
one-step hole at the bottom of the range: the step further out raises, the step onto the
sentinel silently returns NaT.

GH-66510 / GH-66520 closed this for parsing and construction. It is still open across
arithmetic, and it is not a single site — it recurs everywhere pandas does datetimelike
addition, and in a few places as a bare assert rather than a silent NaT.

Reproducible examples

import numpy as np
import pandas as pd
from pandas import Period, PeriodDtype, Timedelta, Timestamp
from pandas._libs.tslibs import iNaT

# 1. scalar Timestamp: one step out raises, landing on the sentinel does not
Timestamp.min - Timedelta(1, "ns")     # -> NaT
Timestamp.min - Timedelta(2, "ns")     # -> OutOfBoundsDatetime

# 2. same for the array forms
pd.DatetimeIndex([Timestamp.min]) - Timedelta(1, "ns")   # -> [NaT]
pd.DatetimeIndex([Timestamp.min]) - Timedelta(2, "ns")   # -> OverflowError

# 3. Timestamp +/- ndarray[timedelta64] wraps on the promotion to the finer unit
Timestamp("2500-01-01").as_unit("s") + np.array([1], "m8[ns]")   # -> 1915-06-14
# ... and in the addition itself
Timestamp("2000-01-01") + np.array([9 * 10**18], "m8[ns]")       # -> 1700-08-23

# 4. Timestamp - Timestamp trips a bare assert. Under `python -O` the assert
#    vanishes and you get a Timedelta whose _value is iNaT but which is not NaT,
#    and for which isna() is False.
Timestamp(iNaT + 1) - Timestamp(1)     # -> AssertionError
Timestamp(iNaT + 1) - Timestamp(2)     # -> OutOfBoundsDatetime("Result is too large...")

# 5. Timedelta, scalar and array
Timedelta.min - Timedelta(1, "ns")                  # -> NaT
Timedelta.min + np.array([-1], "m8[ns]")            # -> [NaT]
Timedelta(1, "ns") * (-(2**63))                     # -> AssertionError
Timedelta(1, "ns") * np.array([-(2**63)])           # -> [NaT]

# 6. Period ordinals
per = Period._from_ordinal(iNaT + 1, PeriodDtype("ns"))
per - 1                                             # -> NaT
per - Timedelta(1, "ns")                            # -> NaT

# 7. vectorized offsets
pd.Series([Timestamp(iNaT + 1)]) + pd.offsets.DateOffset(nanoseconds=-1)   # -> NaT
pd.DatetimeIndex([Timestamp(iNaT + 7 * 86400 * 10**9)]) + pd.offsets.Week(-1)      # -> [NaT]
pd.DatetimeIndex([Timestamp(iNaT + 86400 * 10**9)]) + pd.offsets.BusinessDay(-1)   # -> [NaT]

Expected behavior

A result landing on the sentinel should raise, the same way the step further out of bounds
already does. NaT should come back only when an operand was actually missing.

Inventory

Everything below reaches the sentinel. Grouped by mechanism, since the fix differs.

Scalar paths that return NaT

• _Timestamp.__add__ with a timedelta-like (timestamps.pyx)
• _binary_op_method_timedeltalike — an explicit if res == NPY_NAT: return NaT, with a
  standing TODO: more generally could do an overflowcheck in op? (timedeltas.pyx)
• Period._add_timedeltalike_scalar, Period._add_offset, and the integer branch of
  Period.__add__ — all three build an ordinal and hand it to the Period constructor, which
  renders iNaT as NaT (period.pyx)

Scalar paths that trip a bare assert

• _Timestamp.__sub__ with another Timestamp, and Timedelta * / / — both land on
  assert value != NPY_NAT in _timedelta_from_value_and_reso. This is the worst of the set:
  under python -O it produces a live object whose _value is iNaT but which is not NaT
  and for which isna() returns False.

Array paths that return NaT

• add_overflowsafe (np_datetime.pyx), which covers DatetimeArray/TimedeltaArray/
  PeriodArray add and subtract, and so DatetimeIndex/TimedeltaIndex/PeriodIndex/Series
• Timedelta +/-/*// with an ndarray, which defer to numpy
• _Timestamp.__add__/__sub__ with an ndarray in the tz-naive branch (raw self.asm8 + other,
  which also wraps when numpy promotes to the finer unit)

Vectorized offsets — raw numpy adds

• DatetimeArray._add_offset tz-aware fast path (datetimes.py)
• RelativeDeltaOffset._apply_array, Day._apply_array, Week._apply_array,
  BusinessDay._apply_array (offsets.pyx)
• Week._end_apply_index and BusinessDay._shift_bdays, which add inside a nogil loop and so
  need a checked_add plus a deferred raise rather than a wrapper

Where INT64_MIN is legitimate

Not every INT64_MIN is the sentinel. Period - Period and PeriodArray - PeriodArray return a
count of periods, which is multiplied by freq.base — not an ordinal, and INT64_MIN is a
valid answer there. Any guard added to add_overflowsafe therefore needs an opt-out for
_sub_periodlike (datetimelike.py), and the scalar Period.__sub__ count path must be left
alone. Worth calling out because no existing test covers it: a full pandas/tests run passes
with PeriodArray - PeriodArray regressed.

Exception types

The array paths raise OverflowError and the scalar datetime paths raise OutOfBoundsDatetime,
so "raise whatever the neighbour one step further out raises" keeps each operation
self-consistent even though the two families differ.

Timedelta * and / are the exception: the neighbour there raises a bare OverflowError from
an int64 cast, while TimedeltaIndex raises OutOfBoundsTimedelta. Those cannot both be
matched. The bare OverflowError is arguably its own bug — cf. GH-63275, where the same thing in
the Timedelta constructor was treated as one.

Adjacent, not sentinel-specific

• Timedelta(4, "ns") * np.array([2**62]) wraps to 0 and Timedelta(4, "ns") / np.array([1e-300])
  saturates to INT64_MAX, where TimedeltaIndex raises OutOfBoundsTimedelta.
• A zero divisor is not an overflow: Timedelta(4, "ns") / np.array([0]) gives ±inf, which
  truncates onto INT64_MIN. numpy and TimedeltaIndex both call that NaT, and any sentinel
  check has to exempt it or it will regress.

Performance

Routing an array path through add_overflowsafe costs about 6x a raw numpy add (1M elements,
macOS arm64): arr_M8 + np.timedelta64(1, "D") is 0.76 ms against 4.51 ms for
DatetimeArray + Timedelta on main. The work is memory-bound at roughly 0.75 ms per pass, so
that is per-element branching rather than bandwidth.

astype_overflowsafe already has the remedy for the analogous multiply — see its "Fast path: if
min/max are both within the overflow-safe range" block in np_datetime.pyx. The same applies to
addition: take min/max of both operands, and when neither contains NaT and the sums provably
stay inside the range, fall through to a plain left + right. That is two reduction passes plus
one add, and it would speed up the existing callers, not only any new ones. The bound has to
exclude INT64_MIN itself rather than merely staying inside int64.

[earaghbidikashani]

hi @jbrockmendel
I'd like to work on this one. Will fix the missing sentinel check in add_overflowsafe and the scalar path in timedeltas.pyx