BUG: memory_usage(deep=True) under-reports by ~4x for arrow-backed string Index once the engine is built

[jbrockmendel]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas. (only tested on main, at 5c2fc85)

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import sys

import numpy as np
import pandas as pd

N = 500_000
vals = np.array([f"{'x' * 32}{i:08d}" for i in range(N)], dtype=object)
idx = pd.Index(vals, dtype="str")

reported_before = idx.memory_usage(deep=True)
idx.get_loc(idx[0])  # builds and caches the engine
reported_after = idx.memory_usage(deep=True)

held = idx._engine.values  # object-dtype copy the engine retains
unaccounted = held.nbytes + sum(sys.getsizeof(s) for s in held)

print("reported growth  :", (reported_after - reported_before) // 1000, "kB")
print("engine object arr:", held.nbytes // 1000, "kB")
print("its PyUnicodes   :", sum(sys.getsizeof(s) for s in held) // 1000, "kB")
print("unaccounted      :", unaccounted // 1000, "kB")
print("held is a copy   :", held is not np.asarray(idx._values))

reported growth  : 16908 kB
engine object arr: 4000 kB
its PyUnicodes   : 40500 kB
unaccounted      : 44500 kB
held is a copy   : True

Issue Description

For an arrow-backed string Index, building the engine allocates an object-dtype
copy of the entire Index that the cached engine then holds for its lifetime, and
memory_usage(deep=True) accounts for none of it. Above, 16.9 MB is reported
where 61.4 MB was actually allocated — a 3.6x under-report.

This is not just the boxed strings being missed. There are two separate gaps:

1. IndexEngine.sizeof() never counts self.values. For an object-dtype
   Index that is harmless, because the engine's array is the Index's own array
   and _memory_usage already counted it. But for str/string[pyarrow],
   _get_engine_target() returns vals.astype(object)
   (base.py#L5394),
   so the engine holds a distinct 4 MB pointer array that nothing counts.

2. HashTable.sizeof() ignores its deep argument entirely. Every
   implementation in hashtable_class_helper.pxi.in computes
   overhead + for_flags + for_pairs and never looks at deep, so the 40.5 MB
   of PyUnicode objects is never counted at any setting. For object-dtype those
   strings are shared with the Index and counting them would double-count, but
   for arrow-backed string dtypes the engine owns them exclusively.

The docstring says deep=True includes "the memory consumption of underlying
objects referencing this Index (e.g., the characters of object-dtype values)",
and memory_usage deliberately adds self._engine.sizeof(deep=deep), so the
intent to count the engine is clearly there — it just counts the wrong thing.

Confirmed against RSS (psutil), 500k keys x 40 bytes, engine build only:

dtype	reported delta	actual RSS delta	engine
object	16.9 MB	16.9 MB	ObjectEngine
str	16.9 MB	69.4 MB	StringObjectEngine
string[python]	16.9 MB	8.4 MB	StringObjectEngine

Only the arrow-backed case is wrong: object matches exactly, and
string[python] shares _ndarray with the engine so nothing extra is
allocated (its RSS delta reads low only because the allocator reuses freed
pages).

It also propagates to frames — one .loc lookup on a string-indexed
DataFrame is enough:

df = pd.DataFrame({"a": np.arange(N)}, index=idx)
df.memory_usage(deep=True).sum()  # +16.9 MB reported, +69.2 MB actual

Since str is the default string dtype in pandas 3.x, this affects any
string-indexed object after its first label lookup.

Expected Behavior

memory_usage(deep=True) should account for the object-dtype array the engine
retains and, for dtypes where the engine exclusively owns the boxed values, for
those values too. Concretely, IndexEngine.sizeof could add self.values.nbytes
(plus lib.memory_usage_of_objects when deep=True and the array is not shared
with the Index), or HashTable.sizeof could honour deep.

Related: GH-58529 raises the same two mechanisms — _engine.values not being
counted and _engine.sizeof(deep=True) returning nothing useful — but frames it
for MultiIndex and as an enhancement. Filing separately because the
arrow-backed string case is a plain under-report on a default dtype rather than
a question about which caches to include; happy for them to be merged.

Installed Versions

Details

INSTALLED VERSIONS
------------------
commit                : 5c2fc85c910c54664fe615d449eb55d6f1e175c2
python                : 3.13.11
python-bits           : 64
OS                    : Darwin
OS-release            : 23.3.0
machine               : arm64
byteorder             : little
LC_ALL                : None
LANG                  : en_US.UTF-8
LOCALE                : en_US.UTF-8

pandas                : 3.1.0.dev0+1442.g5c2fc85c910
numpy                 : 2.4.4
dateutil              : 2.9.0.post0
pip                   : 26.2
Cython                : 3.2.4
pyarrow               : 23.0.1

[Shaurav-Vora]

Hi there! I would like to take a look at this issue if thats okay?