Empty column loses its dtype under numpy format (always float32)

[shashvat-singham]

Describe the bug

Under numpy format, a column that holds zero rows comes back as float32 no matter what dtype it was declared with. The same column keeps its dtype as soon as it holds a single row, and the pandas and arrow formatters keep it in both cases.

This is easy to hit in normal code: any filter that matches nothing, or select([]).

Steps or code to reproduce the bug

import numpy as np
from datasets import Dataset, Features, Value

feats = Features({"a": Value("int64")})
ds = Dataset.from_dict({"a": [1, 2, 3]}, features=feats).with_format("numpy")

matched   = ds.filter(lambda x: x["a"] > 1)[:]["a"]
unmatched = ds.filter(lambda x: x["a"] > 99)[:]["a"]

print(matched.dtype)    # int64
print(unmatched.dtype)  # float32

Across dtypes:

declared	non-empty	empty
int64	int64	float32
int32	int64	float32
float64	float32	float32
bool	bool	float32
string	<U1	float32

Only the numpy formatter is affected:

   pandas : int64
   arrow  : int64
   numpy  : float32

Why it matters

The dtype silently changes based on how many rows survived a filter, so a numeric pipeline that concatenates per-batch results gets its integers promoted the moment one batch is empty:

np.concatenate([unmatched, matched]).dtype   # float64, from int64 data

A string column coming back as float32 is the same bug wearing a more obvious hat.

Root cause

NumpyArrowExtractor._arrow_array_to_numpy goes through a Python list:

https://github.com/huggingface/datasets/blob/main/src/datasets/formatting/formatting.py#L188

array: list = pa_array.to_numpy(zero_copy_only=zero_copy_only).tolist()

.tolist() drops the arrow type. For a non-empty list numpy re-infers it correctly; for [] it falls back to float64. NumpyFormatter._tensorize then applies its float default on top:

elif isinstance(value, np.ndarray) and np.issubdtype(value.dtype, np.floating):
    default_dtype = {"dtype": np.float32}

which turns the accidental float64 into float32. Confirmed by tracing an empty int64 column:

arrow schema      : a: int64
arrow col to_numpy: int64      <- still correct here
extract_column    : float64    <- lost by .tolist()
formatted         : float32

To be clear, int32 -> int64 and float64 -> float32 on the non-empty rows are the formatter's documented default_dtype behaviour and not part of this report.

Expected behavior

An empty column should keep the dtype the same column has when non-empty, as it already does under pandas and arrow.

PR: #8470.

Environment info

• datasets version: 5.0.2.dev0 (main @ 48b7ee7)
• Python version: 3.11.9
• Platform: Windows 11
• PyArrow version: 25.0.1
• Pandas version: 3.0.5
• NumPy version: 2.4.6