Skip to content

Fix masking for fill_value and missing_value for numpy 2.4.0 #304

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
valeriupredoi wants to merge 9 commits into
base: main
Choose a base branch
from
Open

Fix masking for fill_value and missing_value for numpy 2.4.0 #304

Show file tree
Hide file tree
Changes from 2 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
4c2e6d3
exhaustive handling of new numpy 2.4.0 behaviour
valeriupredoi Jan 7, 2026
571699a
add test
valeriupredoi Jan 7, 2026
9877dfc
account for unbroadcastable arrays
valeriupredoi Jan 20, 2026
cd09108
add test and fix previous test
valeriupredoi Jan 20, 2026
1f16283
chuck a value error instead
valeriupredoi Jan 20, 2026
f3cdeb2
add tests
valeriupredoi Jan 20, 2026
375dbab
Merge branch 'main' into fix_mask_equal_2
valeriupredoi Jan 26, 2026
f213da9
Merge branch 'main' into fix_mask_equal_2
valeriupredoi Feb 3, 2026
3453b21
Merge branch 'main' into fix_mask_equal_2
valeriupredoi Mar 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion activestorage/active.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -116,7 +116,8 @@ def hfix(x):
missing_value = ds.attrs.get('missing_value')
# see https://github.com/NCAS-CMS/PyActiveStorage/pull/303
if isinstance(missing_value, np.ndarray):
missing_value = missing_value[0]
if missing_value.size == 1:
missing_value = missing_value[0]
valid_min = hfix(ds.attrs.get('valid_min'))
valid_max = hfix(ds.attrs.get('valid_max'))
valid_range = hfix(ds.attrs.get('valid_range'))
Expand Down
10 changes: 8 additions & 2 deletions activestorage/storage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,10 +129,16 @@ def mask_missing(data, missing):
fill_value, missing_value, valid_min, valid_max = missing

if fill_value is not None:
data = np.ma.masked_equal(data, fill_value)
if isinstance(fill_value, np.ndarray) or isinstance(fill_value, list):
data = np.ma.masked_where(data == fill_value, data)
else:
data = np.ma.masked_equal(data, fill_value)

if missing_value is not None:
data = np.ma.masked_equal(data, missing_value)
if isinstance(missing_value, np.ndarray) or isinstance(missing_value, list):
data = np.ma.masked_where(data == missing_value, data)
Copy link
Collaborator

@davidhassell davidhassell Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Surely this only works when missing_value broadcasts to data, which in general it wont.

Copy link
Collaborator Author

@valeriupredoi valeriupredoi Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

well I'm running out of any valid numpy function that can mask with an array then 😁

Copy link
Collaborator Author

@valeriupredoi valeriupredoi Jan 7, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, I got distracted by the news, how do you even apply a missing_value mask array if it's not broadcastable? I am really not familiar with anything missing value that's dim > 1 (or not scalar) - please suggest code, buds 🍺

Copy link
Collaborator Author

@valeriupredoi valeriupredoi Jan 20, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@davidhassell sorry - only just now that I refactored - given this is for storage.py, and that's just an emulator of Reductionist, I decided to pop a ValueError instead of going through with your albeit solid suggestion to go through the individual missing values - you reckon that'll do it? 🍺

else:
data = np.ma.masked_equal(data, missing_value)

if valid_max is not None:
data = np.ma.masked_greater(data, valid_max)
Expand Down
22 changes: 22 additions & 0 deletions tests/unit/test_storage.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,28 @@
import activestorage.storage as st


def test_mask_missing():
"""Test mask missing."""
missing_1 = ([-900.], np.array([-900.]), None, None)
missing_2 = ([-900., 33.], np.array([-900., 33.]), None, None)
data_1 = np.ma.array(
[[[-900., 33.], [33., -900], [33., 44.]]],
mask=False,
fill_value=-900.0,
dtype=float
)
data_2 = np.ma.array(
[[[-900., 33.], [33., -900], [33., 44.]]],
mask=False,
fill_value=[-900.0, 33.],
dtype=float
)
res_1 = st.mask_missing(data_1, missing_1)
print(res_1)
res_2 = st.mask_missing(data_2, missing_2)
print(res_2)


def test_reduce_chunk():
"""Test reduce chunk entirely."""
rfile = "tests/test_data/cesm2_native.nc"
Expand Down
Loading