Conversation
Attributes are stored as JSON, but `json.dumps` quietly accepts two kinds of value that do not round-trip: non-string keys (written as strings) and NaN / infinite floats (written as non-standard NaN / Infinity literals). Add a policy for each, set with the config options `attributes.non_string_keys` and `attributes.non_finite_floats`, taking "allow", "warn" or "raise". Attributes are checked when metadata is serialized for writing, so reading existing documents is unaffected. The defaults keep today's behavior: non-string keys now emit a ZarrFutureWarning saying they will become an error, and non-finite floats are allowed without comment. "raise" opts in to the strict behavior now; "allow" is the fallback once the defaults change. Refs zarr-developers#1125 Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4400 +/- ##
==========================================
+ Coverage 94.36% 94.38% +0.02%
==========================================
Files 93 93
Lines 13142 13196 +54
==========================================
+ Hits 12401 12455 +54
Misses 741 741
🚀 New features to boost your workflow:
|
d-v-b
marked this pull request as ready for review
September 24, 2026 11:12
Contributor
Author
|
requesting review because this starts a deprecation path |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR makes our attributes API stricter about accepting input that requires implicit casting (non-string keys) and values that are invalid JSON (
NaN). The latter is load-bearing for xarray, so they will need to fix that behavior or use the configuration added in this PR to continue writing invalid JSON when we flip the switch on the default behaviorcc @keewis
🤖 AI text below 🤖
Refs #1125. This is the deprecation step toward the error that issue asks for.
Zarr stores attributes as JSON, but Python's
json.dumpsaccepts two kinds of value that don't survive the trip to disk and back:{1: "a"}) are written as strings. The array or group you wrote through still reports{1: 'a'}, but reopening it gives{'1': 'a'}.NaNand infinite floats are written as bareNaN/Infinityliterals, which aren't valid JSON. Strict JSON parsers in other languages reject them.The goal is to make the correct behavior the default, with a gradual deprecation and a way to keep the old behavior for anyone who needs it.
What this PR does
It adds two config options. Each takes
"allow","warn"or"raise":attributes.non_string_keys"warn"ZarrFutureWarningnames each offending key and says it will become an error. The value is still written as before.attributes.non_finite_floats"allow"With the defaults, the only runtime change is the new warning.
"raise"lets you opt in to the strict behavior now, and"allow"is the way back to the old behavior once the defaults change.The check runs when metadata is serialized for writing, in
to_buffer_dictonArrayV2Metadata,ArrayV3MetadataandGroupMetadata. That covers every way of setting attributes: at creation,update_attributes,attrs.put, andattrs[k] = v. Reading is not checked, so opening existing data that hasNaNin its attributes stays silent.Only values that
json.dumpscurrently writes without complaint are checked, at any depth inside the attributes. Keys of other types (tuples, objects) already makejson.dumpsraiseTypeError, and that doesn't change. Warnings and errors list every offending location, e.g.attributes['a'][1].Proposed deprecation schedule
non_finite_floatsto"warn"once xarray has a way offNaNliterals. xarray writes_FillValue: NaNinto attributes, and the last time zarr rejected those it was reverted within a day (convert inf, -inf, nan to JSON #3280). That's why this option starts at"allow"rather than"warn"."raise", with"allow"kept as the fallback.Existing behavior left alone
These are unchanged here, since this PR changes nothing beyond the warning:
TypeErrorfor non-string top-level keys, fromparse_attributesingroup.py. That error still takes precedence over the policy. Nested non-string keys reach the new check on every node type."raise"becomes the default: when a write is refused, the in-memory attributes have already been changed.Attributes.putclears the attributes dict in place, andAsyncGroup.update_attributesupdates it in place, both before the metadata is saved. So after a refused write, the array or group object reports attributes that were never stored. v3 groups already behave this way with the existingTypeError.skip_file_prefixes.Tests
test_attributes_json_policies_write: valid JSON under the default and"raise"policies, non-string keys under"allow", and non-finite floats under the default. Each is written with no warning and read back as expected. It runs for v2 and v3 arrays and groups, both at creation and viaupdate_attributes. The pytest config turns any unexpected warning into a failure."raise"raiseTypeError; non-finite floats under"warn"warn; non-finite floats under"raise"raiseValueError; an invalid policy value raisesValueError.The full suite passes locally (10875 passed), and so do the docs example tests.
🤖 Generated with Claude Code