Conversation
… hosts Running the suite on s390x (QEMU, Debian sid) gave 110 failures on main. Most came from one asymmetry: V3 data types without an explicit byte order defaulted to little-endian, while NumPy dtype names mean host order. On a big-endian host `dtype="float64"` and `dtype=np.float64` produced different data types, and a reopened array changed dtype. - V3 data types default to the host byte order (`HasEndianness`), because V3 metadata carries none. Stored chunks stay little-endian by default. - `ShardingCodec` defaults its inner and index `bytes` codecs to little-endian, so sharded output no longer depends on the host. - `scale_offset` computes in native byte order and restores the input's byte order; `>f8` and `>u8` arrays failed on every host. - Base64 fill values of structured data types in V3 metadata are little-endian bytes in both directions. - Tests no longer assume a little-endian host: explicit `<` dtypes and `endian="little"` where the expectation is little, and a big-endian case for migration and pipeline parity. Refs zarr-developers#3438 Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
A weekly and path-filtered job that runs the data type, codec and metadata tests on an emulated big-endian host, in a Debian sid image that takes numpy and numcodecs from Debian packages so nothing heavy compiles under emulation. Assisted-by: ClaudeCode:claude-opus-5-5
`BytesCodec()` defaulted to `sys.byteorder`, so the same code wrote big-endian chunks on s390x and little-endian chunks elsewhere. The bytes codec originally defaulted to "little"; the host default came in as a side effect of replacing attrs with dataclasses (zarr-developers#1660), and was later pinned by a test that only restated it. `"little"` also matches `default_serializer_v3`. Chunks written on big-endian hosts with a bare `BytesCodec()` are now little-endian. Chunks written before this change remain readable, because the codec records its byte order in the metadata. Refs zarr-developers#3438 Assisted-by: ClaudeCode:claude-opus-5-5
This reverts commit 8dd33e3. Changing the default of `BytesCodec()` changes what existing code writes on big-endian hosts, so it needs a deprecation cycle and will land in its own PR. Assisted-by: ClaudeCode:claude-opus-5-5
…big-endian hosts `BytesCodec()` without an `endian` argument means `sys.byteorder`, so the same code writes big-endian chunks on s390x and little-endian chunks elsewhere. The default will become "little" on every host, which is what the codec originally defaulted to before zarr-developers#1660 and what the default serializer uses. On big-endian hosts, omitting `endian` now raises a ZarrFutureWarning; nothing changes on little-endian hosts. Callers inside zarr (`zarr.testing` strategies, docstrings) and the tests now pass `endian` explicitly, so they do not depend on the default. Refs zarr-developers#3438 Assisted-by: ClaudeCode:claude-opus-5-5
Assisted-by: ClaudeCode:claude-opus-5-5
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #4439 +/- ##
==========================================
+ Coverage 94.37% 94.42% +0.04%
==========================================
Files 93 93
Lines 13174 13198 +24
==========================================
+ Hits 12433 12462 +29
+ Misses 741 736 -5
🚀 New features to boost your workflow:
|
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
This PR depends on #4435 and is based on its branch. Until #4435 merges, the diff here includes its commits too. Only the last two commits belong to this PR: the deprecation and the changelog rename.
A bare
BytesCodec()defaults toendian=sys.byteorder. So the same code writes big-endian chunks on s390x and little-endian chunks everywhere else. That output is valid and readable on every host, but it is not identical across machines.History: the bytes codec in the original zarrita import defaulted to
"little". The host-order default came in with #1660 ("Remove attrs"), and neither that PR's description nor its comments discuss the switch. #3968 later pinned it with a test.Changing the default back would change what existing code writes on big-endian hosts. So this PR only deprecates the current default; the switch itself is left for a later release.
Changes
endianis omitted on a big-endian host,BytesCodec()raises aZarrFutureWarning. The codec still uses"big". The warning says to passendian="big"to keep the current behaviour, orendian="little"to adopt the new default now."little".BytesCodec.from_dictand zarr's default serializer always passendian, so they never warn.ZarrFutureWarningrather thanDeprecationWarningso that end users see it. It points at the caller's line.endian="little". These are thezarr.testingstrategies and state machine, thecreate_arraydocstrings, and about 70 test call sites. That includes two tests that passed the class itself as a factory (default_factory=BytesCodec), which the s390x run caught.test_bytes_codec_endianchecks the resolvedendianfor each combination of host byte order and argument.test_bytes_codec_default_endian_warns_on_big_endian_hostchecks the warning.sys.byteorder, so ordinary CI covers the big-endian case too.Testing
test_examplestests, because the test image has nouv. The first run also exposed the two factory call sites mentioned above; the agent fixed them and re-ran those files on s390x, and they now pass.Refs #3438.
🤖 Generated with Claude Code