Conversation
…t algebra Add tests/test_codecs/test_sharding_cute_oracle.py, which derives the shard -> inner-chunk -> element layouts independently with PyCuTe (logical_divide for the shard/chunk nesting, blocked_product for the data-section layout under each subchunk_write_order) and asserts that a shard written by zarr places every element and every index entry where the algebra predicts -- and, conversely, that a blob assembled purely from the algebra decodes correctly. Covers 1-3D shapes, C/F memory order, C/F within-chunk order (TransposeCodec), lexicographic / colexicographic / power-of-two morton write orders, and both index locations. Chunk localization (% chunks_per_shard) is checked as the divide remainder. PyCuTe is a test-only oracle: it is not on PyPI, so it is pinned to a commit as an extra-dependency of the 'optional' hatch test matrix, and the module importorskips elsewhere. It is never imported at runtime. Assisted-by: ClaudeCode:claude-fable-5
Assisted-by: ClaudeCode:claude-fable-5
Generate independent tilings across dimensions and storage orders. Keep zero-valued singleton chunks outside the fill value so fully populated oracle layouts remain valid, and fail when CuTe is missing from the optional Hatch matrix. Assisted-by: Codex:GPT-6
Assisted-by: Codex:GPT-6
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
🤖 AI text below 🤖
Checks sharding byte offsets and chunk localization against the independent CuTe layout algebra.
logical_dividemaps shard elements into nested chunk coordinates;blocked_productmaps those coordinates into the stored data section. The tests compare every encoded element and index entry with these layouts, then construct complete shard blobs from the oracle and decode them through public full/strided reads and synchronous decoding.Fixed and generated tilings cover one to three dimensions, non-square shapes, unit chunks and grid extents, C/F memory and within-chunk order, index placement at either end, and lexicographic/colexicographic/power-of-two Morton layouts. The generated cases exposed a fixture assumption: a zero-valued singleton chunk was being omitted as fill. The oracle now uses a fill sentinel outside its labels, preserving the fully populated layout it intends to check.
PyCuTe is pinned to commit
2e12e571c3882c5be669f8d326d630853a3571aaas a test-only dependency of the optional Hatch matrix. Missing CuTe fails in that matrix and skips elsewhere. A changelog entry is included.Validation after integrating current main:
Non-power-of-two Morton compaction and unordered layouts remain outside this stride-layout oracle. Other optional Python/platform matrix combinations were not rerun in this local review.