Code

Implement Zarr V3 protocol#898

Code

Merged

joshmoore merged 121 commits intozarr-developers:masterzarr-developers/zarr-python:masterfrom grlee77:v3-store-part6grlee77/zarr-python:v3-store-part6Copy head branch name to clipboardMar 23, 2022

+5,383-1,359

Commits

Commits on Nov 17, 2021

• add v3 store classes

  Show description for 64bbbd8
  grlee77

  committed
  64bbbd8

  Copy full SHA for 64bbbd8

• add TODO comment to meta.py

  grlee77

  committed
  85b8e23

  Copy full SHA for 85b8e23

• Merge remote-tracking branch 'upstream/master' into v3-store-part1

  grlee77

  committed
  5fee6fb

  Copy full SHA for 5fee6fb

Commits on Nov 30, 2021

• fix flake8 errors

  grlee77

  committed
  983d190

  Copy full SHA for 983d190

• Merge remote-tracking branch 'upstream/master' into v3-store-part1

  grlee77

  committed
  4057088

  Copy full SHA for 4057088

Commits on Dec 15, 2021

• follow zarr v3 spec when dealing with extension data types

  grlee77

  committed
  9ed6181

  Copy full SHA for 9ed6181

• fixes to v3 dtype handling

  grlee77

  committed
  662e310

  Copy full SHA for 662e310

• Merge remote-tracking branch 'upstream/master' into v3-store-part1

  grlee77

  committed
  9950a4d

  Copy full SHA for 9950a4d

• flake8 cleanup

  grlee77

  committed
  450c575

  Copy full SHA for 450c575

Commits on Dec 16, 2021

• remove duplicate lines in Metadata2.encode_array_metadata

  grlee77

  committed
  63a9e3c

  Copy full SHA for 63a9e3c

Commits on Dec 17, 2021

• Fix fields in array metadata

  Show description for 1a2a1ef
  grlee77

  committed
  1a2a1ef

  Copy full SHA for 1a2a1ef

• Fix encode/decode of codec metadata

  Show description for bcb2d5f
  grlee77

  committed
  bcb2d5f

  Copy full SHA for bcb2d5f

• add missing level to Zlib in _decode_codec_metadata

  grlee77

  committed
  9bfeacb

  Copy full SHA for 9bfeacb

• add extensions entry to v3 array metadata

  grlee77

  committed
  23aca42

  Copy full SHA for 23aca42

• dimension_separator should not be in the array metadata for v3

  grlee77

  committed
  e47035d

  Copy full SHA for e47035d

• update Attributes, adding StoreV3 support

  Show description for 7549d5b
  grlee77

  committed
  7549d5b

  Copy full SHA for 7549d5b

• add StoreV3 support to core Array object

  grlee77

  committed
  765fe7d

  Copy full SHA for 765fe7d

• update hexdigests

  grlee77

  committed
  5efc786

  Copy full SHA for 5efc786

• handle additional codecs that were not implemented in zarrita

  Show description for 6a308c8
  grlee77

  committed
  6a308c8

  Copy full SHA for 6a308c8

• fix

  grlee77

  committed
  f7883fc

  Copy full SHA for f7883fc

• fix hexdigests

  grlee77

  committed
  5040119

  Copy full SHA for 5040119

• fix indentation

  grlee77

  committed
  7c3736d

  Copy full SHA for 7c3736d

• add StoreV3 support to Group, open_group, etc.

  grlee77

  committed
  d07caed

  Copy full SHA for d07caed

• add StoreV3 support to creation routines

  grlee77

  committed
  00b2cc0

  Copy full SHA for 00b2cc0

• Handle dimension_separator appropriately in open_array

  Show description for 504913a
  grlee77

  committed
  504913a

  Copy full SHA for 504913a

• TST: add tests for open_array and dimension_separator

  grlee77

  committed
  7142974

  Copy full SHA for 7142974

• only allow Codec not a simple str as compressor during array initialization

  grlee77

  committed
  e4ce79f

  Copy full SHA for e4ce79f

• add StoreV3 support to most convenience routines

  Show description for 8bef4fc
  grlee77

  committed
  8bef4fc

  Copy full SHA for 8bef4fc

• set convenience routines default to zarr_version=None

  Show description for 9478ea3
  grlee77

  committed
  9478ea3

  Copy full SHA for 9478ea3

• adjust test have dimension_separator key was removed from v3 metadata

  grlee77

  committed
  2c21761

  Copy full SHA for 2c21761

• Merge branch 'v3-store-part1' into v3-store-part6

  grlee77

  committed
  f28d3d3

  Copy full SHA for f28d3d3

• Merge remote-tracking branch 'upstream/master' into v3-store-part6

  grlee77

  committed
  14a3c05

  Copy full SHA for 14a3c05

• add underscores to imported test classes in test_storage_v3.py

  Show description for 8783f44
  grlee77

  committed
  8783f44

  Copy full SHA for 8783f44

• Merge branch 'v3-store-part1' into v3-store-part6

  grlee77

  committed
  4bd99dc

  Copy full SHA for 4bd99dc

• add underscore to imported TestArrayWithPath in test_core_v3.py

  Show description for 981c6b9
  grlee77

  committed
  981c6b9

  Copy full SHA for 981c6b9

Commits on Dec 19, 2021

• refactore _valid_keys and add tests

  Show description for fc5c1c1
  grlee77

  committed
  fc5c1c1

  Copy full SHA for fc5c1c1

• move KVStoreV3 logic from StoreV3.__eq__ to KVStoreV3.__eq__

  grlee77

  committed
  83cf345

  Copy full SHA for 83cf345

• expand tests for _ensure_store

  grlee77

  committed
  1dbce4a

  Copy full SHA for 1dbce4a

• test exception for v2 store input to _get_hierarchy_metadata

  grlee77

  committed
  5391c46

  Copy full SHA for 5391c46

• test exception for init_array with path=None

  grlee77

  committed
  6feacdb

  Copy full SHA for 6feacdb

• remove unneeded checks from Attributes

  Show description for bf244df
  grlee77

  committed
  bf244df

  Copy full SHA for bf244df

• tests __repr__ of LazyLoader

  grlee77

  committed
  097459b

  Copy full SHA for 097459b

• test load of individual array

  grlee77

  committed
  2f15195

  Copy full SHA for 2f15195

• Add simple test case for zarr.tree convenience method

  grlee77

  committed
  235b445

  Copy full SHA for 235b445

• add tests for copy_store with a V3 store class

  grlee77

  committed
  0b82bda

  Copy full SHA for 0b82bda

• test raising of exception on intialization with mismatched store and chunk_store protocol versions

  grlee77

  committed
  ab71ed5

  Copy full SHA for ab71ed5

Commits on Dec 20, 2021

• add key validation on setitem in v3 stores

  Show description for 9bcfb24
  grlee77

  committed
  9bcfb24

  Copy full SHA for 9bcfb24

• Fix core V3 tests now that keys are validated on __setitem__

  grlee77

  committed
  f4bedd6

  Copy full SHA for f4bedd6

• pep8 in storage_v3 tests

  grlee77

  committed
  95c0a22

  Copy full SHA for 95c0a22

• flake8 in test_convenience.py

  grlee77

  committed
  e321c12

  Copy full SHA for e321c12

• pep8

  grlee77

  committed
  2d9058e

  Copy full SHA for 2d9058e

• fix test_attrs.py

  Show description for 63d8d27
  grlee77

  committed
  63d8d27

  Copy full SHA for 63d8d27

• Fix SQLLiteStore

  Show description for 8417dc3
  grlee77

  committed
  8417dc3

  Copy full SHA for 8417dc3

• fix failing hierarchy test

  grlee77

  committed
  b9c9ed4

  Copy full SHA for b9c9ed4

• update ZipStore tests to make sure they all run on V3

  grlee77

  committed
  efb0b38

  Copy full SHA for efb0b38

• add default rmdir implementation to all StoreV3 classes

  Show description for d2ee391
  grlee77

  committed
  d2ee391

  Copy full SHA for d2ee391

• fix test_sync.py

  grlee77

  committed
  6d1ac20

  Copy full SHA for 6d1ac20

• all rmdir methods for StoreV3 classes need to remove associated metadata

  grlee77

  committed
  e12adac

  Copy full SHA for e12adac

• avoid warning from test_entropy.py

  grlee77

  committed
  d142da1

  Copy full SHA for d142da1

• pep8 fixes

  grlee77

  committed
  2c8098e

  Copy full SHA for 2c8098e

• greatly reduce code duplication in test_storage_v3.py

  Show description for ae2145a
  grlee77

  committed
  ae2145a

  Copy full SHA for ae2145a

Commits on Dec 21, 2021

• remove redundant test_hexdigest methods

  Show description for 9fa4bdc
  grlee77

  committed
  9fa4bdc

  Copy full SHA for 9fa4bdc

• move test_core_v3.py functions back into test_core.py

  grlee77

  committed
  3291890

  Copy full SHA for 3291890

• typing fixes for mypy

  grlee77

  committed
  1850d88

  Copy full SHA for 1850d88

• can assume self.keys() exists since BaseStore inherits from MutableMapping

  grlee77

  committed
  7524764

  Copy full SHA for 7524764

• refactor rmdir methods for v3 and improve coverage

  grlee77

  committed
  4824143

  Copy full SHA for 4824143

• improve coverage of core.py

  grlee77

  committed
  19fcbee

  Copy full SHA for 19fcbee

• improve coverage of convenience.py

  grlee77

  committed
  81d1c9a

  Copy full SHA for 81d1c9a

• expend info tests

  Show description for b0a963a
  grlee77

  committed
  b0a963a

  Copy full SHA for b0a963a

• Expand tests of Array.view

  grlee77

  committed
  2b79be1

  Copy full SHA for 2b79be1

• improve coverage of creation.py

  grlee77

  committed
  02d9535

  Copy full SHA for 02d9535

• improve coverage of hierarchy.py

  grlee77

  committed
  f2dbf7b

  Copy full SHA for f2dbf7b

• improve coverage of meta.py

  grlee77

  committed
  f3378b8

  Copy full SHA for f3378b8

• pep8

  grlee77

  committed
  c86802a

  Copy full SHA for c86802a

• skip FSStoreV3 test when fsspec not installed

  grlee77

  committed
  20338c4

  Copy full SHA for 20338c4

• test raising of PermissionError for setter on views

  grlee77

  committed
  70ea4a5

  Copy full SHA for 70ea4a5

• remove redundant check (_normalize_store_arg will already raise here)

  grlee77

  committed
  f2982d0

  Copy full SHA for f2982d0

• improve coverage and fix bugs in normalize_store_arg

  grlee77

  committed
  9bc92c5

  Copy full SHA for 9bc92c5

• improve coverage of storage.py

  Show description for 82be4c9
  grlee77

  committed
  82be4c9

  Copy full SHA for 82be4c9

• pep8

  grlee77

  committed
  8bea4b6

  Copy full SHA for 8bea4b6

Commits on Dec 22, 2021

• fix StoreV3 tests

  grlee77

  committed
  ce64fab

  Copy full SHA for ce64fab

• Merge remote-tracking branch 'upstream/master' into v3-store-part6

  grlee77

  committed
  9de3849

  Copy full SHA for 9de3849

• fix duplicate zarr_fsstore entry

  grlee77

  committed
  b5a691b

  Copy full SHA for b5a691b

• fix rename

  grlee77

  committed
  47e4354

  Copy full SHA for 47e4354

• remove debug statements

  grlee77

  committed
  2415d85

  Copy full SHA for 2415d85

• fix typo

  grlee77

  committed
  98120df

  Copy full SHA for 98120df

• skip unavailable NumPy dtypes

  grlee77

  committed
  a99519c

  Copy full SHA for a99519c

• pep8

  grlee77

  committed
  bd613bf

  Copy full SHA for bd613bf

• mypy fixes

  grlee77

  committed
  eec4a3e

  Copy full SHA for eec4a3e

• remove redundant check (already done above)

  grlee77

  committed
  5970c69

  Copy full SHA for 5970c69

• remove KeyError check. list_prefix only returns keys that exist

  grlee77

  committed
  77f6fee

  Copy full SHA for 77f6fee

• coverage fixes

  grlee77

  committed
  f2b3ed8

  Copy full SHA for f2b3ed8

Commits on Jan 6, 2022

• Merge remote-tracking branch 'upstream/master' into v3-store-part6

  grlee77

  committed
  05fbb2f

  Copy full SHA for 05fbb2f

• implemented ConsolidatedMetadataStoreV3

  Show description for 347b67d
  grlee77

  committed
  347b67d

  Copy full SHA for 347b67d

• expand ConsolidatedMetadataStoreV3 tests

  Show description for 0f2d583
  grlee77

  committed
  0f2d583

  Copy full SHA for 0f2d583

Commits on Jan 7, 2022

• remove debug statement

  grlee77

  committed
  0c39bd4

  Copy full SHA for 0c39bd4

• fix tests: restore clobber=True

  grlee77

  committed
  993493b

  Copy full SHA for 993493b

• test error path in consolidate_metadata

  grlee77

  committed
  a08d4f7

  Copy full SHA for a08d4f7

• add pragma: no cover for lines in test_meta.py that will only be visited on some architectures

  grlee77

  committed
  43dfa29

  Copy full SHA for 43dfa29

• flake8 fixes

  grlee77

  committed
  41b052f

  Copy full SHA for 41b052f

Commits on Jan 9, 2022

• flake8

  grlee77

  committed
  8af6131

  Copy full SHA for 8af6131

Commits on Jan 13, 2022

• ENH: add ABSStoreV3

  grlee77

  committed
  001510b

  Copy full SHA for 001510b

• flake8

  grlee77

  committed
  92f4b80

  Copy full SHA for 92f4b80

Commits on Feb 28, 2022

• Merge remote-tracking branch 'upstream/master' into v3-store-part6

  grlee77

  committed
  aa64b7e

  Copy full SHA for aa64b7e

Commits on Mar 1, 2022

• fix ABSStore.rmdir test coverage

  grlee77

  committed
  c358506

  Copy full SHA for c358506

Commits on Mar 2, 2022

• always use / in path

  grlee77

  committed
  133ee26

  Copy full SHA for 133ee26

Commits on Mar 4, 2022

• Merge remote-tracking branch 'upstream/master' into v3-store-part6

  grlee77

  committed
  652960f

  Copy full SHA for 652960f

• remove remaining use of clobber argument in new tests

  grlee77

  committed
  34f6747

  Copy full SHA for 34f6747

• remove NestedDirectoryStoreV3

  Show description for 9f56e82
  grlee77

  committed
  9f56e82

  Copy full SHA for 9f56e82

• flake8

  grlee77

  committed
  88405a8

  Copy full SHA for 88405a8

Commits on Mar 9, 2022

• Merge remote-tracking branch 'upstream/master' into v3-store-part6

  grlee77

  committed
  87b9543

  Copy full SHA for 87b9543

• remove rmdir_abs: rmdir method of ABSStore parent class in ABSStoreV3

  grlee77

  committed
  69e5a12

  Copy full SHA for 69e5a12

• define meta_root and data_root variables

  Show description for 58c8c20
  grlee77

  committed
  58c8c20

  Copy full SHA for 58c8c20

• move _valid_key_characters to be a StoreV3 class field

  grlee77

  committed
  5e1ed7f

  Copy full SHA for 5e1ed7f

• make _get_hierarchy_metadata strictly require 'zarr.json'

  Show description for 3593cad
  grlee77

  committed
  3593cad

  Copy full SHA for 3593cad

Commits on Mar 11, 2022

• ignore type checks for _get_metadata_suffix

  grlee77

  committed
  f036215

  Copy full SHA for f036215

• Merge remote-tracking branch 'upstream/master' into v3-store-part6

  grlee77

  committed
  6f08185

  Copy full SHA for 6f08185

• remove unneeded if/else in Array and Hierarchy class __init__

  Show description for 353058d
  grlee77

  committed
  353058d

  Copy full SHA for 353058d

• remove unused import

  grlee77

  committed
  2ec2b2d

  Copy full SHA for 2ec2b2d

• define DEFAULT_ZARR_VERSION so we can later more easily change from 2 to 3

  grlee77

  committed
  2b82967

  Copy full SHA for 2b82967

• add test_get_hierarchy_metadata to test the v3 _get_hierarchy_metadata helper

  grlee77

  committed
  7aa5dff

  Copy full SHA for 7aa5dff