Skip to content

test_storage.py::TestFSStore::test_create fails #825

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
mcepl opened this issue Aug 28, 2021 · 4 comments
Closed

test_storage.py::TestFSStore::test_create fails #825

mcepl opened this issue Aug 28, 2021 · 4 comments

Comments

@mcepl
Copy link

mcepl commented Aug 28, 2021

When packaging zarr 2.9.3 for running the test suite it fails in test_storage.py::TestFSStore::test_create:

[   27s] + PYTHONPATH=/home/abuild/rpmbuild/BUILDROOT/python-zarr-2.9.3-0.x86_64/usr/lib/python3.9/site-packages
[   27s] + PYTHONDONTWRITEBYTECODE=1
[   27s] + pytest-3.9 --ignore=_build. --ignore=_build.python39 --ignore=_build.python38 -v
[   27s] ============================= test session starts ==============================
[   27s] platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /usr/bin/python3.9
[   27s] cachedir: .pytest_cache
[   27s] rootdir: /home/abuild/rpmbuild/BUILD/zarr-2.9.3, configfile: pytest.ini
[   32s] collecting ... collected 2447 items
[   32s]
[ ... snip a zillion passing tests ...]
[  156s]
[  146s]
[  146s] =================================== FAILURES ===================================
[  146s] ___________________________ TestFSStore.test_create ____________________________
[  146s]
[  146s] self = <zarr.tests.test_storage.TestFSStore object at 0x7f1473e8ba00>
[  146s]
[  146s]     def test_create(self):
[  146s]         import zarr
[  146s]         path1 = tempfile.mkdtemp()
[  146s]         path2 = tempfile.mkdtemp()
[  146s]         g = zarr.open_group("file://" + path1, mode='w',
[  146s]                             storage_options={"auto_mkdir": True})
[  146s]         a = g.create_dataset("data", shape=(8,))
[  146s]         a[:4] = [0, 1, 2, 3]
[  146s]         assert "data" in os.listdir(path1)
[  146s]         assert ".zgroup" in os.listdir(path1)
[  146s]
[  146s]         g = zarr.open_group("simplecache::file://" + path1, mode='r',
[  146s]                             storage_options={"cache_storage": path2,
[  146s]                                              "same_names": True})
[  146s] >       assert g.data[:].tolist() == [0, 1, 2, 3, 0, 0, 0, 0]
[  146s]
[  146s] zarr/tests/test_storage.py:975:
[  146s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[  146s] zarr/core.py:662: in __getitem__
[  146s]     return self.get_basic_selection(selection, fields=fields)
[  146s] zarr/core.py:787: in get_basic_selection
[  146s]     return self._get_basic_selection_nd(selection=selection, out=out,
[  146s] zarr/core.py:830: in _get_basic_selection_nd
[  146s]     return self._get_selection(indexer=indexer, out=out, fields=fields)
[  146s] zarr/core.py:1125: in _get_selection
[  146s]     self._chunk_getitems(lchunk_coords, lchunk_selection, out, lout_selection,
[  146s] zarr/core.py:1836: in _chunk_getitems
[  146s]     cdatas = self.chunk_store.getitems(ckeys, on_error="omit")
[  146s] zarr/storage.py:1111: in getitems
[  146s]     results = self.map.getitems(keys_transformed, on_error="omit")
[  146s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[  146s]
[  146s] self = <fsspec.mapping.FSMap object at 0x7f1474185250>, keys = ['data/0']
[  146s] on_error = 'omit'
[  146s]
[  146s]     def getitems(self, keys, on_error="raise"):
[  146s]         """Fetch multiple items from the store
[  146s]
[  146s]         If the backend is async-able, this might proceed concurrently
[  146s]
[  146s]         Parameters
[  146s]         ----------
[  146s]         keys: list(str)
[  146s]             They keys to be fetched
[  146s]         on_error : "raise", "omit", "return"
[  146s]             If raise, an underlying exception will be raised (converted to KeyError
[  146s]             if the type is in self.missing_exceptions); if omit, keys with exception
[  146s]             will simply not be included in the output; if "return", all keys are
[  146s]             included in the output, but the value will be bytes or an exception
[  146s]             instance.
[  146s]
[  146s]         Returns
[  146s]         -------
[  146s]         dict(key, bytes|exception)
[  146s]         """
[  146s]         keys2 = [self._key_to_str(k) for k in keys]
[  146s]         oe = on_error if on_error == "raise" else "return"
[  146s]         try:
[  146s]             out = self.fs.cat(keys2, on_error=oe)
[  146s]         except self.missing_exceptions as e:
[  146s]             raise KeyError from e
[  146s]         out = {
[  146s]             k: (KeyError() if isinstance(v, self.missing_exceptions) else v)
[  146s] >           for k, v in out.items()
[  146s]         }
[  146s] E       AttributeError: 'bytes' object has no attribute 'items'
[  146s]
[  146s] /usr/lib/python3.9/site-packages/fsspec/mapping.py:95: AttributeError
[  146s] =============================== warnings summary ===============================
[  146s] zarr/tests/test_creation.py::test_open_array
[  146s] zarr/tests/test_storage.py::TestN5Store::test_init_array_overwrite_group
[  146s] zarr/tests/test_storage.py::TestN5Store::test_init_array
[  146s] zarr/tests/test_storage.py::TestN5Store::test_init_array_path
[  146s] zarr/tests/test_storage.py::TestN5Store::test_init_array_overwrite
[  146s] zarr/tests/test_storage.py::TestN5Store::test_init_array_overwrite_path
[  146s] zarr/tests/test_storage.py::TestN5Store::test_init_array_overwrite_chunk_store
[  146s] zarr/tests/test_storage.py::TestN5Store::test_filters
[  146s]   /home/abuild/rpmbuild/BUILD/zarr-2.9.3/zarr/n5.py:420: RuntimeWarning: Not all N5 implementations support blosc compression (yet). You might not be able to open the dataset with another N5 library.
[  146s]     warnings.warn(
[  146s]
[  146s] zarr/tests/test_creation.py::test_compression_args
[  146s]   /home/abuild/rpmbuild/BUILD/zarr-2.9.3/zarr/tests/test_creation.py:472: UserWarning: 'compression' keyword argument overridden by 'compressor'
[  146s]     z = create(100, compressor=Zlib(9), compression='bz2', compression_opts=1)
[  146s]
[  146s] zarr/tests/test_creation.py::test_compression_args
[  146s]   /home/abuild/rpmbuild/BUILD/zarr-2.9.3/zarr/tests/test_creation.py:472: UserWarning: 'compression_opts' keyword argument overridden by 'compressor'
[  146s]     z = create(100, compressor=Zlib(9), compression='bz2', compression_opts=1)
[  146s]
[  146s] zarr/tests/test_creation.py::test_compression_args
[  146s]   /home/abuild/rpmbuild/BUILD/zarr-2.9.3/zarr/tests/test_creation.py:478: UserWarning: 'compression_opts' keyword argument overridden by 'compressor'
[  146s]     z = create(100, compressor=Zlib(9), compression_opts=1)
[  146s]
[  146s] -- Docs: https://docs.pytest.org/en/stable/warnings.html
[  146s] ============================= slowest 10 durations =============================
[  146s] 7.74s call     zarr/tests/test_indexing.py::test_set_orthogonal_selection_3d
[  146s] 7.51s call     zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_resize_2d
[  146s] 7.48s call     zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_iter
[  146s] 6.77s call     zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_islice
[  146s] 4.69s call     zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_array_order
[  146s] 4.35s call     zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_append_2d_axis
[  146s] 4.04s call     zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_append_2d
[  146s] 2.69s call     zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_np_ufuncs
[  146s] 2.28s call     zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_dtypes
[  146s] 1.95s call     zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_compressors
[  146s] =========================== short test summary info ============================
[  146s] FAILED zarr/tests/test_storage.py::TestFSStore::test_create - AttributeError:...
[  146s] = 1 failed, 2031 passed, 413 skipped, 2 xfailed, 11 warnings in 118.60s (0:01:58) =

If could be Python 3.9 issue (hello, #629 !), but I think it should be unstable API of whatever package provides self.fs.cat.

Version and installation information

Please provide the following:

  • Value of zarr.__version__ 2.9.3
  • Value of numcodecs.__version__ 0.7.3
  • Version of Python interpreter 3.9.6
  • Operating system (Linux/Windows/Mac) Linux, packaging for openSUSE/Tumbleweed
  • How Zarr was installed (e.g., "using pip into virtual environment", or "using conda") tarball

Complete build log with all versions of packages used and steps taken.
@joshmoore
Copy link
Member

I assume there's no easy way to avoid the use of fsspec-2021.7.0-1.1?

cc: @martindurant

@mcepl
Copy link
Author

mcepl commented Aug 28, 2021

I assume there's no easy way to avoid the use of fsspec-2021.7.0-1.1?

I am sorry, but no. We try to have as recent packages as possible and unless it is a super-important package we won't hold whole distribution back.

@mcepl
Copy link
Author

mcepl commented Aug 28, 2021

Apparently a duplicate of #812.

bmwiedemann pushed a commit to bmwiedemann/openSUSE that referenced this issue Aug 28, 2021
@bmwiedemann
Update python-zarr to version 2.9.3 / rev 4 via SR 914748
7419246 
https://build.opensuse.org/request/show/914748
by user mcepl + dimstar_suse
- Update to 2.9.3:
  - Mark the fact that some tests that require fsspec, without
    compromising the code coverage score. By Ben Williams; #823.
  - Only inspect alternate node type if desired isn’t
    present. By Trevor Manz; #696.
  - Correct conda-forge deployment of Zarr by fixing some Zarr
    tests. By Ben Williams; #821.
- Add skip-825-TestFSStore-test_create.patch
  (gh#zarr-developers/zarr-python#825).
- Update to version 2.9.0
  * This release of Zarr Python is the first release of Zarr to not
    support Python 3.6.
  * Update ABSStore for compatibility with newer azure.storage.
    blob.
  * Pathlib support.
  * Fix dimension_separator support.
  * Extract ABSStore to zarr._storage.absstore.
  * TST: add missing assert in test_hexdigest.
- Add zarr-pr802-fsspec-2021-07.patch
@joshmoore
Copy link
Member

Fixed by #802 (v2.9.5)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@joshmoore @mcepl