test_nested[static_flat] fails on Debian sid Python 3.9 #858

avalentino · 2021-10-22T07:19:52Z

Minimal, reproducible code sample, a copy-pastable example if possible

$ python3.9 -m pytest

============================= test session starts ==============================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-0.13.0
rootdir: /build/zarr-2.10.2+ds, configfile: pytest.ini
collected 2598 items

zarr/tests/test_attrs.py ........                                        [  0%]
zarr/tests/test_convenience.py ......................................... [  1%]
......................................                                   [  3%]
zarr/tests/test_core.py ................................................ [  5%]
........................................................................ [  7%]
....................................................................ssss [ 10%]
sssssssssssssssssssssssssssssssssssssssssss............................. [ 13%]
..................................................................ssssss [ 16%]
ssssssssssssssssssssssssssssssssssssssssss.............................. [ 19%]
.................sssssssssssssss.sssssssss.sssssssssssssssssssssssssssss [ 21%]
sssssss.sssssssss.ssssssssssssss.sssssssssssssssssssss.sssssssss.sssssss [ 24%]
ssssssssssssss.......................................................... [ 27%]
........................................................................ [ 30%]
........................................................................ [ 32%]
........................................................................ [ 35%]
........................................................................ [ 38%]
........................................................................ [ 41%]
.........sssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 43%]
ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss [ 46%]
sssssssssssssssssssssssssssssssssssssssssssssssssssssssssss              [ 49%]
zarr/tests/test_creation.py ..............                               [ 49%]
zarr/tests/test_dim_separator.py .........sssssssssssssss.........sssFx. [ 51%]
...x..sss                                                                [ 51%]
zarr/tests/test_filters.py .......                                       [ 51%]
zarr/tests/test_hierarchy.py ........................................... [ 53%]
....................sssssssssssssssssssss.....................ssssssssss [ 56%]
sssssssssssssssssssssssssssssssssss..................................... [ 58%]
.....ssssssssssssssssssssssssssssssssssssssssss......................... [ 61%]
............................................                             [ 63%]
zarr/tests/test_indexing.py .................................xx          [ 64%]
zarr/tests/test_info.py .                                                [ 64%]
zarr/tests/test_meta.py ................                                 [ 65%]
zarr/tests/test_storage.py ............s........................s....... [ 67%]
.................s...........................................sssssssssss [ 69%]
ssssssssssssssssssssssssssssssssssssssssssss............................ [ 72%]
....................................ssssssssssssssssssssssssssssssssssss [ 75%]
ssssssssssssssssssssssssssss.............s.............................. [ 78%]
...........................................................sssssssssssss [ 80%]
ssssssssssss.........................sssssssssssssssssssssssssssssssssss [ 83%]
sssssssssssssss..................................................sssssss [ 86%]
sssssssssssssssssssssssssssssssssssssssss.............s................s [ 89%]
sssssssssssssssssssssssss..                                              [ 90%]
zarr/tests/test_sync.py ................................................ [ 92%]
........................................................................ [ 94%]
........................................................................ [ 97%]
..............................................                           [ 99%]
zarr/tests/test_util.py ..............                                   [100%]

=================================== FAILURES ===================================
___________________________ test_nested[static_flat] ___________________________

dataset = '/build/zarr-2.10.2+ds/.pybuild/cpython3_3.9_zarr/build/fixture/flat'

    def test_nested(dataset):
        """
        Use NestedDirectoryStore to open the dataset fixture. This is the only
        method that is expected to successfully open legacy nested datasets
        without the dimension_separator metadata. However, for none-Nested
        datasets without any metadata, NestedDirectoryStore will fail.
        """
        failure = (
            "flat_legacy" in dataset or
            "directory_default" in dataset or
            "fs_default" in dataset
        )
>       verify(Array(store=NestedDirectoryStore(dataset)), failure)

zarr/tests/test_dim_separator.py:129:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _

array = <zarr.core.Array (2, 2) int64>, expect_failure = False

    def verify(array, expect_failure=False):
        try:
>           assert_array_equal(array[:], [[1, 2], [3, 4]])
E           AssertionError:
E           Arrays are not equal
E
E           Mismatched elements: 4 / 4 (100%)
E           Max absolute difference: 4
E           Max relative difference: 1.
E            x: array([[0, 0],
E                  [0, 0]])
E            y: array([[1, 2],
E                  [3, 4]])

zarr/tests/test_dim_separator.py:78: AssertionError
=============================== warnings summary ===============================
.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_creation.py::test_open_array
.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_storage.py::TestN5Store::test_init_array_overwrite_group
.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_storage.py::TestN5Store::test_init_array
.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_storage.py::TestN5Store::test_init_array_path
.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_storage.py::TestN5Store::test_init_array_overwrite
.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_storage.py::TestN5Store::test_init_array_overwrite_path
.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_storage.py::TestN5Store::test_init_array_overwrite_chunk_store
.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_storage.py::TestN5Store::test_filters
  /build/zarr-2.10.2+ds/.pybuild/cpython3_3.9_zarr/build/zarr/n5.py:708: RuntimeWarning: Not all N5 implementations support blosc compression (yet). You might not be able to open the dataset with another N5 library.
    warnings.warn(

.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_creation.py::test_compression_args
  /build/zarr-2.10.2+ds/.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_creation.py:472: UserWarning: 'compression' keyword argument overridden by 'compressor'
    z = create(100, compressor=Zlib(9), compression='bz2', compression_opts=1)

.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_creation.py::test_compression_args
  /build/zarr-2.10.2+ds/.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_creation.py:472: UserWarning: 'compression_opts' keyword argument overridden by 'compressor'
    z = create(100, compressor=Zlib(9), compression='bz2', compression_opts=1)

.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_creation.py::test_compression_args
  /build/zarr-2.10.2+ds/.pybuild/cpython3_3.9_zarr/build/zarr/tests/test_creation.py:478: UserWarning: 'compression_opts' keyword argument overridden by 'compressor'
    z = create(100, compressor=Zlib(9), compression_opts=1)

-- Docs: https://docs.pytest.org/en/stable/warnings.html
============================= slowest 10 durations =============================
25.34s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_indexing.py::test_set_orthogonal_selection_3d
6.08s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_storage.py::test_format_compatibility
3.94s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_indexing.py::test_set_orthogonal_selection_2d
2.10s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_core.py::TestArrayWithNestedDirectoryStore::test_object_arrays_vlen_array
2.02s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_indexing.py::test_get_orthogonal_selection_3d
1.80s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_core.py::TestArrayWithBZ2Compressor::test_object_arrays_vlen_array
1.77s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_core.py::TestArrayWithDirectoryStore::test_object_arrays_vlen_array
1.72s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_sync.py::TestArrayWithThreadSynchronizer::test_object_arrays_vlen_array
1.66s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_core.py::TestArrayWithSQLiteStore::test_object_arrays_vlen_array
1.65s call     .pybuild/cpython3_3.9_zarr/build/zarr/tests/test_core.py::TestArrayWithDBMStore::test_object_arrays_vlen_array
=========================== short test summary info ============================
FAILED zarr/tests/test_dim_separator.py::test_nested[static_flat] - Assertion...
= 1 failed, 1768 passed, 825 skipped, 4 xfailed, 11 warnings in 148.25s (0:02:28) =

Problem description

The "test_nested[static_flat]" test fails in the current Debian sid with Python 3.9.
Of course one would expect that all automatic tests pass.

Version and installation information

Value of zarr.__version__ 2.10.2
Value of numcodecs.__version__ 0.8.1
Version of Python interpreter 3.9.7
Operating system (Linux/Windows/Mac) GNU/Linux Debian Sid (22/10/2021)
How Zarr was installed (e.g., "using pip into virtual environment", or "using conda"): setup.py during the deb package build

The text was updated successfully, but these errors were encountered:

joshmoore · 2021-10-22T08:13:44Z

This matches the failure in conda-forge/zarr-feedstock#56. Having this pinpointed outside of conda-forge is useful. Thanks, @avalentino.

jakirkham · 2021-10-22T08:29:21Z

Agreed thank you! 😄

Does this only happen with Python 3.9? What happens on other Python versions (like 3.8)?

joshmoore · 2021-11-08T15:13:32Z

Activating 3.9 doesn't seem to show the same error. I'll try to reproduce in a Sid docker.

joshmoore · 2021-11-15T10:24:15Z

With the following docker file:

# See: https://github.com/zarr-developers/zarr-python/issues/858
FROM debian:sid
RUN apt-get update -y \
 && apt-get install -y \
    git \
    python3 \
    python3-setuptools \
    python3-pip \
    python3-pytest \
    python3-msgpack \
    python3-fsspec \
    python3-numcodecs \
    python3-asciitree \
    python3-fasteners

WORKDIR /tmp
RUN git clone -b v2.10.2 git://github.com/zarr-developers/zarr-python
WORKDIR /tmp/zarr-python/

RUN python3 setup.py build
#RUN python3 -X tracemalloc setup.py test
ENTRYPOINT ["python3", "-m", "pytest"]

I get:

(base) ~/tmp/sid $docker run --rm -ti sid  \-svxk "test_dim_separator and static_flat"
=============================================================================================================== test session starts ================================================================================================================
platform linux -- Python 3.9.8, pytest-6.2.5, py-1.10.0, pluggy-0.13.0 -- /usr/bin/python3
cachedir: .pytest_cache
rootdir: /tmp/zarr-python, configfile: pytest.ini
collected 2598 items / 2590 deselected / 8 selected

zarr/tests/test_dim_separator.py::test_open[static_flat] PASSED
zarr/tests/test_dim_separator.py::test_open[static_flat_legacy] PASSED
zarr/tests/test_dim_separator.py::test_fsstore[static_flat] PASSED
zarr/tests/test_dim_separator.py::test_fsstore[static_flat_legacy] PASSED
zarr/tests/test_dim_separator.py::test_directory[static_flat] PASSED
zarr/tests/test_dim_separator.py::test_directory[static_flat_legacy] PASSED
zarr/tests/test_dim_separator.py::test_nested[static_flat] PASSED
zarr/tests/test_dim_separator.py::test_nested[static_flat_legacy] XFAIL

=============================================================================================================== slowest 10 durations ===============================================================================================================

(10 durations < 0.005s hidden.  Use -vv to show these durations.)
================================================================================================== 7 passed, 2590 deselected, 1 xfailed in 5.00s ===================================================================================================

(and more generally all enabled tests pass). Elided listing of installed libraries:

dpkg -l
...
ii  python3-asciitree          0.3.3-3                        all          Draw tree structures using (ASCII or Unicode) characters
ii  python3-fasteners          0.14.1-2                       all          provides useful locks - Python 3.x
ii  python3-fsspec             0.8.4-1                        all          specification that Python filesystems should adhere to (Python 3)
ii  python3-msgpack            1.0.2-2                        amd64        Python 3 implementation of MessagePack format
ii  python3-numcodecs          0.8.1+ds-3                     amd64        buffer compression and transformation codecs for Python
ii  python3-numpy              1:1.19.5-1                     amd64        Fast array facility to the Python 3 language
ii  python3.9                  3.9.8-2                        amd64        Interactive high-level object-oriented language (version 3.9)

@avalentino: can you help me modify the Dockerfile such that it will more closely match what you are seeing? I didn't see any patches that should have any effect.

Under some test conditions (conda-forge, Debian builds), the fixtures directory is not available and is then re-created. When dimension_separator is not passed to DirectoryStore, then no metadata is assigned to the file. For the "flat" (as opposed to "flat_legacy") fixture, this meant that the NestedDirectoryStore did not correct its behavior leading to a failure. see: - conda-forge/zarr-feedstock#56 - zarr-developers#858

joshmoore · 2021-11-16T12:59:02Z

Ok. I think I've finally tracked down the cause. My guess is that the fixtures/ directory was not available for testing and therefore they were being re-created and that incorrectly. Hopefully #871 should correct the situation.

…lopers#858)" This reverts commit 6f79c56.

…arr-developers#858)"" This reverts commit 568b1cf.

avalentino · 2021-11-17T08:01:11Z

Hi @joshmoore, sorry for the late reply.
Yes in the debian packaging environment I have removed the fixtures folder, and also zarr/tests/data (although removing the second one is probably not necessary).

This have not caused problems in previous versions of zarr but probably this specific test has been added recently.

I will try to test your patch ASAP.

joshmoore · 2021-11-17T08:13:14Z

Thanks, @avalentino!

avalentino · 2021-11-19T07:27:16Z

Hi @joshmoore, I can confirm that the patch in #871 works properly in the debian build environment.
Thanks a lot.

By the way, If I remember correctly, the main reason why I removed the fixtures/ folder is that it results to modified w.r.t. the original tarball after the build.
If you can suggest an easy way to restore the fixtures/ folder to the original state I could avoid to remove it and have a more comprehensive testing during the build of debian packages.

joshmoore · 2021-11-19T08:05:23Z

I can confirm that the patch in #871 works properly in the debian build environment.

Awesome. Thanks. I'll get this into a 2.10.3 release as well as the mainline (2.11 pre-release)

the main reason why I removed the fixtures/ folder is that it results to modified w.r.t. the original tarball after the build

You mean that there are modifications to the fixtures folder after the tests are run? If so, no, I don't know why that would be but I can take a look.

* Pass dimension_separator on fixture generation (fix #858) Under some test conditions (conda-forge, Debian builds), the fixtures directory is not available and is then re-created. When dimension_separator is not passed to DirectoryStore, then no metadata is assigned to the file. For the "flat" (as opposed to "flat_legacy") fixture, this meant that the NestedDirectoryStore did not correct its behavior leading to a failure. see: - conda-forge/zarr-feedstock#56 - #858 * Revert "Pass dimension_separator on fixture generation (fix #858)" This reverts commit 6f79c56. * Add a test for missing fixtures * Revert "Revert "Pass dimension_separator on fixture generation (fix #858)"" This reverts commit 568b1cf.

jakirkham · 2021-11-19T09:51:28Z

Thanks Josh! 😀

* Pass dimension_separator on fixture generation (fix #858) Under some test conditions (conda-forge, Debian builds), the fixtures directory is not available and is then re-created. When dimension_separator is not passed to DirectoryStore, then no metadata is assigned to the file. For the "flat" (as opposed to "flat_legacy") fixture, this meant that the NestedDirectoryStore did not correct its behavior leading to a failure. see: - conda-forge/zarr-feedstock#56 - zarr-developers/zarr-python#858 * Revert "Pass dimension_separator on fixture generation (fix #858)" This reverts commit 6f79c56a3153f6f1a1d36a1a385b6af636a487d9. * Add a test for missing fixtures * Revert "Revert "Pass dimension_separator on fixture generation (fix #858)"" This reverts commit 568b1cf6175513a8c4916eaa09107e36854f53ad.

joshmoore mentioned this issue Oct 22, 2021

Activate Python 3.9 #859

Merged

2 tasks

joshmoore mentioned this issue Oct 25, 2021

Test on Python 3.9 #629

Closed

This was referenced Oct 18, 2024

Surprising N5FSStore test failure zarr-developers/n5py#11

Open

zarr v2.10.2 conda-forge/zarr-feedstock#56

Closed

joshmoore added a commit to joshmoore/zarr-python that referenced this issue Nov 15, 2021

Add debugging for zarr-developers#858

f665e1a

joshmoore added a commit to joshmoore/zarr-python that referenced this issue Nov 15, 2021

Use docker commands from zarr-developers#858

f4e077e

joshmoore mentioned this issue Nov 16, 2021

Pass dimension_separator on fixture generation (fix #858) #871

Merged

2 tasks

joshmoore added a commit to joshmoore/zarr-python that referenced this issue Nov 16, 2021

Revert "Pass dimension_separator on fixture generation (fix zarr-deve…

568b1cf

…lopers#858)" This reverts commit 6f79c56.

joshmoore added a commit to joshmoore/zarr-python that referenced this issue Nov 16, 2021

Revert "Revert "Pass dimension_separator on fixture generation (fix z…

df514df

…arr-developers#858)"" This reverts commit 568b1cf.

joshmoore closed this as completed in #871 Nov 19, 2021

joshmoore mentioned this issue Feb 9, 2022

Blog suggestions zarr-developers/blog#5

Open

19 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

test_nested[static_flat] fails on Debian sid Python 3.9 #858

test_nested[static_flat] fails on Debian sid Python 3.9 #858

avalentino commented Oct 22, 2021

joshmoore commented Oct 22, 2021

jakirkham commented Oct 22, 2021

joshmoore commented Nov 8, 2021

joshmoore commented Nov 15, 2021 •

edited

Loading

joshmoore commented Nov 16, 2021

avalentino commented Nov 17, 2021

joshmoore commented Nov 17, 2021

avalentino commented Nov 19, 2021

joshmoore commented Nov 19, 2021

jakirkham commented Nov 19, 2021

test_nested[static_flat] fails on Debian sid Python 3.9 #858

test_nested[static_flat] fails on Debian sid Python 3.9 #858

Comments

avalentino commented Oct 22, 2021

Minimal, reproducible code sample, a copy-pastable example if possible

Problem description

Version and installation information

joshmoore commented Oct 22, 2021

jakirkham commented Oct 22, 2021

joshmoore commented Nov 8, 2021

joshmoore commented Nov 15, 2021 • edited Loading

joshmoore commented Nov 16, 2021

avalentino commented Nov 17, 2021

joshmoore commented Nov 17, 2021

avalentino commented Nov 19, 2021

joshmoore commented Nov 19, 2021

jakirkham commented Nov 19, 2021

joshmoore commented Nov 15, 2021 •

edited

Loading