Split tests

[clbarnes]

• Refactor test cases to one file per test
• Factor into two types of test
  • "attributes" which represents single attributes object from a single zarr.json
  • "zarr" which represents a zarr hierarchy in a filesystem store (with no chunks)
• Provide a CLI tool which feeds these test cases into a simple "dingus" CLI of the implementor's choice

Example usage:

git clone https://github.com/clbarnes/ngff-spec.git
ngff-spec/ome_zarr_conformance.py attributes -- path/to/my/dingus --dingus-arg

This will iterate over all attributes tests (concurrently) and, with the test path $TEST_PATH, calls path/to/my/dingus --dingus-arg $TEST_PATH.

The dingus is expected to read the test from the given path and return {"valid": true} if it believes the test data is valid (false if not; optionally "message": str with more details). Prints out a TSV of all test names and their status pass/fail/error.

In "attributes" mode, $TEST_PATH is a path to a JSON file containing the Zarr attributes object (i.e. {"ome": {...}}). In "zarr" mode, $TEST_PATH is a path to directory representing the root of a zarr container (i.e. containing a zarr.json).

Test cases are not guaranteed to be correct for the moment.

I recommend ignoring most of the diff (mainly just moving around JSON files); look at ome_zarr_conformance.py and the changes in the schemas/ directory.

[github-actions]

Automated Review URLs

• Readthedocs

[jo-mueller]

3cts from me:

• Version here is 0.6.dev1 but I think in the future (and elsewhere already for 0.6dev2) we/I am using 0.6dev2 instead of 0.6.dev2.
• Was all of this done via code? If so - could you point that code at RFC5: coordinate systems and transformations #17? I already combed through all the examples there and reformatted. The pytest tests are also correctly triggered there, so maybe that would also be a good place "for a fresh start"?
• Maybe move the strict test cases into valid/invalid? Not a must, though. At the end, the point is more to be able to pinpoint a failing test to a particular file, anyway, right?

Edit: PS: dingus is a much better example placeholder than that foo/bar scheme.

[clbarnes]

Easy enough to find&replace spec versions. I can point the scripts at another PR easy.

re. the level of strict tests - I am in two minds about whether to encourage use of the strict tests at all (and to some extent the strict schemas). That's why I had that at the top level - so that the tests people tend to run together are closer together. Implementations must support the non-strict case so what's the point of validating for the strict cases? It just seems like more maintenance burden for us. But this is probably worth another issue.

[clbarnes]

P.S. the dingus name is inspired by markdown https://daringfireball.net/projects/markdown/dingus ; most implementations offer a dingus to use for conformance checking.