Add --doc_mode config option for testing documentation images

README.rst

@@ -59,6 +59,15 @@ Alternatively, you can also install via `conda`_ or `mamba`_ from `conda-forge`_
 
 Usage
 -----
+The plugin has two main use cases:
+
+#. Evaluate images generated by calling ``pl.show()`` in unit tests
+#. Evaluate images generated by the
+   `Sphinx PyVista Plot Directive <https://docs.pyvista.org/extras/plot_directive.html>`_
+   when building documentation.
+
+Unit tests
+----------
 Once installed, you only need to use the command `pl.show()` in your test. The
 plugin will automatically manage the cache generation if it does not exist, and
 the image comparison itself. Make sure you enable `pv.OFF_SCREEN` when loading
@@ -139,7 +148,7 @@ compared to all other cached images for that test. The test is successful if one
 of the comparisons is successful, though a warning is still issued.
 
 Global flags
-------------
+============
 These are the flags you can use when calling ``pytest`` in the command line:
 
 * ``--reset_image_cache`` creates a new image for each test in
@@ -189,7 +198,7 @@ These are the flags you can use when calling ``pytest`` in the command line:
   by the same name below to configure this on a per-test basis.
 
 Test specific flags
--------------------
+===================
 These are attributes of `verify_image_cache`. You can set them as ``True`` if needed
 in the beginning of your test function.
 
@@ -211,6 +220,111 @@ in the beginning of your test function.
   ``verify_image_cache`` fixture is used but no images are generated. The value of this
   flag takes precedence over the global flag by the same name (see above).
 
+Documentation image tests
+-------------------------
+Unlike the unit tests, which use the ``verify_image_cache`` fixture to evaluate test
+images during each test's teardown, the documentation tests do not use a fixture.
+Instead, the documentation tests are invoked with the ``--doc_mode`` flag, and requires
+specifying:
+
+#. A target directory which contains all images to be tested. The directory is specified
+   using the ``--doc_images_dir`` flag.
+#. A cache directory containing all reference images to compare with. The directory is
+   specified using the ``--doc_image_cache_dir`` flag.
+
+Since all images must be available for testing, the documentation tests are typically
+executed `after` building documentation with Sphinx or some other build process. To test
+build images against images in a cache directory use:
+
+.. code-block:: bash
+
+    pytest --doc_mode --doc_images_dir images --doc_image_cache_dir cache
+
+where ``images`` is the target directory of images to test, and ``cache`` is the cache
+directory.
+
+When executed, the test will first pre-process the build images. The images are:
+
+#. Collected from the ``images`` directory (including images in nested directories).
+#. Resized to a maximum of 400x400 pixels.
+#. Renamed so that each file's parent directories are included in the name.
+#. Saved as JPEG images with in a temporary directory. The directory is flat with no
+   sub-directories.
+
+   .. note::
+      These temporary images may be saved using the ``--doc_generated_image_dir`` flag.
+
+Next, the pre-processed images are compared to the cached images.
+The tests have three main modes of failure:
+
+#.  An image is in the cache but is missing from the build.
+#.  An image is in the build but is missing from the cache.
+#.  The error threshold when comparing two images is exceeded.
+
+.. note::
+   Use the ``--doc_failed_image_dir`` flag to save copies of the images for
+   failed tests.
+
+Specifying multiple cache images
+================================
+The cache directory is typically flat with no sub-directories. However,
+it is possible to specify multiple cache images for a single test by
+including a sub-directory with the same name as the image, and including
+multiple "valid" cache images in the sub-directory. For example, a
+single cached image:
+
+.. code-block:: bash
+
+    cache/my_image.jpg
+
+Can be replaced with multiple versions of the image:
+
+.. code-block:: bash
+
+    cache/my_image/0.jpg
+    cache/my_image/1.jpg
+
+.. note::
+
+   The sub-directory name should match the image name generated from the build.
+   The image names in sub-directories can be arbitrary, however, e.g. ``0.jpg`` or
+   ``foo.jpg``, and can even be nested in sub-sub-directories (the names
+   of sub-sub-directories can also be arbitrary).
+
+When there are multiple images, the test will initially compare the build image
+to the first cached image. If that comparison fails, the build image is then
+compared to all other cached images for that test. The test is successful if one
+of the comparisons is successful, though a warning is still issued.
+
+Global flags
+============
+These are the flags you can use when calling ``pytest`` in the command line:
+
+* ``--doc_mode`` is a required flag for testing documentation images. It configures
+  pytest to only collect tests relevant for the image testing.
+
+* ``--doc_images_dir <DIR>``> sets the target directory of images to be tested.
+
+  .. note::
+
+     With Sphinx, build images are typically saved to ``doc/_build/html/_images``.
+
+* ``--doc_image_cache_dir <DIR>`` sets the doc image cache directory, relative to `pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>`.
+  This will override any configuration, see below.
+
+* ``--doc_generated_image_dir <DIR>`` dumps all doc generated test images into the provided
+  directory, relative to `pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>`.
+  This will override any configuration, see below.
+
+  .. note::
+    These are the pre-processed images generated for the tests. They are `not` the images
+    generated by a documentation build (use ``--doc_images_dir`` for specifying that).
+
+* ``--doc_failed_image_dir <DIR>`` dumps copies of cached and generated test images when
+  there is a warning or error raised. This directory is useful for reviewing test
+  failures. It is relative to `pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>`.
+  This will override any configuration, see below.
+
 Configuration
 -------------
 If using ``pyproject.toml`` or any other 
@@ -238,6 +352,16 @@ Similarly, configure the directory that will contain any failed test images:
     [tool.pytest.ini_options]
     failed_image_dir = "failed_images"
 
+Configure directories for when ``--doc_mode`` is used:
+
+.. code-block:: toml
+
+    [tool.pytest.ini_options]
+    doc_failed_image_dir = "failed_test_images"
+    doc_generated_image_dir = "generated_test_images"
+    doc_image_cache_dir = "tests/doc/doc_image_cache"
+    doc_images_dir = "doc/_build/html/_images"
+
 Note that these directories are relative to `pytest root path <https://docs.pytest.org/en/latest/reference/reference.html#pytest.Config.rootpath>`.
 
 Contributing