Mirror scanpy's image-comparison tolerances to fix CPU plotting test failures#1083
Merged
Conversation
eroell
reviewed
Jun 30, 2026
eroell
reviewed
Jun 30, 2026
eroell
reviewed
Jun 30, 2026
eroell
left a comment
Collaborator
There was a problem hiding this comment.
- In the PR comment, please use permalinks. This is generally advisable when referring to code. This link currently used will be outdated at one point: https://github.com/scverse/scanpy/blob/main/tests/test_plotting.py#L567. But if you copy a permalink, it remains clear what was referred to, even if scanpy's coded changes: https://github.com/scverse/scanpy/blob/ee7707bc208132cca8387e542d0532f6967f68cc/tests/test_plotting.py#L567
- Linking the scanpy script for plot testing as I suggest in a comment gives indeed quite an immediate glance at how scanpy sets the testing thresholds to 15-40, depending on the plot; much more loose than we currently had it
- Sorry I made accidentally a few comments standalone, should have all been in one review pass to keep the inbox messages at bay
Co-authored-by: Eljas Roellin <65244425+eroell@users.noreply.github.com>
Co-authored-by: Eljas Roellin <65244425+eroell@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Mirror scanpy's image-comparison tolerances to fix CPU plotting test failures
Fixes #1082
CPU test suite (
hatch-test.py3.12,hatch-test.py3.14) was failing onimage-comparison plotting tests with
AssertionError. The comparisons go through check_same_image intests/conftest.py, which wrapsmatplotlib.testing.compare.compare_images(expected, actual, tol=tol)Thetolis an RMS pixel difference on a 0–255 scale. ehrapy was setting it far too strictly (1e-1–2e-1≈ near pixel-perfect), whereas scanpy uses the same mechanism withtolof 5–40 (most commonly 15). matplotlib is pinned and resolves transitively to 3.10.5; small rendering differences across environments blow past the tiny threshold even when plots are visually equivalent.What this PR does
Mirrors scanpy's tolerance approach for the tolerance-solvable failures (30
tests):
temporary mirror of matplotlib in the scanpy dependency list (pyproject.toml): once released scanpy carries the exclusion, it can be removed
tests/conftest.py:check_same_imagenow defaults totol=15(scanpy'smost common value); individual tests bump it only where rendering legitimately
differs more.
Per-test tolerances assigned in clean tiers (each ≥1.3× the observed RMS):
Sensitivity tests keep their tight
tol=1e-1(e.g.test_CohortTracker_plot_cohort_barplot_test_sensitivity,test_CohortTracker_flowchart_image_sensitivity) so they still detectintentional differences — the tiers were chosen to remain meaningful, not so
loose they defeat the comparison.
Two failures are not tolerance-fixable:
test_catplot_vanilla,test_stratified_table_one_plot— image sizemismatch (tolerance-independent;
compare_imagesbails before computing RMS) -> pinned matplotlibReference code on scanpy repo for image comparison tolerance: https://github.com/scverse/scanpy/blob/ee7707bc208132cca8387e542d0532f6967f68cc/tests/test_plotting.py#L567
Reference on scanpy pyproject.toml for pinning matplotlib: https://github.com/scverse/scanpy/blob/d5a745e1258f5ca6a23c6a790f27b40d7ad05418/pyproject.toml#L60