pytest assertion rewriting does not seem to work with meson editable install

[lesteve]

It could well be partially linked to some of the discussion in #642 but I thought a separate issue may be worth it.

Description of the issue

The pytest assertion rewriting that shows additional debug information when an assert fails does not show up when using meson editable install. It does work fine when using normal (non-editable) meson install. Although not crucial I would argue that this can be actually quite useful in CI logs.

It could also well be a Pytest issue but I thought I had to start somewhere and chose meson-python.

More background about Pytest assertion rewriting, see Pytest doc that links to a somewhat old blog post but I am guessing is probably still relevant?

This was originally seen in scikit-learn (see scikit-learn/scikit-learn#29253 (comment) if you are really curious where it would have actually been quite useful to see intermediary values in the AssertionError in scikit-learn/scikit-learn#29253 (comment)). I put together a reproducer below with numpy hoping this makes it a bit easier to look at. I guess a smaller reproducer could be put together if deemed useful, but it would take a bit of time (I tried with https://github.com/dnicolodi/python-siphash24 and I am not able to reproduce the issue).

Get the reproducer from my numpy branch

git clone https://github.com/lesteve/numpy
cd numpy
git checkout meson-editable-pytest-assertion-rewrite

I added a simple test in numpy/tests/test_mine.py

import numpy as np


def test():
    arr = np.arange(100)
    assert arr.sum() < 100

When running with pytest you pytest assertion rewriting shows the value or arr.sum() (see below).

Normal (non-editable) install

pip install --verbose --no-build-isolation .

Note the assertion rewriting i.e. you get the value of arr.sum()

Partial output:

E       assert np.int64(4950) < 100
E        +  where np.int64(4950) = <built-in method sum of numpy.ndarray object at 0x7b77efd7a370>()

Full output

 ❯ pytest --pyargs numpy.tests.test_mine
========================== test session starts ===========================
platform linux -- Python 3.11.7, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/lesteve
plugins: hypothesis-6.103.2
collected 1 item                                                         

micromamba/envs/numpy-dev/lib/python3.11/site-packages/numpy/tests/test_mine.py F [100%]

================================ FAILURES ================================
__________________________________ test __________________________________

    def test():
        arr = np.arange(100)
>       assert arr.sum() < 100
E       assert np.int64(4950) < 100
E        +  where np.int64(4950) = <built-in method sum of numpy.ndarray object at 0x7b77efd7a370>()
E        +    where <built-in method sum of numpy.ndarray object at 0x7b77efd7a370> = array([ 0,  1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12, 13, 14, 15, 16,\n       17, 18, 19, 20, 21, 22, 23, 24, 25, ...72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84,\n       85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99]).sum

micromamba/envs/numpy-dev/lib/python3.11/site-packages/numpy/tests/test_mine.py:6: AssertionError
======================== short test summary info =========================
FAILED micromamba/envs/numpy-dev/lib/python3.11/site-packages/numpy/tests/test_mine.py::test - assert np.int64(4950) < 100
=========================== 1 failed in 0.82s ============================

Editable install

pip install --verbose --no-build-isolation --editable . --config-settings editable-verbose=true

You don't get the value of arr.sum()

 pytest --pyargs numpy.tests.test_mine
========================== test session starts ===========================
platform linux -- Python 3.11.7, pytest-8.2.2, pluggy-1.5.0
rootdir: /home/lesteve
plugins: hypothesis-6.103.2
collecting ... + /home/lesteve/micromamba/envs/numpy-dev/bin/ninja
[1/1] Generating numpy/generate-version with a custom command
Saving version to numpy/version.py
collected 1 item                                                         

dev/numpy/numpy/tests/test_mine.py F                               [100%]

================================ FAILURES ================================
__________________________________ test __________________________________

    def test():
        arr = np.arange(100)
>       assert arr.sum() < 100
E       AssertionError

dev/numpy/numpy/tests/test_mine.py:6: AssertionError
======================== short test summary info =========================
FAILED dev/numpy/numpy/tests/test_mine.py::test - AssertionError
=========================== 1 failed in 1.02s ============================

[dnicolodi]

I don't think anyone active in maintaining meson-python has the time to dig into this. The pytest maintainer are much better positioned to understand where things go wrong. Once we know what the problem is, we can look into fixing it.

I'm very surprised that pytest works for simple packages but not for more complex ones. The editable package loading code does not care whether the package contains an extension module or not. Thus the issue should be reproducible with a pure Python package too.

[lesteve]

OK thanks a lot for your insights! I'll try to come up with a simpler reproducer and open an issue in Pytest to see whether they have some intuition where the problem lies.

[fcharras]

I have setup a minimal reproducer at https://github.com/fcharras/meson-python-reproducer646 .

It seems that this issue can be worked around by using pytest' importlib import mode, e.g. by uncommenting this line or by passing --import-mode importlib to the pytest command.

It is, in fact, better than a workaround, because this option is also recommended. The default is set on the legacy value prepend for historical or compatibility reasons.

So this issue would mostly impact projects that can't use the importlib import mode.

[lesteve]

cc @DougBurke for the pytest --import-mode importlib work-around for making pytest assertion rewriting work with meson editable install. You may want to double-check switching to --import-mode importlib does not break other things in your test suite though. Full disclosure, it worked fine for scikit-learn scikit-learn/scikit-learn#31209.