Integrate pytest-subtests

[nicoddemus]

This PR copies the files from pytest-subtests and performs minimal integration.
I'm opening this to gauge whether everyone is on board with integrating this feature into the core.

Why?

Pros

• subtests is a standard unittest feature, so it makes sense for pytest to support it as well.
• Provides a simple alternative to parametrization.
• Adds the ability to generate new test cases at runtime during the test execution, which is not possible with parametrization.
• While it can exist as an external plugin, it requires many hacks, and better report integration is not easily achievable without core integration (off the top of my head: issues with terminal reporting, last failed and stepwise support, among others).

Cons

• Adds another maintenance burden to the core.

TODO ✅

If everyone is on board, I will take the time this week to polish it and get it ready to merge ASAP:

• ✅ Cleanup the implementation: Currently it relies on monkey-patching, which should no longer be needed since we can modify the core code directly.
• ✅ Documentation: The feature should be documented as experimental -- meaning that while the feature itself is solid, we are still working out details such as how to best report subtest failures, integration with other plugins, etc.
• ✅ Fix lint and testing failures (obviously).

Related

• Support for Python 3.4 unittest subtests #1367
• Integrate into the core pytest-subtests#71

[Pierre-Sassoulas]

I agree that this feature should be in pytest core because it's a feature in unittest and pytest should aim to be a drop in replacement for unittest (plus the original issue have 40 👍 and no 👎 at time of writing, overwhelming popular support in my book).

[RonnyPfannschmidt]

I recall we need to fix marking the owning test case as failed if one subtest fails

[RonnyPfannschmidt]

But yeah i want to see this in

[webknjaz]

Sounds reasonable

[nicoddemus]

Ready for an initial review folks.

[bluetech]

Nice to see this happening.

I ran out of time for the review for today, so didn't really get to the implementation parts, but already have some comments so submitting a partial review.

[RonnyPfannschmidt]

we might want to bikeshed a api around sub-section, subtests and parameter loops a little - not necessarily for doing right now - but for setting up a roadmap

[nicoddemus]

I was hoping to get this feature into 9.0, if possible.

[RonnyPfannschmidt]

we absolutely want this in 9.0

we shouldnt change the api to ensure compat with existing users
we shoul dplan for mkaing some things nicer

we should try to ensure lastfailed does rerun tests with failed subtests before releasing 9.0

[nicoddemus]

we should try to ensure lastfailed does rerun tests with failed subtests before releasing 9.0

I will add a test, good call.

[bluetech]

Unfortunately ran out of time to do a full review again, but left some comments.

I also noticed two things while I was testing it:

---

If some or all subtests fail, the parent test is still PASSED, is this intentional?

---

For some reason the errors are shown as "Captured log call", seems wrong as there are not log calls in my test.

---

Regarding the name SUBPASS etc., I saw that regular pass is PASSED so I wonder it shouldn't be SUBPASSED etc? (I can check the code later).

[bluetech]

• Can you explain the rationale for these options -- did someone ask for them in pytest-subtests?

• I admit the difference between the options is a bit obscure, I wonder if they can't be folded into a single option?

• I wonder if this shouldn't be an ini option rather than a cli option? I imagine someone would want to set this if it's too noisy, in which case it more of a project setting. But maybe it's more of a user-preference thing, then it's a CLI flag...

• Maybe it should be configurable at the test level, using the fixture itself? (Probably not)

• Should we document the options in the tutorial?

[nicoddemus]

These were external contributions: pytest-dev/pytest-subtests#81 and pytest-dev/pytest-subtests#198.

I'm OK with changing them somehow into a single option, perhaps an option to disable all subtest progress unless it is failure?

[bluetech]

I don't have experience with subtests to say myself. Generally less flags is better :)

pytest-dev/pytest-subtests#81
This allows the native pytest column calculations to not be disrupted and minimizes unneeded output for large CI systems.

Is the "disruption" still a thing?

Otherwise seems to want to minimize output. Though for shortletter it's a bit dubious I must say (it is quite "short"). I suspect what they mean is "we semantically don't view subtests as something that should be reported".

pytest-dev/pytest-subtests#198
I find it extremely annoying that each subtest success is printed to stdout

Wants to minimize output.

an option to disable all subtest progress unless it is failure?

IIUC this is mostly what they want.

[nicoddemus]

Perhaps we should leverage the new "verbosity_" options:

https://docs.pytest.org/en/stable/reference/reference.html#confval-verbosity_assertions

We could include a verbosity_subtests which defaults to only show subtests failure, with additional verbosity being used to also show passed subtests.

[giampaolo]

A few thoughts on this:

1. Since the main motivation appears to be adding a feature unittest already provides, IMO it makes sense to follow its behavior: no extra output if all subtests pass.

2. CLI tools usually disable verbose output by default unless requested. Examples:

• pytest -v ...

• python3 -m unittest -v ...

• PYTHONWARNINGS=always python3 foo.py

• cp -v foo bar

  The PR I made in Add --no-subtest-reports CLI opt pytest-subtests#199 goes in the opposite direction: it adds --no-subtest-reports to suppress verbosity. I think that should not be replicated here.

3. Subtests are mainly for context on failure. If all pass, knowing the exact execution is usually irrelevant. If you do care, it probably means it should be a separate test, not a subtest.

Just my 2 cents, and thanks to @nicoddemus for this very welcome feature.

[nicoddemus]

Now the top-level test will fail with a message in case it contains failed subtests but otherwise does not contain failed assertions on its own:

__________________________________ test_foo ___________________________________
Contains 2 failed subtests

[bluetech]

I don't have experience with subtests to say myself. Generally less flags is better :)

pytest-dev/pytest-subtests#81
This allows the native pytest column calculations to not be disrupted and minimizes unneeded output for large CI systems.

Is the "disruption" still a thing?

Otherwise seems to want to minimize output. Though for shortletter it's a bit dubious I must say (it is quite "short"). I suspect what they mean is "we semantically don't view subtests as something that should be reported".

pytest-dev/pytest-subtests#198
I find it extremely annoying that each subtest success is printed to stdout

Wants to minimize output.

an option to disable all subtest progress unless it is failure?

IIUC this is mostly what they want.

[bluetech]

A few more comments.

[bluetech]

An obscure bug: the logging plugin "live logging" support has stateful calls like self.log_cli_handler.set_when("logreport"). With subtests, it ends up getting a pytest_runtest_logreport while the call of the parent test is still ongoing, it ends up messing the headers ("live log logreport" instead of "live log call"):

def test(subtests):
    logging.error('START')

    with subtests.test():
        logging.error('FIRST')
        assert False

    logging.error('MIDDLE')

    with subtests.test():
        logging.error('SECOND')
        assert False

    logging.error('END')

    assert False

z.py::test 
------------------------------------ live log call ------------------------------------
ERROR    root:z.py:6 START
ERROR    root:z.py:9 FIRST
(<subtest>) SUBFAILED                                                           [100%]
--------------------------------- live log logreport ----------------------------------
ERROR    root:z.py:12 MIDDLE
ERROR    root:z.py:15 SECOND

z.py::test (<subtest>) SUBFAILED                                                [100%]
--------------------------------- live log logreport ----------------------------------
ERROR    root:z.py:18 END

z.py::test FAILED                                                               [100%]

Dunno how much to bother with this...

[bluetech]

In pytest-dev/pytest-subtests#44 I saw something I didn't think about, nested subtests, e.g.

import pytest

def test(subtests):
    with subtests.test("a"):
        with subtests.test("b"):
            1 / 0
        1 / 0

unittest doesn't seem to do anything special with them. Only thing I can think of is to automatically nest the subtest name, e.g. a::b or such. Not sure if it's desirable.

[nicoddemus]

I saw something I didn't think about, nested subtests, e.g.

Good catch. I added a test just to ensure it works and sediment the current behavior. We might decide to add nesting support in the future though.

[nicoddemus]

Thanks @bluetech for the through review!

I implemented most of the suggestions. Still TODO:

• Remove the custom options (Integrate pytest-subtests #13738 (comment)).
• Live logging (Integrate pytest-subtests #13738 (comment))
• Check pytest.skip (Integrate pytest-subtests #13738 (comment))

Hopefully I can get to those tomorrow or over the weekend.