Add timeout flag, to timeout test run after some number of minutes #79
Description
Proposal to add --repeat-timeout=minutes, with the type being a float.
This borrowed functionality from pytest-flakefinder that has a --flake-max-minutes.
--repeat-timeout vs --repeat-max-minutes
I'm thinking --repeat-timeout seems more obvious, and a bit shorter.
int vs float
flakefinder uses an integer. I propose using a float. It doesn't make that much difference in practice.
But testing the feature would be way more convenient if we didn't have to wait a minute for a timeout.
skip vs exit
flakefinder uses skips to skip remaining tests.
That works, but is annoying with tons of skips.
I propose using pytest.exit() to avoid all of the pointless skips.
Also, then you can just guess a big number. "repeat like 20,000 times, but stop after an hour" kind of thing.
implementation
The implementation is fairly simple.
At the start of a test run, store an expiration time of time.time() + (timeout_minutes * 60).
Then at test time, compare the expiration time to the current time.
If timeout, stop testing.
This is pretty simple to implement.
affect on existing usage
My only concern would be if the functionality slows down test runs not using it.
The code added when NOT used is one variable check, and seems to not affect test time if not used.
alternatives
This functionality could also be added as a separate plugin entirely, as there are totally reasons to want to ensure a test suite doesn't run too long.
There is pytest-timeout, but the timeout there applies to individual tests, not the suite.
If added as a new plugin, perhaps something like pytest-suite-timeout or something?
keeping it with pytest-repeat
I could see this being a separate plugin. But also, I think it's a pretty common use case especially with pytest-repeat.
The idea being some way to implement "repeat it a bunch of times, but not more than like 30 minutes".
So, I'm thinking it would be good to include with pytest-repeat.