New feature --mp-clique

[sangongs]

This pr implements a new feature that enables clustering processes to cliques. Cliques are considered to be isolated, which means they do not share resources, thus tests marked as isolated_free or isolated_serial can be run in parallel. This feature is useful if the shared resources are on a remote machine.

The feature is exposed to the user with a command line option --mp-clique. Multiple --mp-clique options are allowed, each option corresponds to a clique. The value of the option specifies the options apply to the clique. One example would be:

py.test --mp-clique=--base-url=https://10.10.0.1 --mp-clique=--base-url=https://10.10.0.2 --mp --np 4

Two new hookers are introduced: pytest_mp_configure and pytest_mp_prefork. pytest_mp_configure is called after the clique specific options passed to --mp-clique are parsed. pytest_mp_prefork is called before forking a new test process.

[simfarm]

Ping @rmfitzpatrick for thoughts.

[rmfitzpatrick]

@sangongs, what is a target use case for this feature? If groups are marked isolated, the intention is that they should be run with no other test activity or they don't really need to be marked isolated at all.

[sangongs]

@rmfitzpatrick Great question! This is used to test against multiple isolated and idempotent testing environments in parallel.

A test is marked as isolated because they need to modify shared resources. For example, if a test tries to restart a server which is used by other tests, that test should be marked as isolated.

However, if we can set up multiple idempotent but isolated servers, then the test can restart one server and other tests can work against other servers simultaneously.

With the two newly introduced hooks, pytest-mp can collaborate with conftest.py to set up isolated and idempotent testing environments. In conftest.py, it can hook pytest_mp_configure() to create testing environments and then switch the environment in pytest_mp_prefork(). For instance, conftest.py can create authorized connections to multiple servers. Thus before forking a new process, it can switch the testing context to an appropriate connection.

Any thoughts will be welcome. Especially for the option name. Clique might not be the best word to epitomize this feature.

[rmfitzpatrick]

@sangongs, my thoughts for what they're worth are that I'm not entirely on board with this feature for a number of reasons, largely stemming from the relative complexity it introduces and the multiple systems under test for a single pytest invocation design it seems to incorporate. TBH the functionality it provides feels like an XY solution where isolation is viewed as a testing deficiency over that of a system constraint and bypassing it with a standby test target isn't something I'd prefer be pushed right.
However, knowing the initial use case for this plugin, in the limited review I've been able to provide it seems to me that the implementation is sound.