Seed based on commit SHA

[adamchainz]

Description

Suggestion from chat with @graingert and @seporaitis: add a --randomly-seed-vcs or similar option that uses the current Git commit SHA as the seed. That way, parallel test runs for a given commit, such as with pytest-split, use the same seed, but it still changes over time.

[seporaitis]

I like the idea of feeding a consistent seed into pytest-randomly, it would definitely help, but I think it should not be linked to a VCS.

One of the appeals of pytest-randomly is the ability to note the seed, and restart the test suite, then come back and fix the flakes later.

Pinning the seed to a commit hash would make the test fail consistently and there the escape hatch would not exist (talking about ci environments here).

I think what would work much better is if pytest-randomly could read the seed from an environment variable. This way it's up to the caller to set it, and e.g. on github actions one could use run id + an attempt number, or store the seed as artifact and reuse it (thinking about my own use case here).

Having written above, I'm also wondering if there's even a clean solution to this. If I use pytest-split and have a matrix of GitHub action jobs, the only way for me to retry a single instance of a job is by pinning the seed, otherwise I'd be running a different set of tests on a reattempt. Perhaps the solution to this is more in pytest-split and not in pytest-randomly