Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use splittable RNG in QCheck2 #318

Merged
merged 19 commits into from
Jan 31, 2025
Merged

Conversation

jmid
Copy link
Collaborator

@jmid jmid commented Jan 27, 2025

Fixes #317

This PR fixes the weird shrinking behaviour in QCheck2 as seen in #317. With it generators built with

  • Gen.bind
  • Gen.bytes_size (and by extension all Gen.bytes* and Gen.string* generators)
  • Gen.list_size (and by extension Gen.list*, Gen.array*, and function generators)

start exhibiting deterministic shrinking. The problem is that in generators such as Gen.(x >>= f)
the underlying Random.State has state, but is computed lazily. As such,

  • we need to split the RNG, so that any delayed generation in f is unaffected by, e.g., fewer RNG invocations in the Random.State passed to x and
  • we need to copy the RNG, so that f when restarted (e.g., with a smaller size input) restarts from the same Random.State.

This is not new, e.g., in #156 (comment) I commented:

It is however harder to observe how the size/spine-shrinker proceeds, as the element-generator is restarted in a different Random.State for each size in the size shrink tree (we really need that splittable RNG):
[...]

The integrated shrinking requires a splittable RNG - and OCaml 4's Random.State module does not offer one.
As such, on OCaml 4, the PR builds on a horrible split hack I added some years back. I cannot see us switching to, e.g., https://github.com/ocaml/stdlib-random, as Random.State is exposed in QCheck's interfaces, so a switch would be backward incompatible. In addition, the Random.State.make used to hack around the absence of split is costly, meaning QCheck2 regrettably performs worse on OCaml4 than on OCaml5+.

  • The PR first documents the weird behavior in QCheck2 Shrink unit tests (with generated input during shrinking marked WTF)
  • It then sets out to fix the above, one after another, and updates the unit test outputs and expect test outputs underway
  • In addition, he PR adds appropriate note about QCheck2's maturity

As such, should anyone be reading this PR, it is recommended to do so, commit-by-commit.

Finally: The PR does not solve all QCheck2's shrinking problems such as #157 and #163.
Getting QCheck2's shrinker trees to act more deterministically and less like bogosort is however a prerequisite and a step towards improving its shrinker algorithms (behaviour + complexity).

@jmid jmid merged commit 4e5b3df into c-cube:main Jan 31, 2025
13 checks passed
@jmid jmid deleted the qcheck2-use-splittable-rng branch January 31, 2025 16:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Ensure independence in QCheck2's monadic combinators
1 participant