Support live visualizations with Tyche, via the `OpenPBTStats` log format?

[Zac-HD]

Tyche is a VS Code extension designed to help users understand what their property-based tests are doing - with nice features like live-updating graphs, and lots of explorable raw data if you want to dig deeper.

I've been collaborating with the author since late last year (as a Hypothesis maintainer), as described in our paper. Notably, this helped us find several performance problems and outright bugs in Hypothesis itself, as well as being useful for our users.

But the reason I'm opening this issue is that Tyche is not Hypothesis-specific; it's instead based on a simple json-lines format which could be emitted from any PBT library. If you want to support it from scalacheck, emitting the type, run_start, property, status, and representation is enough to enable a substantial portion of Tyche’s features.

[Swiddis]

I'm interested in implementing this for our test suite, but struggling to figure out how to do so. I found the TestCallback trait which allows hooking into individual property evaluations, but this doesn't seem to give us access to run_start or representation (or property in a generic way). It seems like implementing this would require directly emitting some custom log lines from within all properties, and then mapping those lines to OpenPBTStats logs after the tests run.

Maybe there's something more invasive that can be done upstream. I could look at making a PR if there's a specific entrypoint that would work well. (Admittedly fairly new to the Scala ecosystem)

[Swiddis]

This feature is experimental, and could have breaking changes or even be removed without notice. Try it out, let us know what you think, but don’t rely on it just yet! (source)

Do we know what the timeline for stabilization is? If the format is stable we could change the issue to "Support OpenPBTStats logging," and Tyche support would be accomplished by proxy.

[Zac-HD]

It's not so much a timeline, as that I've been reserving the ability to make small changes to the format if that would help other PBT frameworks to adopt it. So if ScalaCheck maintainers are happy with the current format I'd be inclined to declare it finished!

[Zac-HD]

FWIW I now personally consider the format stable; it's increasingly widely used and we haven't seen any problems so far.