Skip to content

feat: Port Criterion to Lean #153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 9 commits into from
Jun 20, 2025
Merged

feat: Port Criterion to Lean #153

merged 9 commits into from
Jun 20, 2025

Conversation

samuelburnham
Copy link
Member

@samuelburnham samuelburnham commented Jun 13, 2025

Ports the core benchmarking and statistical analysis logic from https://github.com/bheisler/criterion.rs to Lean. This includes:

  • Warmup time to estimate total runtime
  • Flat and linear sampling modes
  • Sample analysis: Bootstrap sampling, point estimates and distributions for mean, stdDev, median, etc., and linear regression if in linear sampling mode
  • Outlier detection via Tukey's method
  • Disk caching as Ixon data in .lake/benches for comparison between runs
  • Change analysis: Bootstrapped T-test and p value, noise threshold
  • Configurable parameters
  • Formatted output similar to Criterion.rs

Usage: lake exe bench will run all of the benchmarks specified in the Benchmarks.Main file's main function.

Some TODOs for another PR:

  • Throughput measurement
  • General performance enhancements
  • Testing
  • Documentation
  • Logging/verbosity
  • Disk cache for Tukey outliers
  • Plots

@arthurpaulino
Copy link
Member

This is great work already!
Can you break this patch so that you don't need to complete the whole TODO list before we can use it, please?

@samuelburnham samuelburnham marked this pull request as ready for review June 19, 2025 22:47
@samuelburnham samuelburnham changed the title Port Criterion to Lean (WIP) feat: Port Criterion to Lean Jun 19, 2025
Copy link
Member

@arthurpaulino arthurpaulino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Left some comments.

Module docstrings are written as

/-!
# A header if necessary
Some markdown text
-/

Copy link
Member

@arthurpaulino arthurpaulino left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎉 🥳 🎊

@samuelburnham samuelburnham merged commit b598456 into main Jun 20, 2025
12 checks passed
@samuelburnham samuelburnham deleted the benchmarks branch June 20, 2025 16:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants