Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add benchmarks comparing to Cairo VM #692

Merged
merged 39 commits into from
Dec 9, 2024
Merged

Add benchmarks comparing to Cairo VM #692

merged 39 commits into from
Dec 9, 2024

Conversation

JulianGCalderon
Copy link
Contributor

@JulianGCalderon JulianGCalderon commented Jun 18, 2024

This PR benchmarks cairo native against cairo vm with criterion:

  • Benchmarks cairo VM with fibonacci, factorial, and logistic map programs.
  • Groups the benchmarks by program

Compile time is not taken into account for cairo native runs.

Benchmarking Server

Factorial:

factorial_2M/Cached JIT time:   [290.86 ms 291.00 ms 291.15 ms]
Found 3 outliers among 100 measurements (3.00%)
  1 (1.00%) high mild
  2 (2.00%) high severe

factorial_2M/Cached AOT time:   [904.06 ms 906.65 ms 909.17 ms]

factorial_2M/VM         time:   [3.7880 s 3.8034 s 3.8197 s]
Found 2 outliers among 100 measurements (2.00%)
  1 (1.00%) high mild
  1 (1.00%) high severe

Fibonacci:

fibonacci_2M/Cached JIT time:   [6.8784 ms 6.8821 ms 6.8864 ms]
Found 5 outliers among 100 measurements (5.00%)
  5 (5.00%) high severe

fibonacci_2M/Cached AOT time:   [18.645 ms 18.665 ms 18.688 ms]
Found 13 outliers among 100 measurements (13.00%)
  2 (2.00%) low severe
  11 (11.00%) high severe

fibonacci_2M/VM         time:   [3.4395 s 3.4521 s 3.4659 s]
Found 3 outliers among 100 measurements (3.00%)
  3 (3.00%) high severe

Logistic Map:

logistic_map/Cached JIT time:   [45.216 ms 45.244 ms 45.274 ms]
Found 13 outliers among 100 measurements (13.00%)
  8 (8.00%) high mild
  5 (5.00%) high severe

logistic_map/Cached AOT time:   [163.22 ms 163.48 ms 163.76 ms]
Found 16 outliers among 100 measurements (16.00%)
  16 (16.00%) high mild

logistic_map/VM         time:   [37.747 ms 37.778 ms 37.811 ms]
Found 6 outliers among 100 measurements (6.00%)
  5 (5.00%) high mild
  1 (1.00%) high severe

Macbook Air M1

factorial_2M/Cached JIT
                        time:   [271.52 ms 272.40 ms 274.08 ms]
factorial_2M/Cached AOT
                        time:   [734.26 ms 736.76 ms 740.13 ms]
factorial_2M/VM
                        time:   [3.9340 s 3.9436 s 3.9551 s]

fibonacci_2M/Cached JIT
                        time:   [7.6131 ms 7.6222 ms 7.6349 ms]
fibonacci_2M/Cached AOT
                        time:   [17.554 ms 17.562 ms 17.571 ms]
fibonacci_2M/VM
                        time:   [3.7504 s 3.7533 s 3.7565 s]

logistic_map/Cached JIT
                        time:   [45.113 ms 45.124 ms 45.136 ms]
logistic_map/Cached AOT
                        time:   [126.11 ms 126.17 ms 126.25 ms]
logistic_map/VM
                        time:   [42.685 ms 42.710 ms 42.735 ms]

Copy link

github-actions bot commented Jun 18, 2024

Benchmark results Main vs HEAD.

Command Mean [s] Min [s] Max [s] Relative
base factorial_2M.cairo (JIT) 3.378 ± 0.039 3.328 3.444 1.00
base factorial_2M.cairo (AOT) 3.463 ± 0.067 3.370 3.579 1.03 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
:--- ---: ---: ---: ---:
head factorial_2M.cairo (JIT) 3.228 ± 0.020 3.206 3.266 1.00
head factorial_2M.cairo (AOT) 3.397 ± 0.029 3.356 3.434 1.05 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base fib_2M.cairo (JIT) 2.753 ± 0.013 2.734 2.770 1.00
base fib_2M.cairo (AOT) 2.974 ± 0.044 2.932 3.058 1.08 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
:--- ---: ---: ---: ---:
head fib_2M.cairo (JIT) 2.734 ± 0.020 2.713 2.774 1.00
head fib_2M.cairo (AOT) 2.933 ± 0.016 2.904 2.952 1.07 ± 0.01
Command Mean [s] Min [s] Max [s] Relative
base logistic_map.cairo (JIT) 3.073 ± 0.042 3.026 3.164 1.00
base logistic_map.cairo (AOT) 3.112 ± 0.019 3.091 3.152 1.01 ± 0.02
Command Mean [s] Min [s] Max [s] Relative
:--- ---: ---: ---: ---:
head logistic_map.cairo (JIT) 3.182 ± 0.091 3.063 3.318 1.01 ± 0.03
head logistic_map.cairo (AOT) 3.165 ± 0.046 3.106 3.262 1.00

@codecov-commenter
Copy link

codecov-commenter commented Jun 21, 2024

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 82.54%. Comparing base (20e1d9f) to head (abdfcbd).

Additional details and impacted files
@@           Coverage Diff           @@
##             main     #692   +/-   ##
=======================================
  Coverage   82.54%   82.54%           
=======================================
  Files         117      117           
  Lines       33516    33516           
=======================================
+ Hits        27665    27666    +1     
+ Misses       5851     5850    -1     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@JulianGCalderon JulianGCalderon changed the title Add benches against Cairo VM Add benchmarks against Cairo VM Jun 25, 2024
@JulianGCalderon JulianGCalderon changed the title Add benchmarks against Cairo VM Add benchmarks comparing to Cairo VM Jun 25, 2024
@JulianGCalderon JulianGCalderon marked this pull request as ready for review June 28, 2024 14:09

factorial_group.bench_function("VM", |b| {
b.iter(|| {
factorial_runner.run_function_with_starknet_context(
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We need to find another way to execute the VM.
See the fn run_function_with_starknet_context

        let assembled_program = self.casm_program.clone().assemble_ex(&entry_code, &footer);

It clone the casm_program adding lot of overhead to the execution

We need to discuss how we do this, maybe we have to add a new method to the cairo-vm/cairo1-run crate

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that particular issue has been fixed: https://github.com/starkware-libs/cairo/blob/226c3646be247828c08b737a73f6da10de76e0fc/crates/cairo-lang-runner/src/lib.rs#L191

let assembled_cairo_program = self.casm_program.assemble_ex(&header, &footer);

@pefontana
Copy link
Collaborator

Other thing to take into account, maybe is easier to benchmark a contract execution

@azteca1998 azteca1998 added this pull request to the merge queue Dec 9, 2024
Merged via the queue into main with commit 5378651 Dec 9, 2024
32 checks passed
@azteca1998 azteca1998 deleted the benches_vs_vm branch December 9, 2024 20:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants