Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

perf(levm): benchmark block execution #1859

Draft
wants to merge 68 commits into
base: main
Choose a base branch
from
Draft

Conversation

dsocolobsky
Copy link
Contributor

@dsocolobsky dsocolobsky commented Jan 31, 2025

Motivation

Benchmark the execution of a block in LEVM, to test a more real case scenario.

Description

  • Added benchmark_block.rs which fetches a block via RPC and executes with levm/revm.
  • Moved some of the RPC code from PR perf(l2): add prover benchmarks from L1 RPC downloaded blocks #1705 to a new crate network/rpc_client since it's code that I have used in this PR and possibly will be used in other places as well in the future.
  • Changed some levm-related functions to allow using ExecutionDB instead of just Store since I needed this for this PR.

Closes #1831

To test

  • revm: run --package revm_comparison --bin benchmark-block -- --rpc-url SOME_RPC_URL
  • levm: run --package revm_comparison --bin benchmark-block --features levm -- --rpc-url SOME_RPC_URL

This will use the latest block number, fetched via RPC.

This will take a while and create a cache in a file named db.bin, you should take note of the block number and pass it as parameter since in the next run it will use that cached block (and run much faster since it won't download all the data again):

run --package revm_comparison --bin benchmark-block -- --rpc-url SOME_RPC_URL --block-number BLOCKNUM

Remember to delete db.bin if you want to test again with another block.

Copy link

github-actions bot commented Jan 31, 2025

| File                                                                                        | Lines | Diff |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/blockchain/payload.rs                                | 643   | -3   |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/proposer/prover_server.rs                         | 459   | +8   |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/prover/zkvm/interface/src/lib.rs                  | 180   | +16  |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/l2/utils/test_data_io.rs                             | 88    | +3   |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/networking/rpc_client/constants.rs                   | 42    | +42  |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/networking/rpc_client/db.rs                          | 285   | +285 |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/networking/rpc_client/rpc_client.rs                  | 296   | +296 |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/storage/store/storage.rs                             | 1203  | +5   |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/db.rs                                             | 250   | +108 |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/execution_db.rs                                   | 172   | -55  |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/bench/revm_comparison/src/benchmark_block.rs | 77    | +77  |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/levm/src/vm.rs                                    | 753   | -3   |
+---------------------------------------------------------------------------------------------+-------+------+
| /home/runner/work/ethrex/ethrex/crates/vm/vm.rs                                             | 914   | +25  |
+---------------------------------------------------------------------------------------------+-------+------+

Total lines added: +865
Total lines removed: 61
Total lines changed: 926

Copy link

github-actions bot commented Jan 31, 2025

Summary: 9451/14408 (65.60%)

Prague: 2373/2373 (100.00%)
Cancun: 3579/3579 (100.00%)
Shanghai: 221/221 (100.00%)
Paris: 62/62 (100.00%)
London: 39/39 (100.00%)
Berlin: 35/35 (100.00%)
Istanbul: 34/35 (97.14%)
Constantinople: 1042/2439 (42.72%)
Byzantium: 1000/2330 (42.92%)
Homestead: 585/1324 (44.18%)
Frontier: 141/742 (19.00%)

Copy link

github-actions bot commented Jan 31, 2025

Benchmark Results Comparison

PR Results

Benchmark Results: Factorial

Command Mean [ms] Min [ms] Max [ms] Relative
revm_Factorial 231.1 ± 1.1 228.4 232.5 1.00
levm_Factorial 905.7 ± 21.1 890.1 960.8 3.92 ± 0.09

Benchmark Results: Factorial - Recursive

Command Mean [s] Min [s] Max [s] Relative
revm_FactorialRecursive 1.419 ± 0.082 1.320 1.538 1.00
levm_FactorialRecursive 15.613 ± 0.017 15.586 15.637 11.00 ± 0.63

Benchmark Results: Fibonacci

Command Mean [ms] Min [ms] Max [ms] Relative
revm_Fibonacci 207.0 ± 1.0 204.9 208.5 1.00
levm_Fibonacci 897.2 ± 7.1 885.8 908.2 4.33 ± 0.04

Benchmark Results: ManyHashes

Command Mean [ms] Min [ms] Max [ms] Relative
revm_ManyHashes 8.6 ± 0.1 8.5 8.7 1.00
levm_ManyHashes 18.0 ± 0.1 17.8 18.1 2.10 ± 0.02

Benchmark Results: BubbleSort

Command Mean [s] Min [s] Max [s] Relative
revm_BubbleSort 3.207 ± 0.013 3.187 3.230 1.00
levm_BubbleSort 6.044 ± 0.041 6.002 6.111 1.88 ± 0.01

Benchmark Results: ERC20 - Transfer

Command Mean [ms] Min [ms] Max [ms] Relative
revm_ERC20Transfer 257.1 ± 4.5 248.3 261.9 1.00
levm_ERC20Transfer 536.3 ± 3.8 532.7 544.1 2.09 ± 0.04

Benchmark Results: ERC20 - Mint

Command Mean [ms] Min [ms] Max [ms] Relative
revm_ERC20Mint 141.1 ± 0.6 140.3 142.1 1.00
levm_ERC20Mint 349.6 ± 5.0 345.5 362.6 2.48 ± 0.04

Benchmark Results: ERC20 - Approval

Command Mean [s] Min [s] Max [s] Relative
revm_ERC20Approval 1.051 ± 0.005 1.043 1.058 1.00
levm_ERC20Approval 2.013 ± 0.013 1.995 2.035 1.92 ± 0.02

Main Results

Benchmark Results: Factorial

Command Mean [ms] Min [ms] Max [ms] Relative
revm_Factorial 234.5 ± 2.4 232.9 241.1 1.00
levm_Factorial 896.9 ± 9.5 886.6 915.3 3.83 ± 0.06

Benchmark Results: Factorial - Recursive

Command Mean [s] Min [s] Max [s] Relative
revm_FactorialRecursive 1.476 ± 0.103 1.330 1.605 1.00
levm_FactorialRecursive 15.625 ± 0.037 15.575 15.700 10.59 ± 0.74

Benchmark Results: Fibonacci

Command Mean [ms] Min [ms] Max [ms] Relative
revm_Fibonacci 202.0 ± 3.5 199.2 211.7 1.00
levm_Fibonacci 904.3 ± 16.9 885.8 941.7 4.48 ± 0.11

Benchmark Results: ManyHashes

Command Mean [ms] Min [ms] Max [ms] Relative
revm_ManyHashes 8.9 ± 0.3 8.7 9.7 1.00
levm_ManyHashes 18.4 ± 0.3 18.2 19.1 2.07 ± 0.07

Benchmark Results: BubbleSort

Command Mean [s] Min [s] Max [s] Relative
revm_BubbleSort 3.212 ± 0.015 3.181 3.243 1.00
levm_BubbleSort 6.119 ± 0.085 6.040 6.278 1.91 ± 0.03

Benchmark Results: ERC20 - Transfer

Command Mean [ms] Min [ms] Max [ms] Relative
revm_ERC20Transfer 249.1 ± 2.6 247.3 254.9 1.00
levm_ERC20Transfer 536.6 ± 4.1 531.0 545.7 2.15 ± 0.03

Benchmark Results: ERC20 - Mint

Command Mean [ms] Min [ms] Max [ms] Relative
revm_ERC20Mint 143.5 ± 3.0 141.8 151.9 1.00
levm_ERC20Mint 349.8 ± 4.0 346.0 360.4 2.44 ± 0.06

Benchmark Results: ERC20 - Approval

Command Mean [s] Min [s] Max [s] Relative
revm_ERC20Approval 1.041 ± 0.011 1.028 1.068 1.00
levm_ERC20Approval 2.013 ± 0.010 1.997 2.025 1.93 ± 0.02

@dsocolobsky dsocolobsky changed the base branch from l2/rpc_bench to main February 5, 2025 15:13
@dsocolobsky dsocolobsky force-pushed the levm/benchmark_block branch 2 times, most recently from 2e32c92 to ce16c5d Compare February 6, 2025 20:01
@dsocolobsky dsocolobsky force-pushed the levm/benchmark_block branch 3 times, most recently from 8edf075 to 1dd6112 Compare February 6, 2025 20:44
db.bin Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

LEVM: Benchmark a mainnet's block execution
3 participants