Skip to content

Commit f41cd77

Browse files
committed
chore: added benchmarks documentation
1 parent e39ae0b commit f41cd77

File tree

1 file changed

+87
-0
lines changed

1 file changed

+87
-0
lines changed

docs/BENCHMARK.md

+87
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,87 @@
1+
# Addition of `VisitedPcs` trait
2+
3+
Current blockifier doesn't store the complete vector of visited program counters
4+
for each entry point call in an invoke transaction. Instead, visited program
5+
counters are pushed in a `HashSet`. This is a limiting factor to perform
6+
profiling operations which require record of the full trace returned by
7+
`cairo-vm`. More flexibility is added to the blockifier with the introduction of
8+
trait `VisitedPcs` which allows the user to process visited program counters in
9+
the most suitable way for the task.
10+
11+
## Existing code
12+
13+
Visited program counters are kept in the `CachedState` structure as shown below:
14+
15+
```rust
16+
#[derive(Debug)]
17+
pub struct CachedState<S: StateReader> {
18+
pub state: S,
19+
// Invariant: read/write access is managed by CachedState.
20+
// Using interior mutability to update caches during `State`'s immutable getters.
21+
pub(crate) cache: RefCell<StateCache>,
22+
pub(crate) class_hash_to_class: RefCell<ContractClassMapping>,
23+
/// A map from class hash to the set of PC values that were visited in the class.
24+
pub visited_pcs: HashMap<ClassHash, HashSet<usize>>,
25+
}
26+
```
27+
28+
## New code
29+
30+
`VisitedPcs` is an additional generic parameter of `CachedState`.
31+
32+
```rust
33+
#[derive(Debug)]
34+
pub struct CachedState<S: StateReader, V: VisitedPcs> {
35+
pub state: S,
36+
// Invariant: read/write access is managed by CachedState.
37+
// Using interior mutability to update caches during `State`'s immutable getters.
38+
pub(crate) cache: RefCell<StateCache>,
39+
pub(crate) class_hash_to_class: RefCell<ContractClassMapping>,
40+
/// A map from class hash to the set of PC values that were visited in the class.
41+
pub visited_pcs: V,
42+
}
43+
```
44+
45+
An implementation of the trait `VisitedPcs` is included in the blockifier with
46+
the name `VisitedPcsSet` and it mimics the existing `HashSet<usize>`. Also, for
47+
test purposes, `CachedState` is instantiated using `VisitedPcsSet`.
48+
49+
## Performance considerations
50+
51+
Given the importance of the blockifier in the Starknet ecosystem, we want to
52+
measure the performance impact of adding the trait `VisitedPcs`. The existing
53+
bechmark `transfers` doesn't cover operations with `CachedState` therefore we
54+
need to design new ones. We have created two new benchmarks:
55+
56+
- `cached_state`: this benchmark tests the performance impact of populating
57+
`visited_pcs` (implemented using `VisitedPcsSet`) with a realistic amount of
58+
visited program counters. The size of the sets is taken from transaction
59+
`0x0177C9365875CAA840EA8F03F97B0E3A8EE8851A8B952BF157B5DBD4FECCB060` in the
60+
mainnet. This transaction has been chosen randomly, but there is no assurance
61+
that it's representative of the most common Starknet invoke transaction. This
62+
benchmark tests the write performance of visited program counters in the state
63+
struct.
64+
- `execution`: this benchmark simulates a whole invoke transaction using a dummy
65+
contract.
66+
67+
## Performance impact
68+
69+
A script `bench.sh` has been added to benchmark the performance impact of these
70+
changes: it is called as
71+
`bash scripts/bench.sh 14e6a87722c1d0c757b1aa2756ffabe3f248fd7d e39ae0be4cec31938399199e0a1070279b4a78ed`.
72+
The computer running the benchmark is: Debian VM over Windows 10 with VMWare
73+
Workstation 17, i9-9900K, 64GB RAM, Samsung 990 Pro NVME SSD.
74+
75+
Noise threshold and confidence intervals are kept as per default Criterion.rs
76+
configuration.
77+
78+
The results are shown in the following table:
79+
80+
| Benchmark | Time (ms) | Time change (%) | Criterion.rs report |
81+
| ------------ | --------- | --------------- | ----------------------------- |
82+
| transfers | 94.448 | +0.1080 | No change in performance |
83+
| execution | 1.2882 | -1.7216 | Change within noise threshold |
84+
| cached_state | 5.2330 | -0.8703 | No change in performance |
85+
86+
The analysis of Criterion.rs determines that there isn't statistically
87+
significant performance decrese.

0 commit comments

Comments
 (0)