-
Notifications
You must be signed in to change notification settings - Fork 90
Event log rotation #1997
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: master
Are you sure you want to change the base?
Event log rotation #1997
Conversation
ffakenz
commented
May 8, 2025
- CHANGELOG updated or not needed
- Documentation updated or not needed
- Haddocks updated or not needed
- No new TODOs introduced or explained herafter
Transaction costsSizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using
Script summary
|
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | 5837 | 10.56 | 3.35 | 0.52 |
2 | 6038 | 12.37 | 3.91 | 0.54 |
3 | 6239 | 14.26 | 4.50 | 0.57 |
5 | 6645 | 18.55 | 5.85 | 0.63 |
10 | 7644 | 28.68 | 9.02 | 0.78 |
43 | 14282 | 99.11 | 30.98 | 1.80 |
Commit
transaction costs
This uses ada-only outputs for better comparability.
UTxO | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | 558 | 2.44 | 1.16 | 0.20 |
2 | 742 | 3.38 | 1.73 | 0.22 |
3 | 919 | 4.36 | 2.33 | 0.24 |
5 | 1278 | 6.41 | 3.60 | 0.28 |
10 | 2179 | 12.13 | 7.25 | 0.40 |
54 | 10049 | 98.61 | 68.52 | 1.88 |
CollectCom
transaction costs
Parties | UTxO (bytes) | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|---|
1 | 57 | 525 | 24.46 | 7.13 | 0.42 |
2 | 114 | 636 | 34.20 | 9.84 | 0.53 |
3 | 170 | 747 | 42.50 | 12.21 | 0.61 |
4 | 227 | 858 | 53.42 | 15.24 | 0.73 |
5 | 282 | 969 | 63.64 | 18.07 | 0.84 |
6 | 337 | 1081 | 67.70 | 19.48 | 0.89 |
7 | 395 | 1196 | 72.17 | 20.91 | 0.94 |
8 | 450 | 1303 | 87.00 | 24.85 | 1.09 |
9 | 505 | 1414 | 95.07 | 27.57 | 1.18 |
Cost of Increment Transaction
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | 1790 | 24.24 | 8.05 | 0.48 |
2 | 1928 | 25.40 | 9.11 | 0.51 |
3 | 2111 | 28.09 | 10.67 | 0.55 |
5 | 2396 | 31.13 | 12.97 | 0.60 |
10 | 3019 | 38.35 | 18.60 | 0.73 |
40 | 7548 | 94.19 | 56.84 | 1.65 |
Cost of Decrement Transaction
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | 605 | 22.73 | 7.33 | 0.41 |
2 | 813 | 25.45 | 8.75 | 0.45 |
3 | 946 | 26.06 | 9.57 | 0.47 |
5 | 1196 | 29.15 | 11.76 | 0.52 |
10 | 2093 | 40.50 | 18.28 | 0.70 |
40 | 6528 | 96.16 | 53.80 | 1.60 |
Close
transaction costs
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | 659 | 29.00 | 9.24 | 0.48 |
2 | 880 | 29.74 | 10.31 | 0.50 |
3 | 873 | 31.95 | 11.39 | 0.53 |
5 | 1244 | 34.15 | 13.60 | 0.58 |
10 | 2009 | 44.42 | 20.41 | 0.74 |
38 | 6148 | 99.09 | 56.93 | 1.63 |
Contest
transaction costs
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | 682 | 33.70 | 10.50 | 0.53 |
2 | 811 | 35.73 | 11.78 | 0.56 |
3 | 953 | 37.78 | 13.07 | 0.59 |
5 | 1279 | 42.43 | 15.93 | 0.66 |
10 | 2059 | 54.25 | 23.16 | 0.85 |
29 | 4916 | 96.34 | 49.18 | 1.50 |
Abort
transaction costs
There is some variation due to the random mixture of initial and already committed outputs.
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | 5712 | 27.04 | 9.05 | 0.69 |
2 | 5789 | 32.45 | 10.81 | 0.75 |
3 | 5972 | 44.32 | 14.84 | 0.88 |
4 | 6185 | 55.03 | 18.55 | 1.00 |
5 | 6174 | 56.49 | 18.88 | 1.01 |
6 | 6302 | 71.75 | 23.99 | 1.18 |
7 | 6565 | 80.21 | 26.89 | 1.28 |
8 | 6669 | 91.40 | 30.63 | 1.40 |
FanOut
transaction costs
Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.
Parties | UTxO | UTxO (bytes) | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|---|---|
10 | 0 | 0 | 5834 | 19.19 | 6.41 | 0.61 |
10 | 5 | 284 | 6003 | 29.80 | 10.58 | 0.73 |
10 | 10 | 569 | 6173 | 39.90 | 14.59 | 0.85 |
10 | 20 | 1137 | 6511 | 59.32 | 22.30 | 1.08 |
10 | 30 | 1706 | 6853 | 80.09 | 30.47 | 1.32 |
10 | 39 | 2216 | 7155 | 98.99 | 37.90 | 1.54 |
End-to-end benchmark results
This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master
code.
Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.
Generated at 2025-05-09 23:31:27.50569734 UTC
Baseline Scenario
Number of nodes | 1 |
---|---|
Number of txs | 300 |
Avg. Confirmation Time (ms) | 4.279488786 |
P99 | 6.554354709999954ms |
P95 | 4.7713907ms |
P50 | 4.0870145ms |
Number of Invalid txs | 0 |
Memory data
Time | Used | Free |
---|---|---|
2025-05-09 23:30:11.224451968 UTC | 874M | 6494M |
2025-05-09 23:30:16.224330964 UTC | 992M | 6272M |
2025-05-09 23:30:21.224332527 UTC | 998M | 6265M |
2025-05-09 23:30:26.224252212 UTC | 998M | 6265M |
2025-05-09 23:30:31.224265639 UTC | 1016M | 6246M |
2025-05-09 23:30:36.224316378 UTC | 1016M | 6246M |
Three local nodes
Number of nodes | 3 |
---|---|
Number of txs | 900 |
Avg. Confirmation Time (ms) | 28.151331124 |
P99 | 43.0017671ms |
P95 | 37.216210749999995ms |
P50 | 26.982078ms |
Number of Invalid txs | 0 |
Memory data
Time | Used | Free |
---|---|---|
2025-05-09 23:30:49.699914439 UTC | 932M | 6340M |
2025-05-09 23:30:54.700003089 UTC | 1135M | 6135M |
2025-05-09 23:30:59.70035955 UTC | 1200M | 6012M |
2025-05-09 23:31:04.700094328 UTC | 1213M | 5949M |
2025-05-09 23:31:09.699955797 UTC | 1214M | 5948M |
2025-05-09 23:31:14.699940436 UTC | 1218M | 5943M |
2025-05-09 23:31:19.699962555 UTC | 1219M | 5942M |
2025-05-09 23:31:24.699958694 UTC | 1220M | 5940M |
Transaction cost differencesScript summary
|
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | - | - | - | - |
2 | - | - | - | - |
3 | - | - | - | - |
5 | - | - | - | - |
10 | - | - | - | - |
43 | - | - | - | - |
Commit
transaction costs
UTxO | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | - | - | - | - |
2 | - | - | - | - |
3 | - | - | - | - |
5 | - | - | - | - |
10 | - | - | - | - |
54 | - | - | - | - |
CollectCom
transaction costs
Parties | UTxO (bytes) | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|---|
1 | - | - | - | - | - |
2 | - | - | - | - | - |
3 | - | - | - | - | - |
4 | - | - | - | - | - |
5 | - | - | - | - | - |
6 | - | - | - | - | - |
7 | - | - | - | - | - |
8 | - | - | - | - | - |
9 | - | - | - | - | - |
Cost of Increment Transaction
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | - | - | ||
2 | - | - | ||
3 | - | - | - | - |
5 | - | +0.38 | +0.09 | - |
10 | - | - | - | - |
43 | - | - | - | - |
Cost of Decrement Transaction
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | - | - | - | - |
2 | - | - | - | - |
3 | - | - | - | - |
5 | - | - | - | - |
10 | - | - | - | - |
40 | - | - | - | - |
Close
transaction costs
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | - | - | - | - |
2 | - | - | - | - |
3 | - | - | - | - |
5 | - | - | - | - |
10 | - | - | - | - |
35 | - | - | - | - |
Contest
transaction costs
Parties | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|
1 | - | - | - | - |
2 | - | - | - | - |
3 | - | - | - | - |
5 | - | - | - | - |
10 | - | - | - | - |
29 | - | - | - | - |
FanOut
transaction costs
UTxO, Parties | UTxO (bytes) | Tx size | % max Mem | % max CPU | Min fee ₳ |
---|---|---|---|---|---|
(0, 10) | - | - | - | - | - |
(1, 10) | - | - | - | - | - |
(5, 10) | - | - | - | - | - |
(10, 10) | - | - | - | - | - |
(20, 10) | - | - | - | - | - |
(38, 10) | - | - | - | - | - |
currentEvents <- getEvents eventSource | ||
let currentNumberOfEvents = toInteger $ length currentEvents | ||
numberOfEventsV <- newTVarIO currentNumberOfEvents | ||
-- XXX: check rotation on startup | ||
when (currentNumberOfEvents >= toInteger rotateAfterX) $ do | ||
rotateEventLog logIdV numberOfEventsV |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think we need this as the application will call sourceEvents
at least once (see rules above).
So indeed we should hook into the sourceEvents
conduit to do our event counting (and maybe do rotation while sourcing!?)
mkRotatedEventStore rotationConfig checkpointer logId eventStore | ||
|
||
mkChechpointer :: IsChainState tx => ChainStateType tx -> UTCTime -> Checkpointer (StateEvent tx) | ||
mkChechpointer initialChainState time events = |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think the mkCheckpointer
should be a function sitting in a Hydra.Node
module as it is using the specific StateEvent
type (Hydra.Events.Rotation
is otherwise abstracted over e
)
ChainStateType tx -> | ||
EventStore (StateEvent tx) m -> | ||
m (EventStore (StateEvent tx) m) | ||
prepareRotatedEventStore rotationConfig initialChainState eventStore = do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I don't think that we will need this function. If even, it is glue code for Hydra.Node.Run
that ties mkCheckpointer
to mkRotatedEventStore
.
LogId -> | ||
EventStore e m -> | ||
m (EventStore e m) | ||
mkRotatedEventStore config checkpointer logId eventStore = do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could: call this newRotatedEventStore
I often like to call monadic functions that create stuff newXXX
or createXXX
while mkXXX
is something non-monadic.
mkRotatedEventStore config checkpointer logId eventStore = do | |
newRotatedEventStore config checkpointer logId eventStore = do |
>>= primeWith inputsToOpenHead | ||
>>= runToCompletion | ||
rotatedHistory <- getEvents (fst rotatingEventStore) | ||
length rotatedHistory `shouldBe` 2 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As this is testing hydrate
I think it should move to NodeSpec
describe "Rotation algorithm" $ do | ||
prop "rotates on startup" $ | ||
\(Positive x, Positive y) -> | ||
(y > x) ==> do |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Instead of generating both and discarding many values, it is often a better idea to just generate the delta. For example: generate x
and delta
, then y = x + delta
which always yields a good test (and less discarded runs).
(y > x) ==> do | ||
eventStore@(eventSource, eventSink) <- createMockSourceSink | ||
let totalEvents = toInteger y | ||
let events = TrivialEvent <$> [1 .. fromInteger totalEvents] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could: just generate two [TrivialEvent]
directly. Then you also don't need to generate positive numbers (there are no negative number of list items).
Something like prop ... $ \toRotate additionalEvents -> let totalEvents = toRotate <> additionalEvents; let rotationConfig = RotateAfter (length toRotate) ...
now <- getCurrentTime | ||
let checkpointer = mkChechpointer initialChainState now | ||
-- FIXME! | ||
let logId = 0 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yeah.. so where should we start the log file numbering? :)
I see two options that does not require knowing about how many past log files do exist:
- EventId of first or last event (which is strictly increasing)
- Timestamp of when the log was created
Both don't require an initial LogId
when creating the EventStore
, but can be found when we decide to do a rotation.
* this is temp because is faster for testing than using time interval. * will change to time interval in the near future.
* also move it as part of the algorithm spec
b27f621
to
2f1b2c6
Compare