Skip to content

Event log rotation #1997

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 18 commits into
base: master
Choose a base branch
from
Draft

Event log rotation #1997

wants to merge 18 commits into from

Conversation

ffakenz
Copy link
Contributor

@ffakenz ffakenz commented May 8, 2025


  • CHANGELOG updated or not needed
  • Documentation updated or not needed
  • Haddocks updated or not needed
  • No new TODOs introduced or explained herafter

@github-project-automation github-project-automation bot moved this to Triage 🏥 in ☕ Hydra Team Work May 8, 2025
@ffakenz ffakenz moved this from Triage 🏥 to In progress 🕐 in ☕ Hydra Team Work May 8, 2025
@ffakenz ffakenz linked an issue May 8, 2025 that may be closed by this pull request
@ffakenz ffakenz added the 💬 feature A feature on our roadmap label May 8, 2025
@ffakenz ffakenz self-assigned this May 8, 2025
@ffakenz ffakenz requested a review from a team May 8, 2025 10:46
@ffakenz ffakenz added this to the 0.22.0 milestone May 8, 2025
Copy link

github-actions bot commented May 8, 2025

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at 2025-05-09 23:28:18.829781842 UTC
Max. memory units 14000000
Max. CPU units 10000000000
Max. tx size (kB) 16384

Script summary

Name Hash Size (Bytes)
νInitial c8a101a5c8ac4816b0dceb59ce31fc2258e387de828f02961d2f2045 2652
νCommit 61458bc2f297fff3cc5df6ac7ab57cefd87763b0b7bd722146a1035c 685
νHead be6ebc744208c660bf0fdc1cfbb5157477cd305de5b1777e575cbb4c 14665
μHead 1f47a42d1d6edc32ccd834acb19d5db3b2a5232f0bd7eaa8908dc519* 5284
νDeposit ae01dade3a9c346d5c93ae3ce339412b90a0b8f83f94ec6baa24e30c 1102
  • The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

Init transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 5837 10.56 3.35 0.52
2 6038 12.37 3.91 0.54
3 6239 14.26 4.50 0.57
5 6645 18.55 5.85 0.63
10 7644 28.68 9.02 0.78
43 14282 99.11 30.98 1.80

Commit transaction costs

This uses ada-only outputs for better comparability.

UTxO Tx size % max Mem % max CPU Min fee ₳
1 558 2.44 1.16 0.20
2 742 3.38 1.73 0.22
3 919 4.36 2.33 0.24
5 1278 6.41 3.60 0.28
10 2179 12.13 7.25 0.40
54 10049 98.61 68.52 1.88

CollectCom transaction costs

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 57 525 24.46 7.13 0.42
2 114 636 34.20 9.84 0.53
3 170 747 42.50 12.21 0.61
4 227 858 53.42 15.24 0.73
5 282 969 63.64 18.07 0.84
6 337 1081 67.70 19.48 0.89
7 395 1196 72.17 20.91 0.94
8 450 1303 87.00 24.85 1.09
9 505 1414 95.07 27.57 1.18

Cost of Increment Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 1790 24.24 8.05 0.48
2 1928 25.40 9.11 0.51
3 2111 28.09 10.67 0.55
5 2396 31.13 12.97 0.60
10 3019 38.35 18.60 0.73
40 7548 94.19 56.84 1.65

Cost of Decrement Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 605 22.73 7.33 0.41
2 813 25.45 8.75 0.45
3 946 26.06 9.57 0.47
5 1196 29.15 11.76 0.52
10 2093 40.50 18.28 0.70
40 6528 96.16 53.80 1.60

Close transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 659 29.00 9.24 0.48
2 880 29.74 10.31 0.50
3 873 31.95 11.39 0.53
5 1244 34.15 13.60 0.58
10 2009 44.42 20.41 0.74
38 6148 99.09 56.93 1.63

Contest transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 682 33.70 10.50 0.53
2 811 35.73 11.78 0.56
3 953 37.78 13.07 0.59
5 1279 42.43 15.93 0.66
10 2059 54.25 23.16 0.85
29 4916 96.34 49.18 1.50

Abort transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties Tx size % max Mem % max CPU Min fee ₳
1 5712 27.04 9.05 0.69
2 5789 32.45 10.81 0.75
3 5972 44.32 14.84 0.88
4 6185 55.03 18.55 1.00
5 6174 56.49 18.88 1.01
6 6302 71.75 23.99 1.18
7 6565 80.21 26.89 1.28
8 6669 91.40 30.63 1.40

FanOut transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.

Parties UTxO UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
10 0 0 5834 19.19 6.41 0.61
10 5 284 6003 29.80 10.58 0.73
10 10 569 6173 39.90 14.59 0.85
10 20 1137 6511 59.32 22.30 1.08
10 30 1706 6853 80.09 30.47 1.32
10 39 2216 7155 98.99 37.90 1.54

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2025-05-09 23:31:27.50569734 UTC

Baseline Scenario

Number of nodes 1
Number of txs 300
Avg. Confirmation Time (ms) 4.279488786
P99 6.554354709999954ms
P95 4.7713907ms
P50 4.0870145ms
Number of Invalid txs 0

Memory data

Time Used Free
2025-05-09 23:30:11.224451968 UTC 874M 6494M
2025-05-09 23:30:16.224330964 UTC 992M 6272M
2025-05-09 23:30:21.224332527 UTC 998M 6265M
2025-05-09 23:30:26.224252212 UTC 998M 6265M
2025-05-09 23:30:31.224265639 UTC 1016M 6246M
2025-05-09 23:30:36.224316378 UTC 1016M 6246M

Three local nodes

Number of nodes 3
Number of txs 900
Avg. Confirmation Time (ms) 28.151331124
P99 43.0017671ms
P95 37.216210749999995ms
P50 26.982078ms
Number of Invalid txs 0

Memory data

Time Used Free
2025-05-09 23:30:49.699914439 UTC 932M 6340M
2025-05-09 23:30:54.700003089 UTC 1135M 6135M
2025-05-09 23:30:59.70035955 UTC 1200M 6012M
2025-05-09 23:31:04.700094328 UTC 1213M 5949M
2025-05-09 23:31:09.699955797 UTC 1214M 5948M
2025-05-09 23:31:14.699940436 UTC 1218M 5943M
2025-05-09 23:31:19.699962555 UTC 1219M 5942M
2025-05-09 23:31:24.699958694 UTC 1220M 5940M

Copy link

github-actions bot commented May 8, 2025

Transaction cost differences

Script summary

Name Size (Bytes)
νInitial -
νCommit -
νHead -
μHead -
νDeposit -

Init transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 - - - -
2 - - - -
3 - - - -
5 - - - -
10 - - - -
43 - - - -

Commit transaction costs

UTxO Tx size % max Mem % max CPU Min fee ₳
1 - - - -
2 - - - -
3 - - - -
5 - - - -
10 - - - -
54 - - - -

CollectCom transaction costs

Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
1 - - - - -
2 - - - - -
3 - - - - -
4 - - - - -
5 - - - - -
6 - - - - -
7 - - - - -
8 - - - - -
9 - - - - -

Cost of Increment Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 - $${\color{green}-0.38}$$ $${\color{green}-0.09}$$ -
2 - $${\color{green}-0.37}$$ $${\color{green}-0.08}$$ -
3 - - - -
5 - +0.38 +0.09 -
10 - - - -
43 - - - -

Cost of Decrement Transaction

Parties Tx size % max Mem % max CPU Min fee ₳
1 - - - -
2 - - - -
3 - - - -
5 - - - -
10 - - - -
40 - - - -

Close transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 - - - -
2 - - - -
3 - - - -
5 - - - -
10 - - - -
35 - - - -

Contest transaction costs

Parties Tx size % max Mem % max CPU Min fee ₳
1 - - - -
2 - - - -
3 - - - -
5 - - - -
10 - - - -
29 - - - -

FanOut transaction costs

UTxO, Parties UTxO (bytes) Tx size % max Mem % max CPU Min fee ₳
(0, 10) - - - - -
(1, 10) - - - - -
(5, 10) - - - - -
(10, 10) - - - - -
(20, 10) - - - - -
(38, 10) - - - - -

Comment on lines +33 to +38
currentEvents <- getEvents eventSource
let currentNumberOfEvents = toInteger $ length currentEvents
numberOfEventsV <- newTVarIO currentNumberOfEvents
-- XXX: check rotation on startup
when (currentNumberOfEvents >= toInteger rotateAfterX) $ do
rotateEventLog logIdV numberOfEventsV
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think we need this as the application will call sourceEvents at least once (see rules above).

So indeed we should hook into the sourceEvents conduit to do our event counting (and maybe do rotation while sourcing!?)

mkRotatedEventStore rotationConfig checkpointer logId eventStore

mkChechpointer :: IsChainState tx => ChainStateType tx -> UTCTime -> Checkpointer (StateEvent tx)
mkChechpointer initialChainState time events =
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the mkCheckpointer should be a function sitting in a Hydra.Node module as it is using the specific StateEvent type (Hydra.Events.Rotation is otherwise abstracted over e)

ChainStateType tx ->
EventStore (StateEvent tx) m ->
m (EventStore (StateEvent tx) m)
prepareRotatedEventStore rotationConfig initialChainState eventStore = do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't think that we will need this function. If even, it is glue code for Hydra.Node.Run that ties mkCheckpointer to mkRotatedEventStore.

LogId ->
EventStore e m ->
m (EventStore e m)
mkRotatedEventStore config checkpointer logId eventStore = do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could: call this newRotatedEventStore

I often like to call monadic functions that create stuff newXXX or createXXX while mkXXX is something non-monadic.

Suggested change
mkRotatedEventStore config checkpointer logId eventStore = do
newRotatedEventStore config checkpointer logId eventStore = do

>>= primeWith inputsToOpenHead
>>= runToCompletion
rotatedHistory <- getEvents (fst rotatingEventStore)
length rotatedHistory `shouldBe` 2
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As this is testing hydrate I think it should move to NodeSpec

describe "Rotation algorithm" $ do
prop "rotates on startup" $
\(Positive x, Positive y) ->
(y > x) ==> do
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of generating both and discarding many values, it is often a better idea to just generate the delta. For example: generate x and delta, then y = x + delta which always yields a good test (and less discarded runs).

(y > x) ==> do
eventStore@(eventSource, eventSink) <- createMockSourceSink
let totalEvents = toInteger y
let events = TrivialEvent <$> [1 .. fromInteger totalEvents]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could: just generate two [TrivialEvent] directly. Then you also don't need to generate positive numbers (there are no negative number of list items).

Something like prop ... $ \toRotate additionalEvents -> let totalEvents = toRotate <> additionalEvents; let rotationConfig = RotateAfter (length toRotate) ...

now <- getCurrentTime
let checkpointer = mkChechpointer initialChainState now
-- FIXME!
let logId = 0
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah.. so where should we start the log file numbering? :)

I see two options that does not require knowing about how many past log files do exist:

  • EventId of first or last event (which is strictly increasing)
  • Timestamp of when the log was created

Both don't require an initial LogId when creating the EventStore, but can be found when we decide to do a rotation.

@ffakenz ffakenz force-pushed the event-log-rotation branch from b27f621 to 2f1b2c6 Compare May 8, 2025 21:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
💬 feature A feature on our roadmap
Projects
Status: In progress 🕐
Development

Successfully merging this pull request may close these issues.

Event Log Rotation
2 participants