Spike: sqlite persistence + event log rotation #1968

ffakenz · 2025-04-26T12:51:14Z

Follow up on: #1746

CHANGELOG updated or not needed
Documentation updated or not needed
Haddocks updated or not needed
No new TODOs introduced or explained herafter

> based on 743fc1e

> from a hacking made on #1746 (comment)

* to ensure ordering of records over time

github-actions · 2025-04-26T13:01:51Z

Transaction cost differences

Script summary

Name	Size (Bytes)
νInitial	-
νCommit	-
νHead	-
μHead	-
νDeposit	-

`Init` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
40	-	-	-	-

`Commit` transaction costs

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
54	-	-	-	-

`CollectCom` transaction costs

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-	-
2	-	-	-	-	-
3	-	-	-	-	-
4	-	-	-	-	-
5	-	-	-	-	-
6	-	-	-	-	-
7	-	-	-	-	-
8	-	-	-	-	-

Cost of Increment Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	$${\color{green}-0.39}$$	$${\color{green}-0.09}$$	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	$${\color{green}-0.39}$$	$${\color{green}-0.09}$$	$${\color{green}-0.01}$$
37	-	$${\color{green}-0.38}$$	$${\color{green}-0.09}$$	-

Cost of Decrement Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
40	-	-	-	-

`Close` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
34	-	-	-	-

`Contest` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	-	-	-	-
2	-	-	-	-
3	-	-	-	-
5	-	-	-	-
10	-	-	-	-
27	-	-	-	-

`FanOut` transaction costs

UTxO, Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
(0, 10)	-	-	-	-	-
(1, 10)	-	-	-	-	-
(5, 10)	-	-	-	-	-
(10, 10)	-	-	-	-	-
(20, 10)	-	-	-	-	-
(37, 10)	-	-	-	-	-

github-actions · 2025-04-26T13:04:43Z

Transaction costs

Sizes and execution budgets for Hydra protocol transactions. Note that unlisted parameters are currently using arbitrary values and results are not fully deterministic and comparable to previous runs.

Metadata
Generated at	2025-04-27 19:13:38.061062381 UTC
Max. memory units	14000000
Max. CPU units	10000000000
Max. tx size (kB)	16384

Script summary

Name	Hash	Size (Bytes)
νInitial	c8a101a5c8ac4816b0dceb59ce31fc2258e387de828f02961d2f2045	2652
νCommit	61458bc2f297fff3cc5df6ac7ab57cefd87763b0b7bd722146a1035c	685
νHead	0e35115a2c7c13c68ecd8d74e4987c04d4539e337643be20bb3274bd	14756
μHead	57166715eadb8d3135964325c016eea546c21e1c0aae974ca67df9a5*	5541
νDeposit	ae01dade3a9c346d5c93ae3ce339412b90a0b8f83f94ec6baa24e30c	1102

The minting policy hash is only usable for comparison. As the script is parameterized, the actual script is unique per head.

`Init` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	6094	10.80	3.35	0.53
2	6292	13.26	4.11	0.56
3	6499	15.59	4.83	0.60
5	6898	19.98	6.17	0.66
10	7904	31.18	9.61	0.82
40	13935	98.36	30.21	1.78

`Commit` transaction costs

This uses ada-only outputs for better comparability.

UTxO	Tx size	% max Mem	% max CPU	Min fee ₳
1	561	2.44	1.16	0.20
2	741	3.38	1.73	0.22
3	920	4.36	2.33	0.24
5	1280	6.41	3.60	0.28
10	2173	12.13	7.25	0.40
54	10059	98.61	68.52	1.88

`CollectCom` transaction costs

Parties	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
1	57	525	26.47	7.60	0.44
2	113	636	35.90	10.22	0.54
3	171	747	44.51	12.70	0.63
4	227	858	50.69	14.55	0.70
5	284	969	65.60	18.53	0.86
6	338	1085	75.46	21.33	0.96
7	392	1192	85.08	24.03	1.06
8	451	1303	84.45	24.23	1.06

Cost of Increment Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	1794	25.11	8.24	0.49
2	1890	25.63	9.00	0.51
3	2091	28.77	10.68	0.55
5	2412	33.05	13.47	0.62
10	3043	40.74	19.14	0.75
37	7316	96.88	55.47	1.65

Cost of Decrement Transaction

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	597	23.99	7.61	0.43
2	825	26.74	9.04	0.47
3	949	28.17	10.10	0.49
5	1280	32.92	12.75	0.56
10	1912	41.17	18.34	0.70
41	6464	98.47	54.79	1.63

`Close` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	643	30.95	9.68	0.50
2	791	32.89	10.94	0.53
3	924	34.83	12.18	0.56
5	1354	40.48	15.56	0.65
10	2150	48.28	21.48	0.79
34	5621	96.47	52.99	1.56

`Contest` transaction costs

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	680	35.99	11.01	0.55
2	818	38.48	12.46	0.59
3	975	40.32	13.65	0.62
5	1359	46.73	17.19	0.71
10	1980	57.27	23.61	0.87
27	4720	99.49	48.53	1.51

`Abort` transaction costs

There is some variation due to the random mixture of initial and already committed outputs.

Parties	Tx size	% max Mem	% max CPU	Min fee ₳
1	5981	28.32	9.34	0.71
2	6094	37.44	12.33	0.81
3	6223	44.63	14.70	0.89
4	6339	53.64	17.68	0.99
5	6580	67.90	22.47	1.15
6	6594	75.71	25.05	1.23
7	6632	81.37	26.76	1.29
8	6890	93.27	30.82	1.43

`FanOut` transaction costs

Involves spending head output and burning head tokens. Uses ada-only UTXO for better comparability.

Parties	UTxO	UTxO (bytes)	Tx size	% max Mem	% max CPU	Min fee ₳
10	0	0	6092	19.66	6.46	0.62
10	1	57	6125	20.87	6.98	0.64
10	5	285	6261	30.31	10.59	0.75
10	10	569	6430	41.61	14.95	0.88
10	20	1140	6772	62.00	22.90	1.11
10	30	1709	7112	83.57	31.25	1.36
10	37	2108	7350	99.40	37.34	1.55

End-to-end benchmark results

This page is intended to collect the latest end-to-end benchmark results produced by Hydra's continuous integration (CI) system from the latest master code.

Please note that these results are approximate as they are currently produced from limited cloud VMs and not controlled hardware. Rather than focusing on the absolute results, the emphasis should be on relative results, such as how the timings for a scenario evolve as the code changes.

Generated at 2025-04-27 19:16:48.94034884 UTC

Baseline Scenario

Number of nodes	1
Number of txs	300
Avg. Confirmation Time (ms)	4.617904380
P99	10.87255287ms
P95	5.598792650000001ms
P50	4.359876ms
Number of Invalid txs	0

Memory data

Time	Used	Free
2025-04-27 19:15:32.64811941 UTC	885M	6580M
2025-04-27 19:15:37.648173785 UTC	1012M	6348M
2025-04-27 19:15:42.647978734 UTC	1014M	6346M
2025-04-27 19:15:47.648003908 UTC	1014M	6346M
2025-04-27 19:15:52.647921726 UTC	1019M	6340M
2025-04-27 19:15:57.648052039 UTC	1019M	6339M

Three local nodes

Number of nodes	3
Number of txs	900
Avg. Confirmation Time (ms)	29.884015646
P99	48.144610369999995ms
P95	41.46351115ms
P50	28.35397ms
Number of Invalid txs	0

Memory data

Time	Used	Free
2025-04-27 19:16:11.601515467 UTC	934M	6438M
2025-04-27 19:16:16.601742814 UTC	1144M	6225M
2025-04-27 19:16:21.602269803 UTC	1231M	6068M
2025-04-27 19:16:26.601715111 UTC	1227M	6011M
2025-04-27 19:16:31.601738686 UTC	1228M	6010M
2025-04-27 19:16:36.601682325 UTC	1233M	6004M
2025-04-27 19:16:41.601686503 UTC	1235M	6002M
2025-04-27 19:16:46.601510814 UTC	1235M	6002M

* also added pragma busy_timeout: > to wait up to 5 seconds before returning a 'database is locked' error. both in attempt to solve this error: > SQLite3 returned ErrorBusy while attempting to perform step: database is locked which indicates concurrent write access to db.

* also enhanced rotation log spec

ch1bo

What are you looking to get a review on? As this is a spike .. what is it that you found out and need a review on?

If it is about sqlite: What's the immediate benefit of using sqlite over a plain file? Is it faster or smaller?

If it is about checkpointing: I don't see where the stored events are pruned / result in smaller persistence or faster load time? Also see my comment on #1581 (comment).

ch1bo · 2025-04-27T15:36:06Z

hydra-node/src/Hydra/Events.hs

@@ -34,6 +34,7 @@ newtype EventSource e m = EventSource
 getEvents :: (HasEventId e, MonadUnliftIO m) => EventSource e m -> m [e]
 getEvents EventSource{sourceEvents} = runResourceT $ sourceToList sourceEvents

+-- TODO! putEventBatch


No, why?

We have at most 2 (or 3) events to persist at any given time.

not required, just a minor suggestion for improvement.

ch1bo · 2025-04-27T15:38:22Z

hydra-node/src/Hydra/SqlLitePersistence.hs

+withPersistenceIncremental fp checkpointer action = do
+  eventLog@PersistenceIncremental{closeDb} <- createRotatedEventLog fp checkpointerHandle
+  void $ action eventLog
+  closeDb


This may never be executed in presence of exceptions -> use bracket

ch1bo · 2025-04-27T15:39:26Z

hydra-node/src/Hydra/SqlLitePersistence.hs

+  (PersistenceIncremental a IO -> IO b) ->
+  IO ()
+withPersistenceIncremental fp checkpointer action = do
+  eventLog@PersistenceIncremental{closeDb} <- createRotatedEventLog fp checkpointerHandle


Is checkpointing specific to the sqlite persistence? If not than this should be specified orthogonal to it.

yes, it is a bit specific.

lets do it orthogonal in a separate PR!

ch1bo · 2025-04-27T16:19:31Z

hydra-node/src/Hydra/Events/SqlLiteBased.hs

+      atomically getLastSeenEventId >>= \case
+        Nothing -> store evt
+        Just lastSeenEventId
+          | getEventId evt > lastSeenEventId -> store evt


No need to keep track of a last seen event id here as the sqlite database should make it very easy to not write the same key that is already present (or overwrite it).

v0d1ch · 2025-04-28T08:08:50Z

hydra-node/src/Hydra/SqlLitePersistence.hs

+createWriteConnectionV fp = do
+  createDirectoryIfMissing True $ takeDirectory fp
+  conn <- open fp
+  execute_ conn "PRAGMA journal_mode = WAL;"


I remember when working on this we had problems with performance, hopefully these settings make the sqlitedb faster to read/write.

the increase in performance comes from using a singleton connection rather than opening a new one for every db action

WAL mode is still good for failure resiliency

noonio · 2025-04-28T10:51:09Z

As far as I can see, comparing to some other PRs, it seems that this is now almost exactly the same speed as it was before? i.e. no noticable impact? If so, amazing.

Can you run some performance tests on this PR and the main branch so we can be certain of the impact? (And post some graphs!)

Very cool if so :)

ffakenz · 2025-04-29T11:28:25Z

Closing because :

sqlite is not planned
the logic must be orthogonal to persistence (see: Spike: sqlite persistence + event log rotation #1968 (comment))

v0d1ch and others added 11 commits April 25, 2025 21:29

Add sqlite3 to our shell

fe05705

draft base sqlite-persistence module

6c74c01

> based on 743fc1e

apply some enhancements

1bfb44e

> from a hacking made on #1746 (comment)

introduce appendMany for incremental persistence

a8be06f

add auto-increment on keys

9454dfc

* to ensure ordering of records over time

draft base sqlite-events module

8bf4547

enable sqlite persistence

5608191

experimental: long living connection

720472a

introduce checkpoint event

63e760e

draft rotated event log

db3da8c

group io actions

47fef4b

github-project-automation bot moved this to Triage 🏥 in ☕ Hydra Team Work Apr 26, 2025

github-project-automation bot added this to ☕ Hydra Team Work Apr 26, 2025

ffakenz self-assigned this Apr 26, 2025

ffakenz added 2 commits April 26, 2025 14:55

minor renaming

5b20784

update CHANGELOG

21dd19a

ffakenz force-pushed the spike-sqlite-persistence branch from 980f156 to 21dd19a Compare April 26, 2025 12:55

ffakenz linked an issue Apr 26, 2025 that may be closed by this pull request

Event Log Rotation #1581

Open

ffakenz requested a review from a team April 26, 2025 12:57

ffakenz moved this from Triage 🏥 to In progress 🕐 in ☕ Hydra Team Work Apr 26, 2025

ffakenz force-pushed the spike-sqlite-persistence branch from 023bccd to 21dd19a Compare April 26, 2025 13:30

ffakenz added 7 commits April 26, 2025 15:38

decrease rotation rate from 10k to 3k

20bfc0c

fix exception instance

27794ee

introduce checkpointer handle

d655ba9

fix file name index rotation

b0bbb75

test event log rotation

78576bc

make sure to close dbs after every spec

e14ff68

ffakenz added 2 commits April 27, 2025 00:43

minor fmt

7b42aec

fix appendMany check rotation

fc76382

* also enhanced rotation log spec

ch1bo reviewed Apr 27, 2025

View reviewed changes

ffakenz added 3 commits April 27, 2025 20:27

add WAL checkpointing of previous event log after rotation

f310e55

split read-write connection

7333f77

simplify connection for WAL checkpointing

6b113fd

v0d1ch reviewed Apr 28, 2025

View reviewed changes

ffakenz closed this Apr 29, 2025

github-project-automation bot moved this from In progress 🕐 to Done ✔ in ☕ Hydra Team Work Apr 29, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Spike: sqlite persistence + event log rotation #1968

Spike: sqlite persistence + event log rotation #1968

Uh oh!

ffakenz commented Apr 26, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Apr 26, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Apr 26, 2025 •

edited

Loading

Uh oh!

ch1bo left a comment

Uh oh!

ch1bo Apr 27, 2025

Uh oh!

ffakenz Apr 29, 2025

Uh oh!

ch1bo Apr 27, 2025

Uh oh!

ch1bo Apr 27, 2025

Uh oh!

ffakenz Apr 29, 2025 •

edited

Loading

Uh oh!

ch1bo Apr 27, 2025

Uh oh!

v0d1ch Apr 28, 2025

Uh oh!

ffakenz Apr 28, 2025

Uh oh!

ch1bo Apr 28, 2025

Uh oh!

noonio commented Apr 28, 2025

Uh oh!

ffakenz commented Apr 29, 2025 •

edited

Loading

Uh oh!

Uh oh!

Spike: sqlite persistence + event log rotation #1968

Spike: sqlite persistence + event log rotation #1968

Uh oh!

Conversation

ffakenz commented Apr 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions bot commented Apr 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Transaction cost differences

Script summary

Init transaction costs

Commit transaction costs

CollectCom transaction costs

Cost of Increment Transaction

Cost of Decrement Transaction

Close transaction costs

Contest transaction costs

FanOut transaction costs

Uh oh!

github-actions bot commented Apr 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Transaction costs

Script summary

Init transaction costs

Commit transaction costs

CollectCom transaction costs

Cost of Increment Transaction

Cost of Decrement Transaction

Close transaction costs

Contest transaction costs

Abort transaction costs

FanOut transaction costs

End-to-end benchmark results

Baseline Scenario

Memory data

Three local nodes

Memory data

Uh oh!

ch1bo left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ffakenz Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

noonio commented Apr 28, 2025

Uh oh!

ffakenz commented Apr 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

ffakenz commented Apr 26, 2025 •

edited

Loading

github-actions bot commented Apr 26, 2025 •

edited

Loading

`Init` transaction costs

`Commit` transaction costs

`CollectCom` transaction costs

`Close` transaction costs

`Contest` transaction costs

`FanOut` transaction costs

github-actions bot commented Apr 26, 2025 •

edited

Loading

`Init` transaction costs

`Commit` transaction costs

`CollectCom` transaction costs

`Close` transaction costs

`Contest` transaction costs

`Abort` transaction costs

`FanOut` transaction costs

ffakenz Apr 29, 2025 •

edited

Loading

ffakenz commented Apr 29, 2025 •

edited

Loading