Implement `KVStore` for `TestStore`, drop `process_events_async_with_kv_store_sync` #4069

tnull · 2025-09-12T12:15:51Z

We implement the async KVStore trait for TestStore. This is mostly useful in LDK Node tests, where we use TestStore and we now require all stores to implement both variants.

Moreover, we drop the Frankenstein-esque process_events_async_with_kv_store_sync which won't be necessary anymore for LDK Node.

.. to be easier reusable by different trait implementations.

ldk-reviews-bot · 2025-09-12T12:15:54Z

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

TheBlueMatt · 2025-09-12T12:31:55Z

lightning-background-processor/src/lib.rs

@@ -1187,80 +1187,6 @@ fn check_and_reset_sleeper<
 	}
 }

-/// Async events processor that is based on [`process_events_async`] but allows for [`KVStoreSync`] to be used for
-/// synchronous background persistence.
-pub async fn process_events_async_with_kv_store_sync<


Hmm, not quite sure I understand why we want to drop this? ldk-node might not use it, but I imagine some others might? Its the equivalent of our previous async BP loop and keeping it makes upgrades easier for those who might not want to switch to partial-async-kvstore immediately?

IMO it's a very odd middleground API that was introduced when the KVStoreSyncWrapper wasn't public. I fear users might find it confusing, and just using KVStoreSyncWrapper when needed seems way more consistent (as they'd already need to do that for some of the other types they'd hand into that method anyways).

No comment. It's so little code that I think it's fine either way.

IMO we should prefer users not use KVStoreSyncWrapper directly. The docs even explicitly say "It is not necessary to use this type directly." (and I feel like we should #[doc(hidden)] it?).

Hmm, but at least in LDK Node using KVStoreSyncWrapper was unavoidable. I can drop the drop commit if you insist, but IMO it's a pretty awkward confusing API.

TheBlueMatt · 2025-09-12T12:33:01Z

lightning/src/util/test_utils.rs

+	}
+}
+
+impl KVStore for TestStore {


I wonder if we shouldn't actually make these async? eg write the returned future to require two polls or something? Maybe its not worth it but seems like we should consider it. It feels a bit weird to have the test implementation do something that no real implementation should do.

Alright, now added a fixup to do just that.

Now force-pushed a further simplification, also allowing to drop the 'splitting' commit.

Oh I like this double poll thing for testing. A bit too fake otherwise.

codecov · 2025-09-12T12:38:15Z

Codecov Report

❌ Patch coverage is 47.50000% with 42 lines in your changes missing coverage. Please review.
✅ Project coverage is 88.37%. Comparing base (867f084) to head (cd28a16).
⚠️ Report is 14 commits behind head on main.

Files with missing lines	Patch %	Lines
lightning/src/util/test_utils.rs	36.36%	42 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #4069      +/-   ##
==========================================
- Coverage   88.39%   88.37%   -0.02%     
==========================================
  Files         177      177              
  Lines      131314   131518     +204     
  Branches   131314   131518     +204     
==========================================
+ Hits       116069   116224     +155     
- Misses      12596    12638      +42     
- Partials     2649     2656       +7

Flag	Coverage Δ
fuzzing	`21.97% <0.00%> (-0.04%)`	⬇️
tests	`88.20% <47.50%> (-0.02%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

joostjager · 2025-09-12T13:06:04Z

lightning/src/util/test_utils.rs

+		let secondary_namespace = secondary_namespace.to_string();
+		let key = key.to_string();
+		let inner = Arc::clone(&self.inner);
+		Box::pin(async move {


Before returning this, I think we need to record the order?

Or spawn_blocking and await?

Or just call it because it isn't actually blocking?

I don't think so, as it calls through to the inner Mutex? Ah, actually, now that we made it actually async as of #4069 (comment), we might need to?

Or just call it because it isn't actually blocking?

You mean actually async? It is now, so we might need to do the whole write-tracking thing now..

Nevermind, I think now that we retrieve the result sync again (while acquiring the inner Mutex), we should be good ordering-wise?

Yes I think it's good now

lightning/src/util/test_utils.rs

We implement the async `KVStore` trait for `TestStore`. This is mostly useful in LDK Node tests, where we use `TestStore` and we now require all stores to implement both variants.

joostjager

I understand this is going to be used in ldk-node, but I don't think any of the async code is hit in ldk itself. It's basically dead code within that scope. Test coverage of test code is maybe a bit over the top?

joostjager · 2025-09-12T14:30:15Z

lightning-background-processor/src/lib.rs

@@ -1187,80 +1187,6 @@ fn check_and_reset_sleeper<
 	}
 }

-/// Async events processor that is based on [`process_events_async`] but allows for [`KVStoreSync`] to be used for
-/// synchronous background persistence.
-pub async fn process_events_async_with_kv_store_sync<


No comment. It's so little code that I think it's fine either way.

joostjager · 2025-09-12T14:30:36Z

lightning/src/util/test_utils.rs

+		let secondary_namespace = secondary_namespace.to_string();
+		let key = key.to_string();
+		let inner = Arc::clone(&self.inner);
+		Box::pin(async move {


Yes I think it's good now

joostjager · 2025-09-12T14:31:02Z

lightning/src/util/test_utils.rs

+	}
+}
+
+impl KVStore for TestStore {


Oh I like this double poll thing for testing. A bit too fake otherwise.

TheBlueMatt

There's a ton of room to make TestStore a better test, dunno if it has to happen here but if you have a minute it would be nice to make it a really good test.

TheBlueMatt · 2025-09-12T14:44:09Z

lightning/src/util/test_utils.rs

+		let first_poll = &mut inner_lock.0;
+		if *first_poll {
+			*first_poll = false;
+			core::task::Poll::Pending


OH hmm this is more complicated than I thought. Technically this is a malformed future - after we switch to Ready we're supposed to call wake on the context. In our use it doesn't matter but...ugh. I do kinda wonder if we won't eventually want to be able to test having control over when async writes complete, it might well be useful even in testing of #4063. If its too much here then we can just skip it.

OH hmm this is more complicated than I thought. Technically this is a malformed future - after we switch to Ready we're supposed to call wake on the context.

Ah, good point. Now added a fixup that wakes after dropping the lock.

I do kinda wonder if we won't eventually want to be able to test having control over when async writes complete, it might well be useful even in testing of #4063. If its too much here then we can just skip it.

Yeah, tbh. this PR is mostly to make lightningdevkit/ldk-node#633 compile, and not even there we'd actually currently use the async TestStore, the implementation is just needed to fulfill the trait bounds currently. So while I agree it would be nice to upgrade TestStore and use it to write better async-KVStore tests, it's a bit out-of-scope for this PR right now. Happy to pick it up some time in a follow-up though!

TheBlueMatt · 2025-09-12T14:44:37Z

lightning/src/util/test_utils.rs

+	fn read(
+		&self, primary_namespace: &str, secondary_namespace: &str, key: &str,
+	) -> Pin<Box<dyn Future<Output = Result<Vec<u8>, io::Error>> + 'static + Send>> {
+		let res = self.read_internal(&primary_namespace, &secondary_namespace, &key);


Hmmmmm, would be nice to be able to race reads and writes rather than always immediately completing.....

Right, I agree it would be nice to eventually extend this for improved test coverage. But see above: given it's not actually used anywhere it's a bit out-of-scope here.

We previously introduced a variant of the async background processor that still could take a `KVStoreSync` implementation. This was a crutch for LDK Node which uses the async BP but still relied on `KVStoreSync` everywhere. We now update LDK Node to require also as `KVStore` implementation, allowing us to switch back to the proper `process_events_async`. Hence, we're dropping the intermediate crutch here again, as it's a weird middleground API.

Move TestStore logic to _internal methods

9cf4f69

.. to be easier reusable by different trait implementations.

tnull requested review from TheBlueMatt and joostjager September 12, 2025 12:16

tnull force-pushed the 2025-09-async-test-store branch from e636a74 to 6246c04 Compare September 12, 2025 12:18

tnull mentioned this pull request Sep 12, 2025

Add support for native async KVStore persist to ChainMonitor #4063

Draft

tnull force-pushed the 2025-09-async-test-store branch from 6246c04 to 7faae86 Compare September 12, 2025 12:22

TheBlueMatt reviewed Sep 12, 2025

View reviewed changes

joostjager reviewed Sep 12, 2025

View reviewed changes

tnull force-pushed the 2025-09-async-test-store branch from 7faae86 to 3ef76a5 Compare September 12, 2025 13:34

tnull requested a review from TheBlueMatt September 12, 2025 13:35

graphite-app bot reviewed Sep 12, 2025

View reviewed changes

lightning/src/util/test_utils.rs Outdated Show resolved Hide resolved

Implement KVStore for TestStore

21657d5

We implement the async `KVStore` trait for `TestStore`. This is mostly useful in LDK Node tests, where we use `TestStore` and we now require all stores to implement both variants.

tnull force-pushed the 2025-09-async-test-store branch from 3ef76a5 to 9df68fc Compare September 12, 2025 13:55

tnull requested a review from joostjager September 12, 2025 13:57

f Use single Mutex to avoid any races

e0e7a60

tnull force-pushed the 2025-09-async-test-store branch from 9df68fc to 3dc60fd Compare September 12, 2025 14:30

joostjager previously approved these changes Sep 12, 2025

View reviewed changes

TheBlueMatt reviewed Sep 12, 2025

View reviewed changes

tnull added 2 commits September 12, 2025 18:25

f Wake waker on Ready

c47ca84

tnull dismissed joostjager’s stale review via cd28a16 September 12, 2025 16:25

tnull force-pushed the 2025-09-async-test-store branch from 3dc60fd to cd28a16 Compare September 12, 2025 16:25

tnull requested a review from TheBlueMatt September 12, 2025 16:30

Implement KVStore for TestStore, drop process_events_async_with_kv_store_sync #4069

Are you sure you want to change the base?

Implement KVStore for TestStore, drop process_events_async_with_kv_store_sync #4069

Uh oh!

Conversation

tnull commented Sep 12, 2025

Uh oh!

ldk-reviews-bot commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

codecov bot commented Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

joostjager left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TheBlueMatt left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tnull Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tnull Sep 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Implement `KVStore` for `TestStore`, drop `process_events_async_with_kv_store_sync` #4069

Implement `KVStore` for `TestStore`, drop `process_events_async_with_kv_store_sync` #4069

ldk-reviews-bot commented Sep 12, 2025 •

edited

Loading

codecov bot commented Sep 12, 2025 •

edited

Loading

tnull Sep 12, 2025 •

edited

Loading

tnull Sep 12, 2025 •

edited

Loading