Skip to content

Async BumpTransactionEventHandler #3752

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 3 commits into
base: main
Choose a base branch
from

Conversation

joostjager
Copy link
Contributor

@joostjager joostjager commented Apr 28, 2025

Converts BumpTransactionEventHandler to async.

Changes the CoinSelectionSource and WalletSource traits to be async and provides WalletSourceSyncWrapper as a helper for users that want to implement a sync wallet source.

TestWalletSource is kept sync, to prevent a cascade of async conversions in tests.

Fixes #3540

@ldk-reviews-bot
Copy link

ldk-reviews-bot commented Apr 28, 2025

👋 Thanks for assigning @TheBlueMatt as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

@joostjager joostjager force-pushed the async-wallet-bump branch 3 times, most recently from d9b1c75 to 873516b Compare May 1, 2025 11:08
Copy link

codecov bot commented May 1, 2025

Codecov Report

Attention: Patch coverage is 90.08621% with 23 lines in your changes missing coverage. Please review.

Project coverage is 90.78%. Comparing base (35748f6) to head (e7d43db).
Report is 16 commits behind head on main.

Files with missing lines Patch % Lines
lightning/src/events/bump_transaction.rs 89.92% 5 Missing and 9 partials ⚠️
lightning/src/events/bump_transaction_sync.rs 90.00% 8 Missing ⚠️
lightning/src/util/test_utils.rs 85.71% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main    #3752      +/-   ##
==========================================
+ Coverage   89.92%   90.78%   +0.86%     
==========================================
  Files         160      161       +1     
  Lines      129322   137021    +7699     
  Branches   129322   137021    +7699     
==========================================
+ Hits       116290   124393    +8103     
+ Misses      10345     9956     -389     
+ Partials     2687     2672      -15     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@joostjager joostjager force-pushed the async-wallet-bump branch 6 times, most recently from 4b4b898 to 3816c66 Compare May 5, 2025 11:37
@joostjager joostjager force-pushed the async-wallet-bump branch 13 times, most recently from 1a86521 to e1509f1 Compare May 14, 2025 09:55
@joostjager joostjager changed the title Async wallet bump Async BumpTransactionEventHandler May 14, 2025
@joostjager joostjager marked this pull request as ready for review May 14, 2025 11:06
@joostjager joostjager requested a review from tnull May 14, 2025 11:06
@joostjager joostjager added the weekly goal Someone wants to land this this week label May 14, 2025
Copy link
Contributor

@tnull tnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Did a first / quickish pass. Generally looks good so far.

@@ -48,12 +48,14 @@ backtrace = { version = "0.3", optional = true }

libm = { version = "0.2", default-features = false }
inventory = { version = "0.3", optional = true }
tokio = { version = "1.35", features = [ "macros", "rt" ], optional = true }
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please set default-features = false here and below.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added. Why is this? I looked at the tokio crate and it seems there are no default features?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's 'best practice' and will also make sure we'd not add arbitrary dependencies if tokio decided to add default features in some release.

*commitment_tx_fee_satoshis,
anchor_descriptor,
) {
if let Err(_) = self
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO these would be a bit more readable if we did the the whole call including .await on a separate line and then simply did if res.is_err() { .. }

Copy link
Contributor Author

@joostjager joostjager May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tried it, but I don't think it is an improvement with that temp var.

let res = self
	.handle_channel_close(
		*claim_id,
		*package_target_feerate_sat_per_1000_weight,
		commitment_tx,
		*commitment_tx_fee_satoshis,
		anchor_descriptor,
	)
	.await;
if res.is_err() {
	log_error!(
		self.logger,
		"Failed bumping commitment transaction fee for {}",
		commitment_tx.compute_txid()
	);
}

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also tried with map_err, but that doesn't work all that well with the parent fn without a return result.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also tried with map_err, but that doesn't work all that well with the parent fn without a return result.

How about:

				self.handle_channel_close(
					*claim_id,
					*package_target_feerate_sat_per_1000_weight,
					commitment_tx,
					*commitment_tx_fee_satoshis,
					anchor_descriptor,
				)
				.await
				.unwrap_or_else(|_| {
					log_error!(
						self.logger,
						"Failed bumping commitment transaction fee for {}",
						commitment_tx.compute_txid()
					);
				});

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah this is nicer. Wasn't aware of the unwrap_or_else on result. Updated.

#[allow(clippy::expect_used, clippy::diverging_sub_expression)]
{
return tokio::runtime::Builder::new_current_thread()
.enable_all()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FWIW, I don't think we need to enable_all here, could just actually enable features we need. Same goes for all the tests that use [tokio::test], although not sure if it makes a big difference in practice.

Copy link
Contributor Author

@joostjager joostjager May 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just expanded the macro. Can make customizations, but maybe it is better to keep to the original expansion? Also if it doesn't matter in practice...

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, no strong opinion. If we increase the number of tests using tokio, we might want to some benchmarks though, as the runtime of cargo test accumulates over time, so every second we can squeeze out of it would be appreciated.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Removed then. If we get more of these xtests, we might need to duplicate the macro or something like that.

let configs = [(false, false), (false, true), (true, false), (true, true)];
let mut last_err = None;
for (force_conflicting_utxo_spend, tolerate_high_network_feerates) in configs {
log_debug!(
self.logger,
"Attempting coin selection targeting {} sat/kW (force_conflicting_utxo_spend = {}, tolerate_high_network_feerates = {})",
target_feerate_sat_per_1000_weight,
force_conflicting_utxo_spend,
tolerate_high_network_feerates
);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Indent off here.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the indent off. The closing parenthesis?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actually the parenthesis and all arguments of log_debug too (they need to be indented by one tab).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't see the problem anymore in the diff now, but github still shows it on the old code fragment? In my IDE it all looks good with tabs, so I think it is solved.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nope, still there on HEAD:

Screenshot From 2025-05-16 10-47-39

Copy link
Contributor Author

@joostjager joostjager May 16, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, happened in the next commit. Fixed now.

@ldk-reviews-bot
Copy link

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

@ldk-reviews-bot
Copy link

🔔 4th Reminder

Hey @TheBlueMatt! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

Copy link
Contributor

@tnull tnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excuse the delay here, finally got to testing this out with LDK Node.

One request for a minor change, otherwise it seems to work as expected with minmal changes (see https://github.com/tnull/ldk-node/tree/2025-02-upgrade-to-ldk-0.2-with-async-bumper)

fn list_confirmed_utxos<'a>(&'a self) -> AsyncResult<'a, Vec<Utxo>>;
/// Returns a script to use for change above dust resulting from a successful coin selection
/// attempt.
fn get_change_script<'a>(&self) -> AsyncResult<'a, ScriptBuf>;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there a particular reason why we bound &'a self for list_confirmed_utxos but not for get_change_script / sign_psbt? I should be able to work around this, but this simple diff would make my life considerably easier:

diff --git a/lightning/src/events/bump_transaction.rs b/lightning/src/events/bump_transaction.rs
index 8a3d3cb9b..98bd3525a 100644
--- a/lightning/src/events/bump_transaction.rs
+++ b/lightning/src/events/bump_transaction.rs
@@ -366,14 +366,14 @@ pub trait WalletSource {
        fn list_confirmed_utxos<'a>(&'a self) -> AsyncResult<'a, Vec<Utxo>>;
        /// Returns a script to use for change above dust resulting from a successful coin selection
        /// attempt.
-       fn get_change_script<'a>(&self) -> AsyncResult<'a, ScriptBuf>;
+       fn get_change_script<'a>(&'a self) -> AsyncResult<'a, ScriptBuf>;
        /// Signs and provides the full [`TxIn::script_sig`] and [`TxIn::witness`] for all inputs within
        /// the transaction known to the wallet (i.e., any provided via
        /// [`WalletSource::list_confirmed_utxos`]).
        ///
        /// If your wallet does not support signing PSBTs you can call `psbt.extract_tx()` to get the
        /// unsigned transaction and then sign it with your wallet.
-       fn sign_psbt<'a>(&self, psbt: Psbt) -> AsyncResult<'a, Transaction>;
+       fn sign_psbt<'a>(&'a self, psbt: Psbt) -> AsyncResult<'a, Transaction>;
 }

 /// A synchronous version of the [`WalletSource`] trait. Implementations of this trait should be wrapped in
@@ -420,7 +420,7 @@ where

        /// Returns a script to use for change above dust resulting from a successful coin selection attempt. Wraps
        /// [`WalletSourceSync::get_change_script`].
-       fn get_change_script<'a>(&self) -> AsyncResult<'a, ScriptBuf> {
+       fn get_change_script<'a>(&'a self) -> AsyncResult<'a, ScriptBuf> {
                let script = self.0.get_change_script();
                Box::pin(async move { script })
        }
@@ -428,7 +428,7 @@ where
        /// Signs and provides the full [`TxIn::script_sig`] and [`TxIn::witness`] for all inputs within the transaction
        /// known to the wallet (i.e., any provided via [`WalletSource::list_confirmed_utxos`]). Wraps
        /// [`WalletSourceSync::sign_psbt`].
-       fn sign_psbt<'a>(&self, psbt: Psbt) -> AsyncResult<'a, Transaction> {
+       fn sign_psbt<'a>(&'a self, psbt: Psbt) -> AsyncResult<'a, Transaction> {
                let signed_psbt = self.0.sign_psbt(psbt);
                Box::pin(async move { signed_psbt })
        }

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I removed them because they weren't necessary. But seems better to stick with the same pattern everywhere. If we would have used async traits, they would also always expand to the same thing, I think.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I removed them because they weren't necessary. But seems better to stick with the same pattern everywhere. If we would have used async traits, they would also always expand to the same thing, I think.

Well, and just because they are not necessary in for the TestWalletSource, it doesn't mean they aren't needed/helpful elsewhere.

@ldk-reviews-bot
Copy link

🔔 5th Reminder

Hey @TheBlueMatt! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

@joostjager joostjager force-pushed the async-wallet-bump branch from 534ee90 to 4107b83 Compare June 4, 2025 09:41
@joostjager
Copy link
Contributor Author

joostjager commented Jun 4, 2025

Excuse the delay here, finally got to testing this out with LDK Node.

One request for a minor change, otherwise it seems to work as expected with minmal changes (see https://github.com/tnull/ldk-node/tree/2025-02-upgrade-to-ldk-0.2-with-async-bumper)

Looks good that code. I also found myself doing the _inner thing to wrap it easily into async without too much indent.

@joostjager joostjager requested a review from tnull June 4, 2025 09:48
tnull
tnull previously approved these changes Jun 4, 2025
Copy link
Contributor

@tnull tnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Copy link
Collaborator

@TheBlueMatt TheBlueMatt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two doc nits and one actual comment.

@joostjager joostjager force-pushed the async-wallet-bump branch 2 times, most recently from d8f4c67 to 462afa0 Compare June 6, 2025 08:29
@joostjager joostjager requested review from TheBlueMatt and tnull June 6, 2025 12:15
}
}

impl<W: Deref + MaybeSync + MaybeSend, L: Deref + MaybeSync + MaybeSend> CoinSelectionSource
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I believe we should instead implement CoinSelectionSourceSync so that we can use BumpTransactionEventHandlerSync, no?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh man, you are totally right. Now I can revert all the asyncification of the tests 😅

I will be back

It seems that the compiler doesn't recognize the drop and complains
that the mutex crosses an await (introduced in later commit), even
though it doesn't.
This preparatory commit converts do_coin_selection from a closure into
a standard function.
@joostjager joostjager force-pushed the async-wallet-bump branch from 462afa0 to c34c042 Compare June 6, 2025 15:10
@@ -1874,36 +1873,36 @@ impl Drop for TestScorer {

pub struct TestWalletSource {
secret_key: SecretKey,
utxos: RefCell<Vec<Utxo>>,
utxos: Mutex<Vec<Utxo>>,
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Still need this in the sync code it seems, unfortunately.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this is a show stopper for the whole sync wrapping concept. I hope not.

@joostjager joostjager removed the request for review from tnull June 6, 2025 15:33
@joostjager joostjager force-pushed the async-wallet-bump branch from c34c042 to d01009c Compare June 9, 2025 06:58
@joostjager
Copy link
Contributor Author

As already mentioned in #3752 (comment), a lot of the asyncification could be reverted now that the right wrappers are available.

I also isolated the wrappers in their own file.

Copy link
Contributor

@tnull tnull left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems CI is failing now.

@joostjager joostjager force-pushed the async-wallet-bump branch from d01009c to 1892700 Compare June 9, 2025 09:03
@joostjager joostjager force-pushed the async-wallet-bump branch from 1892700 to e7d43db Compare June 9, 2025 11:08
@joostjager
Copy link
Contributor Author

CI passes

@joostjager joostjager requested review from tnull and TheBlueMatt June 9, 2025 14:17
@joostjager
Copy link
Contributor Author

Reviewers, didn't do fix up commits as the commit structure changed substantially. Thanks for your patience with this PR. Should have realized that the sync wrappers could avoid all the test changes that have now been reverted.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
weekly goal Someone wants to land this this week
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Make BumpTransactionEventHandler async-optional
4 participants