feat(background-processor): add BackgroundProcessorBuilder for optional components #3688

Anyitechs · 2025-03-28T08:01:55Z

This PR introduces two main improvements to the BackgroundProcessor:

Adds a BackgroundProcessorBuilder for a more ergonomic way to construct a BackgroundProcessor, and supports optional parameters through builder methods
Introduces BackgroundProcessorConfig to standardize configuration across sync (BackgroundProcessor) and async variants (process_events_async), enabling same configuration to be used for both variants. The Builder returns this config object, which can be used to construct a processor via BackgroundProcessor::from_config

Fixes #3612

…al components

ldk-reviews-bot · 2025-03-28T08:01:58Z

👋 I see @wpaulino was un-assigned.
If you'd like another reviewer assignemnt, please click here.

tnull

Already looks pretty good, one question though.

Also, do you have any idea how we could solve the same issue for the async variant of the background processor, i.e., for the optional arguments of process_events_async?

tnull · 2025-03-28T08:14:17Z

lightning-background-processor/src/lib.rs

@@ -1046,6 +1049,171 @@ impl BackgroundProcessor {
 			None => Ok(()),
 		}
 	}
+
+	/// Creates a new [`BackgroundProcessorBuilder`] to construct a   [`BackgroundProcessor`] with optional components.
+	pub fn builder<'a, PS, EH, M, CM, PGS, RGS, G, UL, L, PM>(


I'm a bit confused on the purpose of this method: in which scenario would we already have a working BackgroundProcessor to call builder on, just to then use the BackgroundProcessorBuilder to create yet another BackgroundProcessor?

This is confusing indeed. I was thinking the BackgroundProcessorBuilder::new() can be an internal method and accessible via builder, but it doesn't make sense. I will remove the builder method here and allow users use BackgroundProcessorBuilder::new() directly.

@tnull this doesn't take self, though?

Either way, normally the builder pattern has the builder() associated function, but works something like:
BackgroundProcessor::builder() constructs a BackgroundProcessorBuilder with "default" fields (which might be hard in this case), and you would not have any parameters for builder().

So yeah, probably removing this is best.

@tnull this doesn't take self, though?

Ah true.

Either way, normally the builder pattern has the builder() associated function, but works something like: BackgroundProcessor::builder() constructs a BackgroundProcessorBuilder with "default" fields (which might be hard in this case), and you would not have any parameters for builder().

Yeah, unclear what defaults would be.

ldk-reviews-bot · 2025-03-28T08:16:53Z

👋 The first review has been submitted!

Do you think this PR is ready for a second reviewer? If so, click here to assign a second reviewer.

Anyitechs · 2025-03-28T09:00:13Z

Already looks pretty good, one question though.

Also, do you have any idea how we could solve the same issue for the async variant of the background processor, i.e., for the optional arguments of process_events_async?

Yea, I think we can extend the new BackgroundProcessorBuilder or provide a new builder for the async variant. But extending the new builder might be the better approach as we can avoid duplicating the builder logic since the sync and async variants share the same optional components.

tnull · 2025-03-28T09:07:32Z

Yea, I think we can extend the new BackgroundProcessorBuilder or provide a new builder for the async variant. But extending the new builder might be the better approach as we can avoid duplicating the builder logic since the sync and async variants share the same optional components.

The question is how this would work? process_events_async is just a method that users need to call regularly, so to 'build' it with different configuration you'd need to provide multiple versions of it, which can get messy real quick, essentially exactly the thing we tried to avoid with the builder pattern. It makes me wonder if we should introduce a BackgroundProcessorParams/BackgroundProcessorConfig object that process_events_async takes as its only argument. And if we do so, maybe we should/could reuse the same object for the non-async BackgroundProcessor, also? Meaning, this object could either replace the builder or we could have the builder return the new config object.

What do you think?

Anyitechs · 2025-03-28T09:53:33Z

Yea, I think we can extend the new BackgroundProcessorBuilder or provide a new builder for the async variant. But extending the new builder might be the better approach as we can avoid duplicating the builder logic since the sync and async variants share the same optional components.

The question is how this would work? process_events_async is just a method that users need to call regularly, so to 'build' it with different configuration you'd need to provide multiple versions of it, which can get messy real quick, essentially exactly the thing we tried to avoid with the builder pattern. It makes me wonder if we should introduce a BackgroundProcessorParams/BackgroundProcessorConfig object that process_events_async takes as its only argument. And if we do so, maybe we should/could reuse the same object for the non-async BackgroundProcessor, also? Meaning, this object could either replace the builder or we could have the builder return the new config object.

What do you think?

Thank you for the suggestion. I agree that introducing a BackgroundProcessorConfig object would be a better option. I'm thinking we can reuse the same object for both variants and have the builder return the new config object.

What do you think about that?

tnull · 2025-03-28T10:02:54Z

What do you think about that?

Sounds good to me, although we might need to account for the fact that the two variants have slightly different type requirements (i.e., trait bounds). But I think we should be able to accommodate that via feature-gates on std and/or futures.

Anyitechs · 2025-03-28T10:17:20Z

Sounds good to me, although we might need to account for the fact that the two variants have slightly different type requirements (i.e., trait bounds). But I think we should be able to accommodate that via feature-gates on std and/or futures.

Yes, that can be handled conditionally using feature-gates.

I guess I can go ahead with the implementation, right?

This allows for consistent configuration of sync and async variant of the BackgroundProcessor

Anyitechs · 2025-04-09T17:11:46Z

This PR is ready for another round of review.

tnull · 2025-04-10T09:23:21Z

lightning-background-processor/src/lib.rs

-	onion_messenger: Option<OM>, gossip_sync: GossipSync<PGS, RGS, G, UL, L>, peer_manager: PM,
-	logger: L, scorer: Option<S>, sleeper: Sleeper, mobile_interruptable_platform: bool,
-	fetch_time: FetchTime,
+	config: BackgroundProcessorConfig<


nit: I think we might want to add a #[rustfmt::skip] here/above to avoid having rustfmt making this that vertical.

tnull · 2025-04-10T09:26:18Z

lightning-background-processor/src/lib.rs

@@ -1048,6 +1160,227 @@ impl BackgroundProcessor {
 	}
 }

+/// Configuration for both synchronous [`BackgroundProcessor`] and asynchronous [`process_events_async`] event processing.


I'm not sure these doc links will work under all circumstances as process_events_async will only be exposed when the futures feature is set. You might need to alter the docs based on the set feature via cfg_attr, see the changes to the background processor docs in #3509 for reference how to do this.

Ah true! Thank you.

tnull · 2025-04-10T09:29:09Z

lightning-background-processor/src/lib.rs

+	_phantom: PhantomData<(&'a (), CF, T, F, P)>,
+}
+
+/// A builder for constructing a [`BackgroundProcessor`] with optional components.


This should say BackgroundProcessorConfig now, no? And do we want to rename the builder to BackgroundProcessorConfigBuilder?

I agree. The builder builds a BackgroundProcessorConfig object, so renaming that to BackgroundProcessorConfigBuilder might be more suitable.

tnull · 2025-04-10T09:29:32Z

lightning-background-processor/src/lib.rs

+	PM::Target: APeerManager + Send + Sync,
+{
+	/// Creates a new builder instance.
+	pub(crate) fn new(


Seems CI is failing due to this being pub(crate)?

tnull · 2025-04-10T09:30:58Z

Did another round of review, will take another look once CI passes.

tnull · 2025-04-10T09:32:18Z

lightning-background-processor/src/lib.rs

+	/// .build();
+	/// let bg_processor = BackgroundProcessor::from_config(config);
+	/// ```
+	pub fn from_config<


I wonder if this should even replace start (i.e., whether we should have start just take a BackgroundProcessorConfig)?

I thought about this too but not sure if we want to make that change on start immediately.

Anyitechs · 2025-04-10T10:10:10Z

Did another round of review, will take another look once CI passes.

Thank you very much for the review. I'll address all the feedbacks and update the PR.

ldk-reviews-bot · 2025-04-12T00:00:30Z

🔔 1st Reminder

Hey @dunxen! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-04-14T00:00:50Z

🔔 2nd Reminder

Hey @dunxen! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

The builder s and returns a object, so renaming it to seems more suitable.

Anyitechs · 2025-04-14T14:57:02Z

CI still failing, fixing it.

feat(background-processor): add BackgroundProcessorBuilder for option…

d7880ef

…al components

tnull self-requested a review March 28, 2025 08:11

tnull reviewed Mar 28, 2025

View reviewed changes

Anyitechs added 2 commits April 8, 2025 19:38

remove the builder() method in BackgroundProcessor

1e63c70

introduce BackgroundProcessorConfig

5b17f80

This allows for consistent configuration of sync and async variant of the BackgroundProcessor

Anyitechs marked this pull request as ready for review April 9, 2025 17:11

ldk-reviews-bot requested a review from wpaulino April 9, 2025 17:12

Anyitechs requested review from tnull and dunxen April 9, 2025 17:12

tnull removed the request for review from wpaulino April 9, 2025 17:17

tnull reviewed Apr 10, 2025

View reviewed changes

Anyitechs added 3 commits April 14, 2025 01:13

make BackgroundProcessorConfig docs conditional on futures feature

1d64687

rename builder to BackgroundProcessorConfigBuilder

00ac430

The builder s and returns a object, so renaming it to seems more suitable.

fix(CI): fix CI build errors

b967d87

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat(background-processor): add BackgroundProcessorBuilder for optional components #3688

feat(background-processor): add BackgroundProcessorBuilder for optional components #3688

Anyitechs commented Mar 28, 2025 •

edited

Loading

ldk-reviews-bot commented Mar 28, 2025 •

edited

Loading

tnull left a comment

tnull Mar 28, 2025

Anyitechs Mar 28, 2025

dunxen Mar 29, 2025

tnull Mar 31, 2025

ldk-reviews-bot commented Mar 28, 2025

Anyitechs commented Mar 28, 2025

tnull commented Mar 28, 2025

Anyitechs commented Mar 28, 2025

tnull commented Mar 28, 2025

Anyitechs commented Mar 28, 2025

Anyitechs commented Apr 9, 2025

tnull Apr 10, 2025

tnull Apr 10, 2025

Anyitechs Apr 10, 2025

tnull Apr 10, 2025

Anyitechs Apr 10, 2025

tnull Apr 10, 2025

tnull commented Apr 10, 2025

tnull Apr 10, 2025

Anyitechs Apr 10, 2025

Anyitechs commented Apr 10, 2025

ldk-reviews-bot commented Apr 12, 2025

ldk-reviews-bot commented Apr 14, 2025

Anyitechs commented Apr 14, 2025

feat(background-processor): add BackgroundProcessorBuilder for optional components #3688

Are you sure you want to change the base?

feat(background-processor): add BackgroundProcessorBuilder for optional components #3688

Conversation

Anyitechs commented Mar 28, 2025 • edited Loading

ldk-reviews-bot commented Mar 28, 2025 • edited Loading

tnull left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ldk-reviews-bot commented Mar 28, 2025

Anyitechs commented Mar 28, 2025

tnull commented Mar 28, 2025

Anyitechs commented Mar 28, 2025

tnull commented Mar 28, 2025

Anyitechs commented Mar 28, 2025

Anyitechs commented Apr 9, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tnull commented Apr 10, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Anyitechs commented Apr 10, 2025

ldk-reviews-bot commented Apr 12, 2025

ldk-reviews-bot commented Apr 14, 2025

Anyitechs commented Apr 14, 2025

Anyitechs commented Mar 28, 2025 •

edited

Loading

ldk-reviews-bot commented Mar 28, 2025 •

edited

Loading