swarm/: Support generic connection management

[mxinden]

Description

Today a user can:

1. Set a limit for incoming/outgoing/pending connections, global and per peer.
2. Close a connection via NetworkBehaviour.
3. Keep a connection alive via ConnectionHandler::keep_alive.

A user is restricted in (1) as they can not do anything beyond setting upper bound limits. E.g. they can not decide whether to accept or deny an incoming connection based on the IP, the PeerId or on their current CPU or memory utilization.

(Taken from #2118 (comment).)

I see 3 ways how we can enable users to do advanced connection management:

1. Make the pool::ConnectionCounter generic and allow users to provide their own.
   • Yet another moving piece.
   • When not Boxed requires an additional generic parameter.
   • Can only do connection management. Not as powerful as the NetworkBehaviour trait and thus limited in its decision making.
2. Extend NetworkBehaviour with methods called to review a pending or established connection.
   • Yet another set of methods on NetworkBehaviour.
   • Has knowledge of the full state, thus allows powerful decision making.
   • Users are already familiar with it.
3. Emit a SwarmEvent::ReviewConnection requiring the user to manually accept each pending/established connection via e.g. Swarm::accept().
   • Complicates the getting-started experience. User has to explicitly call a method to make progress. Not intuitive.

Motivation

Allows downstream users to do advanced connection management and can simplify existing implementations working around this today.

• https://github.com/sigp/lighthouse/tree/stable/beacon_node/lighthouse_network/src/peer_manager
• https://github.com/paritytech/substrate/blob/master/client/peerset/src/lib.rs
• https://github.com/n0-computer/iroh/tree/main/iroh-p2p

Are you planning to do it yourself in a pull request?

Yes

[mxinden]

I am currently working on proposal (2), i.e. extending NetworkBehaviour. Hope to publish a first proof-of-concept this week.

[thomaseizinger]

How is (2) going to work in terms of composing behaviours? One NetworkBehaviour declining a connection means it will be denied? i.e. Need full agreement across all NetworkBehaviours?

[mxinden]

How is (2) going to work in terms of composing behaviours? One NetworkBehaviour declining a connection means it will be denied? i.e. Need full agreement across all NetworkBehaviours?

Correct.

[mxinden]

Draft implementation of (2) is happening on #2828. Still work in progress, but might help understand the idea behind (2). Happy to expand on any of this in written form. No need to dig through the code in case you don't want to.

[thomaseizinger]

Going to start on this once #3011 is merged to avoid the churn.

[mxinden]

I think we should design this with a concrete use-case in mind. @divagant-martian would you volunteer overseeing
this from the lighthouse user perspective?

@thomaseizinger can you keep @divagant-martian in the loop?

[divagant-martian]

@mxinden for sure! I'm also relatively familiar with substrate's peer management and can probably get in touch with the iroh guys to check what needs we have in common and where do those differ

[thomaseizinger]

NetworkBehaviour already has quite a few callbacks, although we are getting rid of some of them in #3011. Thus, I am a bit hesitant to continue in the direction of #2828.

So here is an alternative idea:

Make IntoConnectionHandler fallible

NetworkBehaviour::new_handler is already invoked for every incoming connection. If a NetworkBehaviour would like to impose a policy on which connections should be established, it can pass the required data to a prototype implementation of IntoConnectionHandler and make the decision as soon as into_handler is called with the PeerId etc.

There are a few drawbacks with the current design but I think those are all fixable:

1. If somehow possible, I'd like to get rid of IntoConnetionHandler and delay calling new_handler until we have established the connection. This would allows us to directly make new_handler fallible which would make this feature much easier to use. Additionally, it means we won't run into race conditions with simple limit connection managers as new_handler is called with &mut self and thus, will be called sequentially for each connection.
2. NetworkBehaviourAction::Dial currently also emits an entire handler. To make sure connection limits are enforced in a single place, I'd replace this mechanism with an "OpenInfo" style where Dial can carry some "OpenInfo" struct that is passed back in new_handler.

I see the following benefits of this design:

• Good use of the type system: We need a ConnectionHandler for each connection. If we fail to construct one, we can never actually establish the connection.
• Synchronous decision making: Async decision making would require some kind of timeout so we would potentially hold every connection for some amount of time before we can consider it to be established.
• Small API surface: Very little change to the existing API of NetworkBehaviour.

cc @divagant-martian @dignifiedquire @rkuhn @mxinden