SimplePool v2

[miklish]

SimpleConnectionPool v2

SimpleConnectionPool is a simple, natively mockable, threadsafe, resilient, high-performance connection pool intended to replace com.networknt.client.http.Http2ClientConnectionPool.

Use of this connection pool fully resolves issue 1656 (#1656) by replacing the use of Http2ClientConnection in com.networknt.consul.client.ConsulClientImpl.lookupHealthService() with SimpleURIConnectionPool.

SimplePool v2 also includes changes from the following issues:

• Optimize synchronization in SimpleConnectionPool and fix NPE
• Remove unused code in SimpleUndertowConnectionMaker
• Ensure leaked connections are closed in SimpleURIConnectionPool
• Ensure XnioSSL and XnioWorker resources are created only once
• Use Generic Name for SimplePool Connection Threads (1.6.x)

Breaking Changes

• class SimpleConnectionHolder renamed to SimpleConnectionState
• class SimpleClientConnection renamed to SimpleUndertowConnection
• class SimpleClientConnectionMaker renamed to SimpleUndertowConnectionMaker
• method reuseConnection() removed from the SimpleConnectionMaker interface

See issue #1905 for more details on changes.

Focus on Stability and Testability

The primary goal of SimpleConnectionPool is operational stability and reliability. Optimizations were considered only after stability and resilience were confirmed.

Easy testability was vital to confirm the correct behaviour of the connection pool under a wide array of common and corner-case conditions. To this end, the connection pool was developed with mockability built-in.

Avoidance of Consul Connection Bug

The following Consul connection-leak bug was reported in August 2020, but has yet to be resolved.

When a Consul client cancels a blocking HTTP-query, the TCP connection is not closed correctly by the server. The TCP connection stays in the FIN_WAIT-2 state until it's tcp_fin_timeout expires. The FIN_WAIT-2 means that the client is waiting for an ACK from the server.

When a lot of blocking queries are cancelled and retried at the same time, the servers http_max_conns_per_client can be hit the further client queries will fail.

Bug Ticket URL: hashicorp/consul#8524

SimpleConnectionPool avoids this bug entirely by never closing connections that are in use. While this can cause connection expiry times to not be precisely honoured, it also ensures that Consul blocking queries are never canceled before they complete, thereby avoiding the conditions that manifest this bug.

Multi-Layer Architecture

The connection pool was developed using a multilayer architecure.

Level 1: SimpleConnectionPool routes requests to URI connection pools

Threads that need a connection to a URI must 'borrow' a connection from the pool and then restore that connection to the pool when they are done with it.

A SimpleConnectionPool routes these URI-connection requests to a SimpleURIConnectionPool for that URI. If a SimpleURIConnectionPool does not exist for that URI, the SimpleConnectionPool will create one.

The SimpleConnectionPool needs only minimal thread synchronization since the SimpleURIConnectionPools are already fully thread safe. A benefit of this, is increased opportunity for concurrency. For example: N threads can request connections to N distinct URIs concurrently.

Level 2: SimpleURIConnectionPools manage connections to a single URI

A SimpleURIConnectionPool manages connections to a single URI where the connections it manages have the ability to be used by 1 or more threads concurrently. A SimpleURIConnectionPool:

• Enforces a configurable maximum number of connections
• Closes connections after a configurable 'expiry' time, but ensures that it never closes connections that are in use
• Safely manages connection resources (if client behaves as expected)
• Safely closes leaked connections that may be created by connection-creation callback threads after a connection-creation timeout has occurred in the parent thread (more on this below)
• Is threadsafe

Also, note that SimpleURIConnectionPools can be used used independently of SimpleConnectionPool. This means that, any code that only needs to connect to a single URI can use a SimpleURIConnectionPool directly--it does not need to make requests to borrow connections from a SimpleConnectionPool (which is only needed for code that needs to connect to multiple distinct URIs).

The SimpleURIConnectionPool needs very little information about connections to manage the pool. It only needs to know if the connection is:

• borrowed
• borrowable
• expired
• closed

Note: More than one of these properties can simultaneously apply to a connection (e.g.: an HTTP/2 connection may be both borrowed and borrowable).

Level 3: SimpleConnectionStates manage connection states

SimpleConnectionState objects both create and manage the state of a single network connection.

SimpleConnectionStates have a 1-1 relationship with their connection. In other words:

• every connection in the pool is created by -- and contained in -- a single SimpleConnectionState, and
• every SimpleConnectionState contains the single connection it created.

When a SimpleURIConnectionPool needs a new connection, it will create a new SimpleConnectionState object. The connection pool will then manage that connection exclusively by communicating through the connection's enclosing SimpleConnectionState, rather than managing the connection directly.

The SimpleConnectionState provides the following information about a connection to the pool:

• borrowed
  • A connection is considered 'borrowed' if 1 or more threads are currently using the connection at the moment.
• borrowable
  • A connection is considered 'borrowable' if the connection can be used by a new thread
    • HTTP/1.1: HTTP/1.1 connections are only borrowable when 0 connection tokens are currently borrowed
    • HTTP/2: HTTP/2 connections are always be borrowable
• expired / valid
  • A connection is 'expired' once its age exceeds the pool's connection expiry time setting
  • A connection is 'valid' if it is not yet expired
• closed
  • A connection is closed if the the pool or the OS closes it

The state diagram of a SimpleConnectionState is shown below.

SimpleConnectionState - State diagram for a connection

              |
             \/
    [ NOT_BORROWED_VALID ]  --(borrow)-->  [ BORROWED_VALID ]
              |             <-(restore)--          |
              |                                    |
           (expire)                             (expire)
              |                                    |
             \/                                   \/
    [ NOT_BORROWED_EXPIRED ] <-(restore)-- [ BORROWED_EXPIRED ]
             |
          (close) (*)
             |
            \/
        [ CLOSED ]
 

(*) NOTE

A connection can be closed explicitly by the connection pool, or it can be unexpectedly closed at any time by the OS. If it is closed unexpectedly by the OS, then the state can jump directly to CLOSED regardless of what state it is currently in.

Level 4: SimpleConnections and SimpleConnectionMakers

SimpleConnection interfaces wrap physical 'raw' network connections. For example, the SimpleUndertowConnection class implements SimpleConnection and wraps Undertow's ClientConnection.

SimpleConnectionMaker's are factories that create SimpleConnection objects, and are used by SimpleConnectionState objects to create the connection they manage. Different implementations of SimpleConnectionMaker can be used to create different types of raw connections. For example: A SimpleUndertowConnectionMaker uses the Undertow networking library to create connections, a SimpleApacheConnectionMaker could use the Apache libraries, and a SimpleMockConnectionMaker could create mock connections used for testing.

Note: SimpleConnectionStates only deal with SimpleConnection objects -- they never deal directly with 'raw' connections (such as Undertow's ClientConnection objects).

Creating a new network (raw) connection

1. Create a SimpleConnectionMaker (e.g.: a SimpleUndertowConnectionMaker)

[Connection Pool]   [Connection State]   [Connection Maker]     [Raw Connection]     [Simple Connection]
       [|]                   |                    |                      |                     |
       [|]   << creates >>   |                    |                      |                     |
       [|]-------------------|------------------>[|]                     |                     |
       [|]                   |                   [|]                     |                     |

2. Pool creates new connections by instantiating new SimpleConnectState objects (passing them a SimpleConnectionMaker in their constructor)
3. SimpleConnectionState uses the SimpleConnectionMaker to create SimpleConnection objects that wrap raw network connections

[Connection Pool]   [Connection State]   [Connection Maker]     [Raw Connection]     [Simple Connection]
       [|]                   |                   [|]                     |                     |
       [|]   << creates >>   |                   [|]                     |                     |
       [|]----------------->[|]                  [|]                     |                     |
       [|]                  [|]    << uses >>    [|]                     |                     |
       [|]                  [|]----------------->[|]                     |                     |
       [|]                  [|]                  [|]  << creates con >>  |                     |
       [|]                  [|]                  [|]------------------->[|]                    |
       [|]                  [|]                  [|]                    [|]                    |       
       [|]                  [|]                  [|] << wraps con in >> [|]                    |
       [|]                  [|]                  [|]--------------------[|]------------------>[|]

Finding and closing leaked connections

Typically, new connections are created in a callback thread spawned by the thread that will use the connection.

Example of connection leak detection: SimpleUndertowConnectionMaker

To create a new connection, a SimpleUndertowConnectionMaker will spawn a new thread to create the connection. It will then wait a configurable amount of time (a timeout) for the connection-creation thread to finish creating the connection.

Below we detail 3 cases: Cases 1 and 2 detail situations in which no connection leak occurs, while case 3 details the situation where a leak does occur.

Case 1: Connection creation failed (OK)

If the connection-creation thread is unable to create the connection, then SimpleUndertowConnectionMaker will throw a 'connection-creation' exception and SimpleURIConnectionPool will not recieve a new connection.

Case 2: Connection successfully created before timeout (OK)

If the connection is successfully created before the timeout expires, then SimpleUndertowConnectionMaker will get the new connection from the connection-creation thread and return it to SimpleConnectionState (and, in turn, the new SimpleConnectionState holding the new connection will be returned to the pool).

Case 3: Connection successfully created after timeout (!)

If the connection is successfully created after the timeout expires, then -- as in case 1 -- SimpleUndertowConnectionMaker will throw a 'connection-creation' exception and SimpleURIConnectionPool will not receive a new connection.

However, in contrast to case 2: since a new connection was successfully created but not returned to the pool, it means that this new connection will exist but be unknown to the SimpleURIConnectionPool. Without knowledge of the connection, SimpleURIConnectionPool cannot not close it and so this untracked connection is considered a connection leak.

Handling connections created after timeouts

Case 3 results in connections that have been created by a SimpleConnectionMaker but are unknown to the pool and are called 'untracked' connections (conversely, connections known to the pool are called 'tracked' connections).

To ensure that all untracked connections are closed, users of a SimpleConnectionMaker must pass a threadsafe Set<SimpleConnection> to the SimpleConnectionMaker's makeConnection() method. Implementations of SimpleConnectionMaker must add all connections they create to this Set.

To ensure all untracked connections are closed, SimpleURIConnectionPool will periodically remove (but not close) all tracked connections from this set (e.g.: it will remove the connections it knows about). It will then close close any connections that remain in the set, since these remaining connections are untracked. This ensures that any untracked connections in the Set are safely closed.

Mockability

Since SimpleConnectState objects use a SimpleConnectionMaker to create their connections, and since the connections that SimpleConnectionMakers create implement the SimpleConnection interface, it means that SimpleURIConnectionPool, and SimpleConnectionState are able to manage the pool without having any knowledge of the raw connections they are managing.

This means mock connections can be created by simply implementing SimpleConnection and creating a SimpleConnectionMaker that instantiates these mock connections.

Mock Test Harness

SimpleConnectionPool comes with a test harness that can be used to easily test mock connections. For example, to test how the connection pool will handle a connection that randomly closes can be built as follows:

1. Develop the mock connection

public class MockRandomlyClosingConnection implements SimpleConnection {
    private volatile boolean closed = false;
    private boolean isHttp2 = true;
    private String MOCK_ADDRESS = "MOCK_HOST_IP:" + ThreadLocalRandom.current().nextInt((int) (Math.pow(2, 15) - 1.0), (int) (Math.pow(2, 16) - 1.0));

    /***
     * This mock connection simulates a multiplexable connection that has a 5% chance of closing
     *  every time isOpen() is called
     */
    public MockRandomlyClosingConnection(boolean isHttp2) { this.isHttp2 = isHttp2; }
    @Override public boolean isOpen() {
        if(ThreadLocalRandom.current().nextInt(20) == 0)
            closed = true;
        return !closed;
    }
    @Override public Object getRawConnection() {
        throw new RuntimeException("Mock connection has no raw connection");
    }
    @Override public boolean isMultiplexingSupported() { return isHttp2; }
    @Override public String getLocalAddress() { return MOCK_ADDRESS; }
    @Override public void safeClose() { closed = true; }
}

2. Test how the connection pool handles the connection

public class TestRandomlyClosingConnection {
    public static void main(String[] args) {
        new TestRunner()
            // set connection properties
            .setConnectionPoolSize(100)
            .setSimpleConnectionClass(MockRandomlyClosingConnection.class)
            .setCreateConnectionTimeout(5)
            .setConnectionExpireTime(5)
            .setHttp2(true)

            // configure borrower-thread properties
            .setNumBorrowerThreads(8)
            .setBorrowerThreadStartJitter(3)
            .setBorrowTimeLength(5)
            .setBorrowTimeLengthJitter(5)
            .setWaitTimeBeforeReborrow(2)
            .setWaitTimeBeforeReborrowJitter(2)

            // execute test
            .setTestLength(10*60)
            .executeTest();
    }
}

Plugability

SimpleConnectionPool requires very litte information about a connection in order to manage that connection. This lends itself to supporting many different networking APIs. If one can implement a SimpleConnection and SimpleConnectionMaker using a networking API, then connections created using that API can be managed by SimpleConnectionPool.

ClientConnections created using undertow's libraries can be wrapped this way. See the simplepool.undertow package to see how support for undertow was implemented.

How to safely use SimpleConnectionPool and SimpleURIConnectionPool

The following code snippet demonstrates the recommended way to structure code that borrows a connection.

Note: The code below is the same whether pool is a SimpleConnectionPool or a SimpleURIConnectionPool.

SimpleConnectionState.ConnectionToken borrowToken = null;
try {
    borrowToken = pool.borrow(createConnectionTimeout, isHttp2);
    ClientConnection connection = (ClientConnection) borrowToken.getRawConnection();

    // Use connection...
    
} finally {
    // restore token
    pool.restore(borrowToken);
}    

...and this demonstrates the recommended way to borrow connections in a loop...

while(true) {
    SimpleConnectionState.ConnectionToken borrowToken = null;
    try {
        borrowToken = pool.borrow(createConnectionTimeout, isHttp2);
        ClientConnection connection = (ClientConnection) borrowToken.getRawConnection();

        // Use connection...
        
    } finally {
        // restore token
        pool.restore(borrowToken);
    }    
}

[miklish]

Hi @stevehu , we hope you can review and merge this on Tuesday. Thanks!

[miklish]

Hi @stevehu ,

If possible, please merge this at your earliest convenience, or please let us know if you have any concerns about this PR so we can discuss and address them.

thanks!