S3 download leak connection

[sbxz]

Describe the bug

We are experiencing connection leak issues with the AWS SDK when downloading a file to S3.
If an error occurs just before subscribing to the donwload stream (or just if we never subscribe to the stream), the connection is never released.

Expected Behavior

I assume the connection should be released after a certain period of inactivity? However, none of the timeouts I have configured seem to have any effect.

Current Behavior

Once all the connections in the pool are occupied, we get connection acquisition errors. Even after an hour.

Caused by: java.lang.Throwable: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.
Consider taking any of the following actions to mitigate the issue: increase max connections, increase acquire timeout, or slowing the request rate.
Increasing the max connections can increase client throughput (unless the network interface is already fully utilized), but can eventually start to hit operation system limitations on the number of file descriptors used by the process. If you already are fully utilizing your network interface or cannot further increase your connection count, increasing the acquire timeout gives extra time for requests to acquire a connection before timing out. If the connections doesn't free up, the subsequent requests will still timeout.
If the above mechanisms are not able to fix the issue, try smoothing out your requests so that large traffic bursts cannot overload the client, being more efficient with the number of times you need to call AWS, or by increasing the number of hosts sending requests.
	at software.amazon.awssdk.http.nio.netty.internal.utils.NettyUtils.decorateException(NettyUtils.java:69) ~[netty-nio-client-2.25.24.jar:na]
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.handleFailure(NettyRequestExecutor.java:307) ~[netty-nio-client-2.25.24.jar:na]
	at software.amazon.awssdk.http.nio.netty.internal.NettyRequestExecutor.makeRequestListener(NettyRequestExecutor.java:188) ~[netty-nio-client-2.25.24.jar:na]
	at io.netty.util.concurrent.DefaultPromise.notifyListener0(DefaultPromise.java:590) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.DefaultPromise.notifyListenersNow(DefaultPromise.java:557) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.DefaultPromise.access$200(DefaultPromise.java:35) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.DefaultPromise$1.run(DefaultPromise.java:503) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569) ~[netty-transport-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:994) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at java.base/java.lang.Thread.run(Thread.java:1583) ~[na:na]
Caused by: java.util.concurrent.TimeoutException: Acquire operation took longer than 10000 milliseconds.
	at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.timeoutAcquire(HealthCheckedChannelPool.java:77) ~[netty-nio-client-2.25.24.jar:na]
	at software.amazon.awssdk.http.nio.netty.internal.HealthCheckedChannelPool.lambda$acquire$0(HealthCheckedChannelPool.java:67) ~[netty-nio-client-2.25.24.jar:na]
	at io.netty.util.concurrent.PromiseTask.runTask(PromiseTask.java:98) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	at io.netty.util.concurrent.ScheduledFutureTask.run(ScheduledFutureTask.java:153) ~[netty-common-4.1.111.Final.jar:4.1.111.Final]
	... 7 common frames omitted

Reproduction Steps

For example, here I have a simple test with one connection in the pool, and if I try to call /download twice, the second one will never be able to acquire a connection.

    @GetMapping("/download")
    public Mono<Void> download() {
        return Mono.fromFuture(this.s3AsyncClient.getObject(r ->
                    r.bucket(BUCKET)
                        .key(S3_KEY),
                AsyncResponseTransformer.toPublisher()))
            .then();
    }
    
    @Bean
    public S3AsyncClient s3AsyncClients(final S3Configuration s3Configuration,
        final SdkAsyncHttpClient sdkAsyncHttpClient)
        throws URISyntaxException {

        return S3AsyncClient.builder()
            .httpClient(sdkAsyncHttpClient)
            .serviceConfiguration(s3Configuration)
            .forcePathStyle(Boolean.TRUE)
            .endpointOverride(new URI(ENDPOINT))
            .region(Region.of(REGION))
            .credentialsProvider(() -> AwsBasicCredentials.create(ACCESS_KEY, PRIVATE_KEY))
            .build();
    }

    @Bean
    public SdkAsyncHttpClient sdkAsyncHttpClient() {
        return NettyNioAsyncHttpClient.builder()
            .maxConcurrency(1)
            .connectionTimeToLive(Duration.ofSeconds(2))
            .connectionTimeout(Duration.ofSeconds(2))
            .connectionAcquisitionTimeout(Duration.ofSeconds(10))
            .connectionMaxIdleTime(Duration.ofSeconds(2))
            .writeTimeout(Duration.ofSeconds(2))
            .tlsNegotiationTimeout(Duration.ofSeconds(2))
            .readTimeout(Duration.ofSeconds(2))
            .tcpKeepAlive(false)
            .useIdleConnectionReaper(true)
            .build();
    }

    @Bean
    public S3Configuration s3Configuration() {
        return S3Configuration.builder()
            .checksumValidationEnabled(Boolean.FALSE)
            .chunkedEncodingEnabled(Boolean.TRUE)
            .build();
    }

Possible Solution

No response

Additional Information/Context

No response

AWS Java SDK version used

2.26.12

JDK version used

21

Operating System and version

Windows 11

[kuzd4niil]

Hello @sbxz, I faced the same problem.
I think this problem is related to reactore-core-3541

[BastianBue]

We are currently facing the issue.

│ ClientException: Unable to execute HTTP request: Timeout waiting for connection from pool\n\tat s.a.a.c.e.SdkClientException$BuilderImpl.build(SdkClientException.java:111)\n\tat s.a.a.c.e.SdkClientException.create(SdkC │ │ lientException.java:47)\n\tat s.a.a.c.i.h.p.s.u.RetryableStageHelper2.setLastException(RetryableStageHelper2.java:226)\n\tat s.a.a.c.i.h.p.s.RetryableStage2.execute(RetryableStage2.java:65)\n\tat s.a.a.c.i.h.p.s.Retrya │ │ bleStage2.execute(RetryableStage2.java:36)\n\tat s.a.a.c.i.h.p.RequestPipelineBuilder$ComposingRequestPipelineStage.execute(RequestPipelineBuilder.java:206)\n\tat s.a.a.c.i.h.StreamManagingStage.execute(StreamManagingS │ │ tage.java:53)\n\tat s.a.a.c.i.h.StreamManagingStage.execute(StreamManagingStage.java:35)\n\t... 33 frames truncated (including 14 common frames)\nCaused by: o.a.h.c.ConnectionPoolTimeoutException: Timeout waiting for c │ │ onnection from pool\n\tat o.a.h.i.c.PoolingHttpClientConnectionManager.leaseConnection(PoolingHttpClientConnectionManager.java:316)\n\tat o.a.h.i.c.PoolingHttpClientConnectionManager$1.get(PoolingHttpClientConnectionMa │ │ nager.java:282)\n\tat s.a.a.h.a.i.c.ClientConnectionRequestFactory$DelegatingConnectionRequest.get(ClientConnectionRequestFactory.java:92)\n\tat s.a.a.h.a.i.c.ClientConnectionRequestFactory$InstrumentedConnectionReques │ │ t.get(ClientConnectionRequestFactory.java:69)\n\t... 5 frames excluded\n\tat s.a.a.h.a.i.i.ApacheSdkHttpClient.execute(ApacheSdkHttpClient.java:72)\n\tat s.a.a.h.a.ApacheHttpClient.execute(ApacheHttpClient.java:254)\n\ │ │ tat s.a.a.h.a.ApacheHttpClient.access$500(ApacheHttpClient.java:104)\n\tat s.a.a.h.a.ApacheHttpClient$1.call(ApacheHttpClient.java:231)\n\t... 59 frames truncated (including 37 common frames)\n"} │ │

[bhoradc]

Hello @sbxz,

Thanks for reporting the issue. Below is the minimal reproducible code sample I used with the latest SDK version 2.31.4 that throws the TimeoutException (File size = 1KB).

Main.java

package org.example;

import org.springframework.web.bind.annotation.GetMapping;
import reactor.core.publisher.Mono;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.core.async.AsyncResponseTransformer;
import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3Configuration;

import java.net.URISyntaxException;

public class Main {
    private static final String BUCKET = "**";
    private static final String S3_KEY = "**";
    private static final String REGION = "us-east-1";
    private static final String ACCESS_KEY = "**";
    private static final String PRIVATE_KEY = "**";

    private final S3AsyncClient s3AsyncClient;

    public Main() throws URISyntaxException {
        this.s3AsyncClient = s3AsyncClients(s3Configuration(), sdkAsyncHttpClient());
    }
    @GetMapping("/download")
    public Mono<Void> download() {
        return Mono.fromFuture(this.s3AsyncClient.getObject(r ->
                                r.bucket(BUCKET)
                                        .key(S3_KEY),
                        AsyncResponseTransformer.toPublisher()))//.toBytes()))
                .then();
    }
    public S3AsyncClient s3AsyncClients(final S3Configuration s3Configuration,
                                        final SdkAsyncHttpClient sdkAsyncHttpClient)
            throws URISyntaxException {

        return S3AsyncClient.builder()
                .httpClient(sdkAsyncHttpClient)
                .serviceConfiguration(s3Configuration)
                .forcePathStyle(Boolean.TRUE)
                .region(Region.of(REGION))
                .credentialsProvider(() -> AwsBasicCredentials.create(ACCESS_KEY, PRIVATE_KEY))
                .build();
    }
    public SdkAsyncHttpClient sdkAsyncHttpClient() {
        return NettyNioAsyncHttpClient.builder()
                .maxConcurrency(1)
                .build();
    }
    public S3Configuration s3Configuration() {
        return S3Configuration.builder()
                .checksumValidationEnabled(Boolean.FALSE)
                .chunkedEncodingEnabled(Boolean.TRUE)
                .build();
    }
    public static void main(String[] args) {
        try {
            Main main = new Main();
            main.download().block();
            main.download().block();
        } catch (URISyntaxException e) {
            e.printStackTrace();
        }
    }
}

Pom.xml

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>org.example</groupId>
    <artifactId>V2_S3DownLeak_5355</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <aws.sdk.version>2.31.4</aws.sdk.version>
    </properties>
    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>software.amazon.awssdk</groupId>
                <artifactId>bom</artifactId>
                <version>2.31.4</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
            <dependency>
                <groupId>com.fasterxml.jackson</groupId>
                <artifactId>jackson-bom</artifactId>
                <version>2.15.0</version>
                <type>pom</type>
            </dependency>
            <dependency>
                <groupId>org.apache.logging.log4j</groupId>
                <artifactId>log4j-bom</artifactId>
                <version>2.19.0</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
    <dependencies>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
        </dependency>
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>netty-nio-client</artifactId>
            <version>2.31.4</version>
        </dependency>
        <dependency>
            <groupId>io.projectreactor</groupId>
            <artifactId>reactor-core</artifactId>
            <version>3.5.8</version>
        </dependency>
        <dependency>
            <groupId>org.springframework</groupId>
            <artifactId>spring-web</artifactId>
            <version>6.0.11</version>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-core</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-api</artifactId>
        </dependency>
        <dependency>
            <groupId>org.apache.logging.log4j</groupId>
            <artifactId>log4j-slf4j2-impl</artifactId>
        </dependency>
    </dependencies>
</project>

If it's a connection availability issue rather, then increasing timeouts may not help especially if the concurrent connections in pool is limited to 1, which is never getting released.

I will review this issue with the team and shall keep you posted.

Regards,
Chaitanya

[D4RK463]

Is anybody still working on this issue?