S3 download leak connection #5355
Comments
sbxz
added
bug
|
Hello @sbxz, I faced the same problem. |
|
We are currently facing the issue.
|
|
Hello @sbxz, Thanks for reporting the issue. Below is the minimal reproducible code sample I used with the latest SDK version Main.javapackage org.example;
import org.springframework.web.bind.annotation.GetMapping;
import reactor.core.publisher.Mono;
import software.amazon.awssdk.auth.credentials.AwsBasicCredentials;
import software.amazon.awssdk.core.async.AsyncResponseTransformer;
import software.amazon.awssdk.http.async.SdkAsyncHttpClient;
import software.amazon.awssdk.http.nio.netty.NettyNioAsyncHttpClient;
import software.amazon.awssdk.regions.Region;
import software.amazon.awssdk.services.s3.S3AsyncClient;
import software.amazon.awssdk.services.s3.S3Configuration;
import java.net.URISyntaxException;
public class Main {
private static final String BUCKET = "**";
private static final String S3_KEY = "**";
private static final String REGION = "us-east-1";
private static final String ACCESS_KEY = "**";
private static final String PRIVATE_KEY = "**";
private final S3AsyncClient s3AsyncClient;
public Main() throws URISyntaxException {
this.s3AsyncClient = s3AsyncClients(s3Configuration(), sdkAsyncHttpClient());
}
@GetMapping("/download")
public Mono<Void> download() {
return Mono.fromFuture(this.s3AsyncClient.getObject(r ->
r.bucket(BUCKET)
.key(S3_KEY),
AsyncResponseTransformer.toPublisher()))//.toBytes()))
.then();
}
public S3AsyncClient s3AsyncClients(final S3Configuration s3Configuration,
final SdkAsyncHttpClient sdkAsyncHttpClient)
throws URISyntaxException {
return S3AsyncClient.builder()
.httpClient(sdkAsyncHttpClient)
.serviceConfiguration(s3Configuration)
.forcePathStyle(Boolean.TRUE)
.region(Region.of(REGION))
.credentialsProvider(() -> AwsBasicCredentials.create(ACCESS_KEY, PRIVATE_KEY))
.build();
}
public SdkAsyncHttpClient sdkAsyncHttpClient() {
return NettyNioAsyncHttpClient.builder()
.maxConcurrency(1)
.build();
}
public S3Configuration s3Configuration() {
return S3Configuration.builder()
.checksumValidationEnabled(Boolean.FALSE)
.chunkedEncodingEnabled(Boolean.TRUE)
.build();
}
public static void main(String[] args) {
try {
Main main = new Main();
main.download().block();
main.download().block();
} catch (URISyntaxException e) {
e.printStackTrace();
}
}
}Pom.xml<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>V2_S3DownLeak_5355</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<aws.sdk.version>2.31.4</aws.sdk.version>
</properties>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>bom</artifactId>
<version>2.31.4</version>
<type>pom</type>
<scope>import</scope>
</dependency>
<dependency>
<groupId>com.fasterxml.jackson</groupId>
<artifactId>jackson-bom</artifactId>
<version>2.15.0</version>
<type>pom</type>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-bom</artifactId>
<version>2.19.0</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
<dependencies>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>s3</artifactId>
</dependency>
<dependency>
<groupId>software.amazon.awssdk</groupId>
<artifactId>netty-nio-client</artifactId>
<version>2.31.4</version>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-core</artifactId>
<version>3.5.8</version>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-web</artifactId>
<version>6.0.11</version>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-slf4j2-impl</artifactId>
</dependency>
</dependencies>
</project>If it's a connection availability issue rather, then increasing timeouts may not help especially if the concurrent connections in pool is limited to 1, which is never getting released. I will review this issue with the team and shall keep you posted. Regards, |
bhoradc
added
needs-review
Describe the bug
We are experiencing connection leak issues with the AWS SDK when downloading a file to S3.
If an error occurs just before subscribing to the donwload stream (or just if we never subscribe to the stream), the connection is never released.
Expected Behavior
I assume the connection should be released after a certain period of inactivity? However, none of the timeouts I have configured seem to have any effect.
Current Behavior
Once all the connections in the pool are occupied, we get connection acquisition errors. Even after an hour.
Reproduction Steps
For example, here I have a simple test with one connection in the pool, and if I try to call
/downloadtwice, the second one will never be able to acquire a connection.Possible Solution
No response
Additional Information/Context
No response
AWS Java SDK version used
2.26.12
JDK version used
21
Operating System and version
Windows 11
The text was updated successfully, but these errors were encountered: