SNS: Server unable to handle too much load — AWS SDK connection pool acquire timeout

[mariuszbluestonepim]

Under a heavy volume of SNS Publish calls, the AWS SDK fails with:

java.util.concurrent.CompletionException: software.amazon.awssdk.core.exception.SdkClientException: Unable to execute HTTP request: Acquire operation took longer than the configured maximum time. This indicates that a request cannot get a connection from the pool within the specified maximum time. This can be due to high request rate.

When the publish rate is high, the moto server can't process messages fast enough to keep up, so the client's connection pool stays saturated and acquisition times out. It appears to be a single-process throughput limit.

Note: the SNS topic has an SQS subscription (SNS → SQS fan-out), so each publish also performs delivery to the subscribed queue inline within the publish request.

The same client configuration works without errors against LocalStack, which points to a server-side throughput difference rather than a client issue.

Environment

moto 5.2.2, server mode (motoserver/moto Docker image)
AWS SDK for Java v2 (Netty async client, default pool settings)

Expected: moto server sustains a heavy SNS publish load (including SNS → SQS delivery) without exhausting the client connection pool, comparably to LocalStack.

[bblommers]

Hi @mariuszbluestonepim! Can you share a small reproducible sample?

And second - what do you consider a heavy volume? How many RPS/messages does one need to send before this becomes a problem?

[mariuszbluestonepim]

Hi. Reproducible sample attached, plus the RPS numbers you asked for.

Setup:
build.gradle

plugins {
    id 'java'
}

group = 'org.example'
version = '1.0-SNAPSHOT'

java {
    toolchain {
        languageVersion = JavaLanguageVersion.of(25)
    }
}

repositories {
    mavenCentral()
}

dependencies {
    implementation(platform("software.amazon.awssdk:bom:2.46.21"))
    implementation("software.amazon.awssdk:s3")
    implementation("software.amazon.awssdk:s3-transfer-manager")
    implementation("software.amazon.awssdk.crt:aws-crt:0.47.2")
    implementation("software.amazon.awssdk:sns")
    implementation("software.amazon.awssdk:sqs")
}

test {
    useJUnitPlatform()
}

docker-compose.yml

services:
  motoserver:
    image: motoserver/moto:5.2.2
    environment:
      - MOTO_PORT=4466
      - MOTO_ACCOUNT_ID=000000000000
      - AWS_DEFAULT_REGION=eu-central-1
      - AWS_ACCESS_KEY_ID=dev
      - AWS_SECRET_ACCESS_KEY=dev
    ports:
      - "127.0.0.1:4466:4466"
    deploy:
      resources:
        limits:
          memory: 1g
  localstack:
    image: localstack/localstack:4.14.0
    environment:
      - LOCALSTACK_HOST=localstack
      - SQS_ENDPOINT_STRATEGY=path
      - AWS_DEFAULT_REGION=eu-central-1
      - AWS_ACCESS_KEY_ID=dev
      - AWS_SECRET_ACCESS_KEY=dev
      - AWS_DEFAULT_REGION=eu-central-1
    ports:
      - "127.0.0.1:4566:4566"
    deploy:
      resources:
        limits:
          memory: 1g

Sample application:

Test3.java

Test results:
motoserver:

Endpoint http://localhost:4466 | ramp 100..1200 RPS (+100 every 5s)

rps       attempted       ok   failed  inflight     ok/s
100             500      500        0         0      100
200            1001     1000        0         1      200
300            1501     1342        0       160      268
400            2001      671        0      1489      134
500            2502      444        0      3546       88
600            3002      355      642      5551       71
700            3503      294     2208      6551       58
800            4004      295     2711      7549       59
900            4502      254     3250      8547       50
1000           5000      221     3776      9550       44
1100           5501      228     4780     10042       45
1200           6004      210     5791     10045       42

===== SUMMARY =====
endpoint            : http://localhost:4466
first failing RPS   : 600
total attempted     : 39021
total ok            : 5871
total failed        : 29003
  of which ACQUIRE-TIMEOUT: 21697
received by SQS     : 3013
failure breakdown:
   15504  TimeoutException: Acquire operation took longer than configured maximum time
    6193  TimeoutException: Acquire operation took longer than 10000 milliseconds.
    5786  IOException: The connection was closed during the request. The request will usually succeed on a retry, but if it does not: consider ...
    1506  IllegalStateException: Too many outstanding acquire operations
      14  SocketException: Connection reset
       1  SdkInterruptedException: null

localstack:

Endpoint http://localhost:4566 | ramp 100..1200 RPS (+100 every 5s)

rps       attempted       ok   failed  inflight     ok/s
100             500      500        0         0      100
200            1001      998        1         2      199
300            1500     1493        0         9      298
400            2002     1993        0        18      398
500            2501     2495        0        24      499
600            3002     3005        0        20      601
700            3501     3377        0       144      675
800            4001     4030        0       115      806
900            4501     4001        0       615      800
1000           5003     3988        0      1629      797
1100           5501     4162        0      2968      832
1200           6005     4025        0      4948      805

===== SUMMARY =====
endpoint            : http://localhost:4566
first failing RPS   : 200
total attempted     : 39018
total ok            : 36425
total failed        : 1
  of which ACQUIRE-TIMEOUT: 0
received by SQS     : 10241
failure breakdown:
       1  SdkInterruptedException: null

[Mide69]

Can this be assigned to me if it is available?