(aws-s3-deployment): CDKBucketDeployment custom resource poor s3 sync performance #35487
Description
Describe the bug
Deploying a static app using the BucketDeployment construct in aws-s3-deployment results in extremely poor AWS S3 sync performance. For example, the log below (with details removed)
2025-09-15T01:24:28.544Z
START RequestId: [REQUEST-ID] Version: $LATEST
2025-09-15T01:24:28.545Z
[INFO] 2025-09-15T01:24:28.545Z [REQUEST-ID] {'RequestType': 'Update', 'ServiceToken': 'arn:aws:lambda:ap-southeast-2:[ACCOUNT-ID]:function:[STACK-NAME]-CustomCDKBucketDeployment[HASH]-[RESOURCE-ID]', 'StackId': 'arn:aws:cloudformation:ap-southeast-2:[ACCOUNT-ID]:stack/[STACK-NAME]/[STACK-ID]', 'RequestId': '[CF-REQUEST-ID]', 'LogicalResourceId': 'frontenddeploymentCustomResource[HASH]', 'PhysicalResourceId': 'aws.cdk.s3deployment.[UUID]', 'ResourceType': 'Custom::CDKBucketDeployment', 'ResourceProperties': {'ServiceToken': 'arn:aws:lambda:ap-southeast-2:[ACCOUNT-ID]:function:[STACK-NAME]-CustomCDKBucketDeployment[HASH]-[RESOURCE-ID]', 'Prune': 'true', 'OutputObjectKeys': 'true', 'SourceBucketNames': ['cdk-[CDK-QUALIFIER]-assets-[ACCOUNT-ID]-ap-southeast-2'], 'DestinationBucketName': '[STACK-NAME]-frontendstaticwebbucket[HASH]-[BUCKET-SUFFIX]', 'DistributionId': '[CLOUDFRONT-DISTRIBUTION-ID]', 'SourceObjectKeys': ['[ASSET-HASH].zip'], 'DistributionPaths': ['/*']}, 'OldResourceProperties': {'ServiceToken': 'arn:aws:lambda:ap-southeast-2:[ACCOUNT-ID]:function:[STACK-NAME]-CustomCDKBucketDeployment[HASH]-[RESOURCE-ID]', 'Prune': 'true', 'OutputObjectKeys': 'true', 'SourceBucketNames': ['cdk-[CDK-QUALIFIER]-assets-[ACCOUNT-ID]-ap-southeast-2'], 'DestinationBucketName': '[STACK-NAME]-frontendstaticwebbucket[HASH]-[BUCKET-SUFFIX]', 'DistributionId': '[CLOUDFRONT-DISTRIBUTION-ID]', 'SourceObjectKeys': ['[OLD-ASSET-HASH].zip'], 'DistributionPaths': ['/*']}}
2025-09-15T01:24:28.545Z
[INFO] 2025-09-15T01:24:28.545Z [REQUEST-ID] | s3_dest: s3%3A//[DESTINATION-BUCKET]/
2025-09-15T01:24:28.545Z
[INFO] 2025-09-15T01:24:28.545Z [REQUEST-ID] | old_s3_dest: s3%3A//[DESTINATION-BUCKET]/
2025-09-15T01:24:28.546Z
[INFO] 2025-09-15T01:24:28.546Z [REQUEST-ID] | workdir: /tmp/[TEMP-DIR]
2025-09-15T01:24:28.546Z
[INFO] 2025-09-15T01:24:28.546Z [REQUEST-ID] archive: /tmp/[TEMP-DIR]/[UUID]
2025-09-15T01:24:28.546Z
[INFO] 2025-09-15T01:24:28.546Z [REQUEST-ID] | aws s3 cp s3://cdk-[CDK-QUALIFIER]-assets-[ACCOUNT-ID]-ap-southeast-2/[ASSET-HASH].zip /tmp/[TEMP-DIR]/[UUID]
2025-09-15T01:24:40.181Z
Completed 256.0 KiB/7.7 MiB (510.2 KiB/s) with 1 file(s) remaining Completed 512.0 KiB/7.7 MiB (984.0 KiB/s) with 1 file(s) remaining Completed 768.0 KiB/7.7 MiB (1.4 MiB/s) with 1 file(s) remaining Completed 1.0 MiB/7.7 MiB (1.9 MiB/s) with 1 file(s) remaining Completed 1.2 MiB/7.7 MiB (2.2 MiB/s) with 1 file(s) remaining Completed 1.5 MiB/7.7 MiB (2.7 MiB/s) with 1 file(s) remaining Completed 1.8 MiB/7.7 MiB (3.1 MiB/s) with 1 file(s) remaining Completed 2.0 MiB/7.7 MiB (3.6 MiB/s) with 1 file(s) remaining Completed 2.2 MiB/7.7 MiB (3.6 MiB/s) with 1 file(s) remaining Completed 2.5 MiB/7.7 MiB (4.0 MiB/s) with 1 file(s) remaining Completed 2.8 MiB/7.7 MiB (4.2 MiB/s) with 1 file(s) remaining Completed 3.0 MiB/7.7 MiB (4.5 MiB/s) with 1 file(s) remaining Completed 3.2 MiB/7.7 MiB (4.9 MiB/s) with 1 file(s) remaining Completed 3.5 MiB/7.7 MiB (4.9 MiB/s) with 1 file(s) remaining Completed 3.8 MiB/7.7 MiB (5.2 MiB/s) with 1 file(s) remaining Completed 4.0 MiB/7.7 MiB (5.5 MiB/s) with 1 file(s) remaining Completed 4.2 MiB/7.7 MiB (5.9 MiB/s) with 1 file(s) remaining Completed 4.5 MiB/7.7 MiB (5.8 MiB/s) with 1 file(s) remaining Completed 4.8 MiB/7.7 MiB (6.1 MiB/s) with 1 file(s) remaining Completed 5.0 MiB/7.7 MiB (6.4 MiB/s) with 1 file(s) remaining Completed 5.2 MiB/7.7 MiB (6.1 MiB/s) with 1 file(s) remaining Completed 5.5 MiB/7.7 MiB (6.2 MiB/s) with 1 file(s) remaining Completed 5.8 MiB/7.7 MiB (6.5 MiB/s) with 1 file(s) remaining Completed 6.0 MiB/7.7 MiB (6.8 MiB/s) with 1 file(s) remaining Completed 6.2 MiB/7.7 MiB (7.1 MiB/s) with 1 file(s) remaining Completed 6.5 MiB/7.7 MiB (6.9 MiB/s) with 1 file(s) remaining Completed 6.8 MiB/7.7 MiB (7.2 MiB/s) with 1 file(s) remaining Completed 7.0 MiB/7.7 MiB (7.3 MiB/s) with 1 file(s) remaining Completed 7.2 MiB/7.7 MiB (7.5 MiB/s) with 1 file(s) remaining Completed 7.5 MiB/7.7 MiB (7.8 MiB/s) with 1 file(s) remaining Completed 7.7 MiB/7.7 MiB (7.7 MiB/s) with 1 file(s) remaining download: s3://cdk-[CDK-QUALIFIER]-assets-[ACCOUNT-ID]-ap-southeast-2/[ASSET-HASH].zip to ../../tmp/[TEMP-DIR]/[UUID]
2025-09-15T01:24:42.162Z
[INFO] 2025-09-15T01:24:42.162Z [REQUEST-ID] | extracting archive to: /tmp/[TEMP-DIR]/contents
2025-09-15T01:24:42.162Z
[INFO] 2025-09-15T01:24:42.162Z [REQUEST-ID] | markers: {}
2025-09-15T01:24:44.622Z
[INFO] 2025-09-15T01:24:44.622Z [REQUEST-ID] | aws s3 sync --delete /tmp/[TEMP-DIR]/contents s3://[DESTINATION-BUCKET]/
2025-09-15T01:24:56.781Z
Completed 4.6 KiB/~19.4 KiB (8.2 KiB/s) with ~7 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/app.component-[HASH].css.map to s3://[DESTINATION-BUCKET]/app.component-[HASH].css.map
2025-09-15T01:24:57.101Z
Completed 4.6 KiB/~19.4 KiB (8.2 KiB/s) with ~6 file(s) remaining (calculating...) Completed 6.1 KiB/~19.4 KiB (7.4 KiB/s) with ~6 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/chunk-[HASH].js.map to s3://[DESTINATION-BUCKET]/chunk-[HASH].js.map
2025-09-15T01:24:57.741Z
Completed 6.1 KiB/~19.4 KiB (7.4 KiB/s) with ~5 file(s) remaining (calculating...) Completed 10.0 KiB/~33.6 KiB (6.7 KiB/s) with ~8 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/ClusterAdminDialog.component-[HASH].css.map to s3://[DESTINATION-BUCKET]/ClusterAdminDialog.component-[HASH].css.map
2025-09-15T01:24:58.161Z
Completed 10.0 KiB/~33.6 KiB (6.7 KiB/s) with ~7 file(s) remaining (calculating...) Completed 16.7 KiB/~39.3 KiB (8.6 KiB/s) with ~9 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/chunk-[HASH].js.map to s3://[DESTINATION-BUCKET]/chunk-[HASH].js.map
2025-09-15T01:24:58.801Z
Completed 16.7 KiB/~39.3 KiB (8.6 KiB/s) with ~8 file(s) remaining (calculating...) Completed 17.4 KiB/~326.5 KiB (6.7 KiB/s) with ~12 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/chunk-[HASH].js.map to s3://[DESTINATION-BUCKET]/chunk-[HASH].js.map
2025-09-15T01:24:58.941Z
Completed 17.4 KiB/~326.5 KiB (6.7 KiB/s) with ~11 file(s) remaining (calculating...) Completed 21.0 KiB/~326.5 KiB (7.9 KiB/s) with ~11 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/chunk-[HASH].js to s3://[DESTINATION-BUCKET]/chunk-[HASH].js
2025-09-15T01:24:58.941Z
Completed 21.0 KiB/~326.5 KiB (7.9 KiB/s) with ~10 file(s) remaining (calculating...) Completed 21.8 KiB/~326.5 KiB (8.1 KiB/s) with ~10 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/change-my-password.component-[HASH].css.map to s3://[DESTINATION-BUCKET]/change-my-password.component-[HASH].css.map
2025-09-15T01:24:59.301Z
Completed 21.8 KiB/~326.5 KiB (8.1 KiB/s) with ~9 file(s) remaining (calculating...) Completed 23.2 KiB/~523.7 KiB (7.9 KiB/s) with ~13 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/chunk-[HASH].js to s3://[DESTINATION-BUCKET]/chunk-[HASH].js
2025-09-15T01:24:59.441Z
Completed 23.2 KiB/~523.7 KiB (7.9 KiB/s) with ~12 file(s) remaining (calculating...) Completed 23.6 KiB/~717.5 KiB (7.3 KiB/s) with ~14 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/chunk-[HASH].js to s3://[DESTINATION-BUCKET]/chunk-[HASH].js
2025-09-15T01:24:59.901Z
Completed 23.6 KiB/~717.5 KiB (7.3 KiB/s) with ~13 file(s) remaining (calculating...) Completed 25.9 KiB/~857.5 KiB (7.8 KiB/s) with ~14 file(s) remaining (calculating...) Completed 30.8 KiB/~857.5 KiB (9.0 KiB/s) with ~14 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/chunk-[HASH].js to s3://[DESTINATION-BUCKET]/chunk-[HASH].js
2025-09-15T01:24:59.901Z
Completed 30.8 KiB/~857.5 KiB (9.0 KiB/s) with ~13 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/chunk-[HASH].js.map to s3://[DESTINATION-BUCKET]/chunk-[HASH].js.map
2025-09-15T01:25:00.861Z
Completed 30.8 KiB/~18.4 MiB (9.0 KiB/s) with ~37 file(s) remaining (calculating...) Completed 132.3 KiB/~18.7 MiB (29.5 KiB/s) with ~41 file(s) remaining (calculating...) upload: ../../tmp/[TEMP-DIR]/contents/chunk-[HASH].js.map to s3://[DESTINATION-BUCKET]/chunk-[HASH].js.map
[... continues with multiple file uploads showing transfer progress ...]
2025-09-15T01:32:21.921Z
Completed 18.8 MiB/25.0 MiB (44.7 KiB/s) with 26 file(s) remaining
2025-09-15T01:39:28.247Z
END RequestId: [REQUEST-ID]
2025-09-15T01:39:28.247Z
REPORT RequestId: [REQUEST-ID] Duration: 900000.00 ms Billed Duration: 900000 ms Memory Size: 128 MB Max Memory Used: 127 MB Status: timeout
You can see it eventually timed out after 900 seconds!
I theorise this is due to the low default memory limit - now I'm looking into it - 128MB. I will validate this is the case and then possibly create a PR to bump the default memory limit. This is sneaky and can cause CloudFormation deployment hang-ups which end up costing a lot more money/time vs the extra Lambda memory.
Regression Issue
- Select this option if this issue appears to be a regression.
Last Known Working CDK Library Version
No response
Expected Behavior
Complete deployment without timing out.
Current Behavior
Default 128MB memory limit (possibly) causing deployment hang up.
Reproduction Steps
Deploy a S3 bucket deployment with a reasonable number of files.
Possible Solution
Change default memory limit for the lambda custom resource.
Additional Information/Context
No response
AWS CDK Library version (aws-cdk-lib)
2.202.0
AWS CDK CLI version
2.1019.1
Node.js Version
v24.8.0
OS
Distributor ID: Ubuntu Description: Ubuntu 24.04.3 LTS Release: 24.04 Codename: noble
Language
TypeScript
Language Version
No response
Other information
No response