diff --git a/cloudfront-apigw-large-uploads/README.md b/cloudfront-apigw-large-uploads/README.md new file mode 100644 index 000000000..c92e0b1aa --- /dev/null +++ b/cloudfront-apigw-large-uploads/README.md @@ -0,0 +1,141 @@ +# Route large API payloads to alternate origin +Amazon API Gateway has a [payload limit of 10mb](https://docs.aws.amazon.com/apigateway/latest/developerguide/limits.html#http-api-quotas). This pattern shows how to use Lambda@Edge to route client POST request to an alternative origin based on payload size of the POST request. + +Learn more about this pattern at Serverless Land Patterns: https://serverlessland.com/patterns/cloudfront-apigw-large-uploads + +Important: this application uses various AWS services and there are costs associated with these services after the Free Tier usage - please see the [AWS Pricing page](https://aws.amazon.com/pricing/) for details. You are responsible for any AWS costs incurred. No warranty is implied in this example. + +## Requirements + +* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources. +* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured +* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git) +* [AWS Cloud Development Kit](https://docs.aws.amazon.com/cdk/v2/guide/getting_started.html) (AWS CDK) installed + +## Deployment Instructions + +1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository: + ``` + git clone https://github.com/aws-samples/serverless-patterns + ``` +1. Change directory to the pattern directory: + ``` + cd cloudfront-apigw-large-uploads + ``` +1. Set region to us-east-1 for your CDK environment, this is because Lambda@edge functions need to exist us-east-1 region. +1. Bootstrap CDK: + ``` + cdk bootstrap + ``` +1. From the command line, use AWS CDK to deploy the AWS resources for the pattern as specified in the `large_uploads_test/test_stack.py` file: + ``` + python3 -m pip install -r requirements.txt + cdk synth + cdk deploy + ``` + +## How it works +An Amazon Cloudfront distribution is created with 2 origins: +* Default Origin - Amazon API Gateway with an AWS Lambda Integration returning stub responses based on HTTP request method. +* Custom Origin - A mock HTTP endpoint which sends a response with information from the request. We are using echo.free.beeceptor.com but you can use any other endpoint. + +When a HTTP POST request is handled by CloudFront, a Lambda@Edge function is invoked as an Origin Request. This Lambda@Edge function uses the method and `content-length` HTTP headers from the client request to determine whether the request should go to a custom origin. + +To demonstrate this behaviour, we used a filesize limit of 300 bytes. In real world applications, you would set this higher by modifying the `MAX_FILE_SIZE` variable in `lambda.mjs`. +![diagram](diagram.png) + + +## Testing + +1. Once deployed, find the CloudFront distribution URL and use it in the following `curl` commands + +#### Test 1: +A GET request that CloudFront will route to API gateway + +``` +curl https:// -i +``` + +Example output: +``` + HTTP/2 200 + content-type: application/json + ...more headers... + + {"message": "GET request processed by API Gateway!"} +``` + +#### Test 2: +A POST request that CloudFront will route to API gateway as the payload size is smaller than `MAX_FILE_SIZE` defined in `lambda.mjs`. You will need to replace the `smallFile.example` filename in the following command with your file. + +``` +curl https:// -i -X POST -F 'data=@smallFile.example' -H 'content-type: application/json' +``` + +Example output: +``` + HTTP/2 201 + content-type: application/json + ...more headers... + + {"message": "POST request processed by API Gateway!"} +``` + +#### Test 3: +A POST request that CloudFront will route to a custom origin as the payload size is larger than `MAX_FILE_SIZE` defined in `lambda.mjs`. You will need to replace the `largeFile.example` filename in the following command with your file. + +``` +curl https:// -i -X POST -F 'data=@largeFile.example' +``` + +Example output: +``` + HTTP/2 200 + content-type: application/json + ...more headers... + + { + "method": "POST", + "protocol": "https", + "host": "echo.free.beeceptor.com", + ...more response body +``` + + +#### Test 4: +A POST request to a specific URL that CloudFront will route to a custom origin based on the Behaviour defined in `app.py`. You will need to replace the `smallFile.example` filename in the following command with your file. + +``` +curl https:///upload/ -i -X POST -F 'data=@smallFile.example' +``` + +Example output: +``` + HTTP/2 200 + content-type: application/json + ...more headers... + + { + "method": "POST", + "protocol": "https", + "host": "echo.free.beeceptor.com", + ...more response body +``` + + +## Cleanup + +1. Delete the stack + ``` + cdk destroy + ``` + + +## Notes +There are restrictions to what the Lambda function can modify as part of the Origin Request. For example, CloudFront will not allow you to change the protocol from HTTPS to HTTP. Doing so will result in an origin configuration error from CloudFront. + + +---- +Copyright 2025 Amazon.com, Inc. or its affiliates. All Rights Reserved. + +SPDX-License-Identifier: MIT-0 diff --git a/cloudfront-apigw-large-uploads/app.py b/cloudfront-apigw-large-uploads/app.py new file mode 100644 index 000000000..e5319bcd3 --- /dev/null +++ b/cloudfront-apigw-large-uploads/app.py @@ -0,0 +1,133 @@ +import aws_cdk as cdk +from aws_cdk import aws_cloudfront as cloudfront +from aws_cdk import aws_cloudfront_origins as origins +from aws_cdk import aws_apigateway as apigw +from aws_cdk import aws_lambda as lambda_ +from aws_cdk import aws_iam as iam + +class CloudFrontApigwLargeUploadsStack(cdk.Stack): + def __init__(self, scope, construct_id, **kwargs): + super().__init__(scope, construct_id, **kwargs) + + # Create an API Gateway HTTP endpoint that will return a mock response + nonUploadApi = apigw.RestApi( + self, + "nonUploadApi", + endpoint_types=[apigw.EndpointType.REGIONAL], + deploy_options=apigw.StageOptions( + stage_name="mock", + throttling_rate_limit=100, + throttling_burst_limit=1000, + ), + cloud_watch_role=True, + deploy=True, + ) + + # Create a mock integration for the nonUploadApi / path that returns status 200 for GET requests + nonUploadApi.root.add_method( + "GET", + apigw.MockIntegration( + integration_responses=[ + apigw.IntegrationResponse( + status_code="200", + response_templates={ + "application/json": '{"message": "GET request processed by API Gateway!"}' + }, + ) + ], + passthrough_behavior=apigw.PassthroughBehavior.NEVER, + request_templates={"application/json": '{"statusCode": 200}'}, + ), + method_responses=[apigw.MethodResponse(status_code="200")], + ) + + # Create a mock integration for the nonUploadApi / path that returns status 200 for POST requests + nonUploadApi.root.add_method( + "POST", + apigw.MockIntegration( + integration_responses=[ + apigw.IntegrationResponse( + status_code="201", + response_templates={ + "application/json": '{"message": "POST request processed by API Gateway!"}' + }, + ) + ], + passthrough_behavior=apigw.PassthroughBehavior.NEVER, + request_templates={"application/json": '{"statusCode": 200}'}, + ), + method_responses=[apigw.MethodResponse(status_code="201")], + ) + + # Create Lamdba@edge function + edge_function = lambda_.Function( + self, + "OriginRequestFunction", + runtime=lambda_.Runtime.NODEJS_LATEST, + handler="lambda.handler", + code=lambda_.Code.from_asset("lambda.zip") + ) + + # Add Lambda@Edge permission + edge_function.add_permission( + "EdgeFunctionPermission", + principal=iam.ServicePrincipal("edgelambda.amazonaws.com"), + action="lambda:InvokeFunction" + ) + + # Add execution role permissions for Lambda@Edge + edge_function.role.add_to_policy( + iam.PolicyStatement( + effect=iam.Effect.ALLOW, + actions=[ + "logs:CreateLogGroup", + "logs:CreateLogStream", + "logs:PutLogEvents" + ], + resources=["arn:aws:logs:*:*:*"] + ) + ) + + # Create a CloudFront distribution with custom origin + distribution = cloudfront.Distribution( + self, + "testLargeUploadDistribution", + price_class=cloudfront.PriceClass.PRICE_CLASS_100, # Change this if you need more regions enabled in CloudFront + # Default behavior now points to API Gateway + default_behavior=cloudfront.BehaviorOptions( + origin=origins.RestApiOrigin( + nonUploadApi, + origin_path="/mock" + ), + edge_lambdas=[ + cloudfront.EdgeLambda( + function_version=edge_function.current_version, + event_type=cloudfront.LambdaEdgeEventType.ORIGIN_REQUEST + ) + ], + viewer_protocol_policy=cloudfront.ViewerProtocolPolicy.HTTPS_ONLY, + origin_request_policy=cloudfront.OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER, + cache_policy=cloudfront.CachePolicy.CACHING_DISABLED, + allowed_methods=cloudfront.AllowedMethods.ALLOW_ALL, + ), + # Additional behavior for /upload/* paths to hit a test endpoint httpbin which will provide a generic response. + additional_behaviors={ + "/upload/*": cloudfront.BehaviorOptions( + origin=origins.HttpOrigin( + "echo.free.beeceptor.com", # This is the URL of a mock file server. + protocol_policy=cloudfront.OriginProtocolPolicy.HTTPS_ONLY, + https_port=443, + origin_path="/" # The origin path is set to / for the example endpoint. It will need to be modified to the correct URI for real world use-cases. + ), + viewer_protocol_policy=cloudfront.ViewerProtocolPolicy.HTTPS_ONLY, + cache_policy=cloudfront.CachePolicy.CACHING_DISABLED, + allowed_methods=cloudfront.AllowedMethods.ALLOW_ALL, # Allow all HTTP methods for uploads + origin_request_policy=cloudfront.OriginRequestPolicy.ALL_VIEWER_EXCEPT_HOST_HEADER # Forward all headers and query strings + ) + }, + comment="Distribution to test alternative origin for large uploads" + ) + +app = cdk.App() +CloudFrontApigwLargeUploadsStack(app, "CloudFrontApiGatewayLargeUploads") +app.synth() diff --git a/cloudfront-apigw-large-uploads/cdk.json b/cloudfront-apigw-large-uploads/cdk.json new file mode 100644 index 000000000..5be9dde5e --- /dev/null +++ b/cloudfront-apigw-large-uploads/cdk.json @@ -0,0 +1,26 @@ +{ + "app": "python3 app.py", + "watch": { + "include": [ + "**" + ], + "exclude": [ + "README.md", + "cdk*.json", + "requirements*.txt", + "**/__init__.py", + "python/__pycache__", + "tests" + ] + }, + "context": { + "@aws-cdk/core:target-partitions": [ + "aws", + "aws-cn" + ], + "@aws-cdk/aws-cloudfront:defaultSecurityPolicyTLSv1.2_2021": true, + "@aws-cdk/aws-apigateway:requestValidatorUniqueId": true, + "@aws-cdk/aws-lambda:recognizeLayerVersion": true, + "@aws-cdk/core:checkSecretUsage": true + } +} \ No newline at end of file diff --git a/cloudfront-apigw-large-uploads/cloudfront-apigw-large-uploads.json b/cloudfront-apigw-large-uploads/cloudfront-apigw-large-uploads.json new file mode 100644 index 000000000..877ee7e50 --- /dev/null +++ b/cloudfront-apigw-large-uploads/cloudfront-apigw-large-uploads.json @@ -0,0 +1,44 @@ +{ + "title": "Amazon Cloudfront to APIGW routing to alternative origin for large payloads", + "description": "Amazon Cloudfront to APIGW routing to alternative origin for large payloads using Lambda@Edge", + "language": "Python", + "level": "200", + "framework": "CDK", + "introBox": { + "headline": "How it works", + "text": [ + "This pattern shows how to use Lambda@Edge to route client POST request to an alternative origin when a POST request exceeds the 10mb payload limit for API Gateway." + ] + }, + "gitHub": { + "template": { + "repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/cloudfront-apigw-large-uploads", + "templateURL": "serverless-patterns/cloudfront-apigw-large-uploads", + "projectFolder": "cloudfront-apigw-large-uploads", + "templateFile": "app.py" + } + }, + "deploy": { + "text": [ + "cdk deploy" + ] + }, + "testing": { + "text": [ + "See the GitHub repo for detailed testing instructions." + ] + }, + "cleanup": { + "text": [ + "Delete the stack: cdk destroy." + ] + }, + "authors": [ + { + "name": "Shaun Guo", + "image": "https://media.licdn.com/dms/image/C5103AQG3KMyMdEIKpA/profile-displayphoto-shrink_800_800/0/1517283953925?e=1692835200&v=beta&t=AxJ9ST_8K_bw8nqTPDaJB2F5dnQspES9FuJ64DBScC8", + "bio": "Shaun is a Senior Technical Account Manager at Amazon Web Services based in Australia", + "linkedin": "shaun-guo" + } + ] +} \ No newline at end of file diff --git a/cloudfront-apigw-large-uploads/diagram.png b/cloudfront-apigw-large-uploads/diagram.png new file mode 100644 index 000000000..5804403e6 Binary files /dev/null and b/cloudfront-apigw-large-uploads/diagram.png differ diff --git a/cloudfront-apigw-large-uploads/lambda.mjs b/cloudfront-apigw-large-uploads/lambda.mjs new file mode 100644 index 000000000..d4238b0a1 --- /dev/null +++ b/cloudfront-apigw-large-uploads/lambda.mjs @@ -0,0 +1,27 @@ +// This is a copy of the file found in lambda.zip. + +// Using a small value of 300 bytes to demonstrate the behaviour of Lambda@Edge function. +// In real world applications, you would set this higher. +const MAX_FILE_SIZE = 300; + +// Using a random public facing endpoint that will respond to requests. +const LARGE_UPLOAD_ORIGIN = "echo.free.beeceptor.com"; + +export const handler = async (event) => { + const request = event.Records[0].cf.request; + const headers = request.headers; + const origin = event.Records[0].cf.request.origin; + + if (request.method === 'POST') { + if (headers['content-length'] && parseInt(headers['content-length'][0].value) < MAX_FILE_SIZE ) { + console.log(`Request size less than: ` + MAX_FILE_SIZE); + } else { + console.log(`Request has no content-length or request size is greater than: ` + MAX_FILE_SIZE); + origin.custom.domainName = LARGE_UPLOAD_ORIGIN; + request.headers.host[0].value = LARGE_UPLOAD_ORIGIN; + request.uri = '/'; + } + } + + return request; +}; \ No newline at end of file diff --git a/cloudfront-apigw-large-uploads/lambda.zip b/cloudfront-apigw-large-uploads/lambda.zip new file mode 100644 index 000000000..3bc3085c6 Binary files /dev/null and b/cloudfront-apigw-large-uploads/lambda.zip differ diff --git a/cloudfront-apigw-large-uploads/largeFile.example b/cloudfront-apigw-large-uploads/largeFile.example new file mode 100644 index 000000000..ad5d9616f Binary files /dev/null and b/cloudfront-apigw-large-uploads/largeFile.example differ diff --git a/cloudfront-apigw-large-uploads/requirements.txt b/cloudfront-apigw-large-uploads/requirements.txt new file mode 100644 index 000000000..863bba3c5 --- /dev/null +++ b/cloudfront-apigw-large-uploads/requirements.txt @@ -0,0 +1 @@ +aws-cdk-lib==2.181.1 \ No newline at end of file diff --git a/cloudfront-apigw-large-uploads/smallFile.example b/cloudfront-apigw-large-uploads/smallFile.example new file mode 100644 index 000000000..75483961a --- /dev/null +++ b/cloudfront-apigw-large-uploads/smallFile.example @@ -0,0 +1 @@ +1 \ No newline at end of file