Skip to content

Commit 93e7b3e

Browse files
julianwoodjulianwood
authored
Merge pull request #2513 from sarika-subram/main
New serverless pattern: s3-lambda-dynamodb-terraform
2 parents b552c9c + 87aa8bd commit 93e7b3e

15 files changed

+544
-0
lines changed

‎s3-lambda-dynamodb-terraform/README.md

+73
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,73 @@
1+
# Load data from JSON files in Amazon S3 into Amazon DynamoDB using S3 Event Notification and AWS Lambda
2+
3+
This pattern in [Terraform](https://www.terraform.io/) offers a complete solution to load data from JSON files uploaded to S3. The following resources are created:
4+
1. S3 Bucket with event notification on object created
5+
2. DynamoDB Table with on-demand billing mode
6+
3. Lambda function that runs python with an environment variable containing the dynamodb table name
7+
8+
## Requirements
9+
10+
* [Create an AWS account](https://portal.aws.amazon.com/gp/aws/developer/registration/index.html) if you do not already have one and log in. The IAM user that you use must have sufficient permissions to make necessary AWS service calls and manage AWS resources.
11+
* [AWS CLI](https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2.html) installed and configured
12+
* [Git Installed](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git)
13+
* [Terraform](https://developer.hashicorp.com/terraform/tutorials/aws-get-started/install-cli) version 1.x (this pattern has been tested with version 1.9.8)
14+
15+
## Deployment Instructions
16+
17+
1. Create a new directory, navigate to that directory in a terminal and clone the GitHub repository:
18+
```bash
19+
git clone https://github.com/aws-samples/serverless-patterns
20+
```
21+
2. Change directory to the pattern directory its source code folder:
22+
```bash
23+
cd s3-lambda-dynamodb-terraform/
24+
```
25+
3. From the command line, use Terraform to deploy the AWS resources for the pattern as specified in the main.tf file::
26+
```
27+
terraform init
28+
terraform apply --auto-approve
29+
```
30+
4. Note the outputs from the Terraform deployment process. These contain the resource names which are used for testing.
31+
32+
## Testing
33+
34+
### Initiate the data load process
35+
1. Once deployment has completed, locate the S3 Bucket Name and DynamoDB table name in the output, for example:
36+
``` bash
37+
module.s3_event.aws_s3_bucket.json_bucket: Creation complete after 2s [id=s3-lambda-dynamodb-terraform-json-store]
38+
39+
module.dynamodb.aws_dynamodb_table.basic-dynamodb-table: Creation complete after 7s [id=dev-test]
40+
```
41+
42+
2. A sample JSON file is provided in the `samples` folder. You can upload it using AWS CLI:
43+
``` bash
44+
aws s3 cp ./samples/test.json s3://s3-lambda-dynamodb-terraform-json-store
45+
```
46+
47+
> **Important**: When uploading your own JSON files, ensure they contain the following mandatory field:
48+
> - `UserId`: Unique identifier for the record
49+
50+
Example JSON format:
51+
```json
52+
{
53+
"UserId": "user123",
54+
"name": "John Doe",
55+
"email": "[email protected]"
56+
}
57+
```
58+
59+
3. Verify the data in DynamoDB:
60+
```bash
61+
aws dynamodb scan --table-name dev-test
62+
```
63+
64+
## Documentation
65+
- [Amazon S3 Event Notifications](https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html)
66+
67+
## Cleanup
68+
69+
1. Delete the stack
70+
```bash
71+
terraform destroy --auto-approve
72+
```
73+
2. Confirm the removal and wait for the resource deletion to complete.

‎s3-lambda-dynamodb-terraform/example-pattern.json

+56
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,56 @@
1+
{
2+
"title": "Amazon S3 to AWS Lambda to Amazon DynamoDB",
3+
"description": "Upload object data from S3 to DynamoDB via Lambda.",
4+
"language": "Python",
5+
"level": "200",
6+
"framework": "Terraform",
7+
"introBox": {
8+
"headline": "How it works",
9+
"text": [
10+
"This pattern in Terraform offers a complete solution to load data from JSON files stored on S3. The following resources are created:",
11+
"- S3 Bucket with event notification on object creates",
12+
"- DynamoDB Table with on-demand billing mode",
13+
"- Lambda function that runs python and takes environment variables of bucket name, and dynamodb table"
14+
]
15+
},
16+
"gitHub": {
17+
"template": {
18+
"repoURL": "https://github.com/aws-samples/serverless-patterns/tree/main/s3-lambda-dynamodb-terraform",
19+
"templateURL": "serverless-patterns/s3-lambda-dynamodb-terraform",
20+
"projectFolder": "s3-lambda-dynamodb-terraform",
21+
"templateFile": "main.tf"
22+
}
23+
},
24+
"resources": {
25+
"bullets": [
26+
{
27+
"text": "Amazon S3 Event Notifications",
28+
"link": "https://docs.aws.amazon.com/AmazonS3/latest/userguide/NotificationHowTo.html"
29+
}
30+
]
31+
},
32+
"deploy": {
33+
"text": [
34+
"terraform init",
35+
"terraform apply --auto-approve"
36+
]
37+
},
38+
"testing": {
39+
"text": [
40+
"See the GitHub repo for detailed testing instructions."
41+
]
42+
},
43+
"cleanup": {
44+
"text": [
45+
"Delete the stack: <code>terraform destroy --auto-approve</code>."
46+
]
47+
},
48+
"authors": [
49+
{
50+
"name": "Sarika Subramaniam",
51+
"image": "link-to-your-photo.jpg",
52+
"bio": "Sarika is a Solutions Architect at Amazon Web Services based in London.",
53+
"linkedin": "sarika-subramaniam-9ba591118/"
54+
}
55+
]
56+
}

‎s3-lambda-dynamodb-terraform/main.tf

+19
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
provider "aws" {
2+
region = var.aws_region
3+
default_tags {
4+
tags = {
5+
Project = var.project
6+
Team = var.team
7+
CostCentre = var.costcentre
8+
}
9+
}
10+
}
11+
12+
module "s3_event" {
13+
source = "./modules/s3_event"
14+
table_name = module.dynamodb.table_name
15+
}
16+
17+
module "dynamodb" {
18+
source = "./modules/dynamodb"
19+
}

‎s3-lambda-dynamodb-terraform/modules/dynamodb/main.tf

+15
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,15 @@
1+
locals {
2+
table_name = "${var.project}-${var.table_name}"
3+
}
4+
5+
# Create the Dynamodb table
6+
resource "aws_dynamodb_table" "basic-dynamodb-table" {
7+
name = local.table_name
8+
billing_mode = "PAY_PER_REQUEST"
9+
hash_key = var.hash_key
10+
11+
attribute {
12+
name = var.hash_key
13+
type = "S"
14+
}
15+
}

‎s3-lambda-dynamodb-terraform/modules/dynamodb/outputs.tf

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
output "table_name" {
2+
value = aws_dynamodb_table.basic-dynamodb-table.name
3+
}

‎s3-lambda-dynamodb-terraform/modules/dynamodb/variables.tf

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
variable "table_name" {
2+
description = "Name of the DynamoDB table"
3+
type = string
4+
default = "test"
5+
}
6+
7+
variable "hash_key" {
8+
description = "Hash key for the DynamoDB table"
9+
type = string
10+
default = "UserId"
11+
}
12+
13+
variable "project" {
14+
description = "Project name"
15+
type = string
16+
default = "s3-lambda-dynamodb-terraform"
17+
}

‎s3-lambda-dynamodb-terraform/modules/s3_event/code/lambda_function.py

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
import json
2+
import boto3
3+
import os
4+
5+
s3 = boto3.client('s3')
6+
dynamodb = boto3.resource('dynamodb')
7+
8+
# DynamoDB table name
9+
table_name = os.environ['DYNAMODB_TABLE_NAME']
10+
11+
def lambda_handler(event, context):
12+
try:
13+
# Get the S3 bucket and key (file path) from the event
14+
bucket_name = event['Records'][0]['s3']['bucket']['name']
15+
key = event['Records'][0]['s3']['object']['key']
16+
17+
# Download the JSON file from S3
18+
response = s3.get_object(Bucket=bucket_name, Key=key)
19+
data = json.load(response['Body'])
20+
21+
# Write the data to the DynamoDB table
22+
table = dynamodb.Table(table_name)
23+
table.put_item(Item=data)
24+
25+
return {
26+
'statusCode': 200,
27+
'body': json.dumps('Data successfully written to DynamoDB table!')
28+
}
29+
30+
except Exception as e:
31+
return {
32+
'statusCode': 500,
33+
'body': json.dumps(f'Error: {str(e)}')
34+
}

‎s3-lambda-dynamodb-terraform/modules/s3_event/code/tests/s3_put_event.json

+14
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
{
2+
"Records": [
3+
{
4+
"s3": {
5+
"bucket": {
6+
"name": "s3-lambda-dynamodb-terraform-json-store"
7+
},
8+
"object": {
9+
"key": "test.json"
10+
}
11+
}
12+
}
13+
]
14+
}

‎s3-lambda-dynamodb-terraform/modules/s3_event/main.tf

+111
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,111 @@
1+
locals {
2+
s3_name = "${var.project}-${var.bucket_name}"
3+
}
4+
5+
data "aws_iam_policy_document" "assume_role" {
6+
statement {
7+
effect = "Allow"
8+
9+
principals {
10+
type = "Service"
11+
identifiers = ["lambda.amazonaws.com"]
12+
}
13+
14+
actions = ["sts:AssumeRole"]
15+
}
16+
}
17+
18+
data "archive_file" "python_lambda_package" {
19+
type = "zip"
20+
source_file = "${path.module}/code/lambda_function.py"
21+
output_path = "lambda_function_payload.zip"
22+
}
23+
24+
resource "aws_iam_role" "iam_for_lambda" {
25+
name = "iam_for_lambda"
26+
assume_role_policy = data.aws_iam_policy_document.assume_role.json
27+
}
28+
29+
resource "aws_iam_policy" "write_to_ddb" {
30+
name = "write_to_ddb"
31+
path = "/"
32+
description = "Write to ddb policy"
33+
34+
# Terraform's "jsonencode" function converts a
35+
# Terraform expression result to valid JSON syntax.
36+
policy = jsonencode({
37+
Version = "2012-10-17"
38+
Statement = [
39+
{
40+
"Sid": "Statement1",
41+
"Effect": "Allow",
42+
"Action": [
43+
"dynamodb:UpdateTable",
44+
"dynamodb:CreateTable",
45+
"dynamodb:BatchWriteItem",
46+
"dynamodb:PutItem",
47+
"dynamodb:UpdateItem"
48+
],
49+
"Resource": [
50+
"*"
51+
]
52+
}
53+
]
54+
})
55+
}
56+
57+
resource "aws_iam_role_policy_attachment" "lambda_basic_policy" {
58+
role = aws_iam_role.iam_for_lambda.name
59+
policy_arn = "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
60+
}
61+
62+
resource "aws_iam_role_policy_attachment" "s3_read_only" {
63+
role = aws_iam_role.iam_for_lambda.name
64+
policy_arn = "arn:aws:iam::aws:policy/AmazonS3ReadOnlyAccess"
65+
}
66+
67+
resource "aws_iam_role_policy_attachment" "write_to_ddb" {
68+
role = aws_iam_role.iam_for_lambda.name
69+
policy_arn = aws_iam_policy.write_to_ddb.arn
70+
}
71+
72+
resource "aws_lambda_permission" "allow_bucket" {
73+
statement_id = "AllowExecutionFromS3Bucket"
74+
action = "lambda:InvokeFunction"
75+
function_name = aws_lambda_function.test_lambda.arn
76+
principal = "s3.amazonaws.com"
77+
source_arn = aws_s3_bucket.json_bucket.arn
78+
}
79+
80+
resource "aws_lambda_function" "test_lambda" {
81+
filename = "lambda_function_payload.zip"
82+
function_name = "upload_to_ddb"
83+
role = aws_iam_role.iam_for_lambda.arn
84+
handler = "lambda_function.lambda_handler"
85+
86+
source_code_hash = data.archive_file.python_lambda_package.output_base64sha256
87+
88+
runtime = "python3.12"
89+
90+
environment {
91+
variables = {
92+
DYNAMODB_TABLE_NAME = var.table_name
93+
}
94+
}
95+
}
96+
97+
resource "aws_s3_bucket" "json_bucket" {
98+
bucket = local.s3_name
99+
force_destroy = true
100+
}
101+
102+
resource "aws_s3_bucket_notification" "bucket_notification" {
103+
bucket = aws_s3_bucket.json_bucket.id
104+
105+
lambda_function {
106+
lambda_function_arn = aws_lambda_function.test_lambda.arn
107+
events = ["s3:ObjectCreated:*"]
108+
}
109+
110+
depends_on = [aws_lambda_permission.allow_bucket]
111+
}

‎s3-lambda-dynamodb-terraform/modules/s3_event/outputs.tf

Whitespace-only changes.

‎s3-lambda-dynamodb-terraform/modules/s3_event/variables.tf

+16
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,16 @@
1+
variable "bucket_name" {
2+
description = "Name of the S3 bucket for website hosting"
3+
type = string
4+
default = "json-store"
5+
}
6+
7+
variable "table_name" {
8+
type = string
9+
description = "Name of the DynamoDB table"
10+
}
11+
12+
variable "project" {
13+
description = "Project name"
14+
type = string
15+
default = "s3-lambda-dynamodb-terraform"
16+
}

0 commit comments

Comments
 (0)