Skip to content

Commit 3b64ce8

Browse files
lsidalsida
authored
Merge pull request awslabs#108 from etiennemunnich/BuildLayers
Build Lambda Layers Bash Scripts
2 parents e59720c + cc2f3be commit 3b64ce8

11 files changed

+361
-0
lines changed

‎Lambda/BuildLambdaLayers/LICENSE.txt

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Copyright 2019-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
#
3+
# This file is licensed under the Apache License, Version 2.0 (the "License").
4+
# You may not use this file except in compliance with the License. A copy of the
5+
# License is located at
6+
#
7+
# http://aws.amazon.com/apache2.0/
8+
#
9+
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
10+
# OF ANY KIND, either express or implied. See the License for the specific
11+
# language governing permissions and limitations under the License.

‎Lambda/BuildLambdaLayers/NOTICE

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
BuildLambdaLayers
2+
Copyright [2019]-[2019] Amazon.com, Inc. or its affiliates. All Rights Reserved.

‎Lambda/BuildLambdaLayers/README.md

+34
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,34 @@
1+
# Lambda - Build Layers Shell Script
2+
3+
Lambda Layers allow for updating of libraries/mobdules for Lambda quickly, without the efforts of creating a custom Lambda runtime. These bash scripts help automate the download of libraries/modules and optionally, upload the layer and associate with a Lambda function.
4+
5+
This is particularily helpful when adding [AWS X-Ray SDK](https://docs.aws.amazon.com/xray/latest/devguide/aws-xray.html), latest versions of the AWS SDK or older/newer versions of other libraries/modules needed when troubleshooting or deploying functions for A/B type testing.
6+
7+
For more information on managing [Lambda layers](https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html)
8+
9+
## Requirements
10+
If you wish the script to upload the Lambda layer and deploy the layer to a Lambda function you will need [jq](https://stedolan.github.io/jq/) and zip.
11+
12+
[AWSCLI](https://docs.aws.amazon.com/cli/latest/userguide/install-macos.html) will need to be installed and configured if you wish to upload layers and/or associate layers with Lambda functions.
13+
14+
Each runtime will need to have the relevant package manager installed. Details are in the relevant README.md of each runtime.
15+
16+
## Permissions
17+
18+
The following User/Role IAM permissions are required in your AWS Account to upload the layer and associate the layer with a Lambda function:
19+
* Layer Development and Use [https://docs.aws.amazon.com/lambda/latest/dg/access-control-identity-based.html#permissions-user-layer]
20+
* This script does NOT assign resource policy's to the Lambda, you may need to consider adding permissions in certain scenarios. [https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-permissions]
21+
22+
## Usage
23+
24+
Each runtime has variences in how it is used - please check the README:
25+
[python/README.md]()
26+
[nodejs/README.md]()
27+
28+
This script will use the credentials and region configured with the AWS CLI if available. If not, AWSCLI it will ask for an AWS access key and secret key.
29+
30+
### NOTE: Please test these sample scripts thoroughly to see if they suit your use case.
31+
32+
33+
34+
Work Hard, Have Fun, Make History

‎Lambda/BuildLambdaLayers/nodejs/LICENSE.txt

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Copyright 2019-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
#
3+
# This file is licensed under the Apache License, Version 2.0 (the "License").
4+
# You may not use this file except in compliance with the License. A copy of the
5+
# License is located at
6+
#
7+
# http://aws.amazon.com/apache2.0/
8+
#
9+
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
10+
# OF ANY KIND, either express or implied. See the License for the specific
11+
# language governing permissions and limitations under the License.

‎Lambda/BuildLambdaLayers/nodejs/README.md

+36
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,36 @@
1+
# Lambda - Build Layers Shell Script - NodeJS
2+
3+
This bash script will help with downloading modules and libraries to a local folder, zip and optionally upload the layer as a new version and associate with a Lambda Function if desired.
4+
5+
## Usage
6+
7+
'''bash
8+
./buildlayer.sh requirements.txt nodejs-version [lambda-version-name] [function-name]
9+
'''
10+
11+
- requirements.txt is the list of python packages you wish to add to the layer.
12+
- node-version is the version of runtime the modules are downloaded for, eg 10.16.3
13+
* https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html for the runtime currently supported.
14+
- lambda-version-name (optional) if you want to upload the zip as a new version of an existing layer or as a new layer if no layer yet exists.
15+
- function-name (optional) if you want to add the layer to the $lastest of a lambda function. This updates the Lambda immediately!
16+
17+
This script will use the credentials and region configured with the AWS CLI if available. If not, AWSCLI it will ask for an AWS access key and secret key.
18+
19+
## Requirements
20+
* If you wish the script to upload the Lambda layer and deploy the layer to a Lambda function you will need [jq](https://stedolan.github.io/jq/) and zip.
21+
* [virtualenv](https://pip.pypa.io/en/stable/installing/)
22+
* [pip](https://virtualenv.pypa.io/en/latest/)
23+
* [npm](https://www.npmjs.com/get-npm)
24+
* [nodeenv](https://github.com/ekalinin/nodeenv#install)
25+
* [AWSCLI](https://docs.aws.amazon.com/cli/latest/userguide/install-macos.html) will need to be installed and configured if you wish to upload layers and/or associate layers with Lambda functions.
26+
27+
## Permissions
28+
The following User/Role IAM permissions are required in your AWS Account to upload the layer and associate the layer with a Lambda function:
29+
* Layer Development and Use [https://docs.aws.amazon.com/lambda/latest/dg/access-control-identity-based.html#permissions-user-layer]
30+
* This script does NOT assign resource policy's to the Lambda layer, you may need to consider adding permissions in certain scenarios. [https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-permissions]
31+
32+
### NOTE: Please test these sample scripts thoroughly to see if they suit your use case.
33+
34+
35+
36+
Work Hard, Have Fun, Make History

‎Lambda/BuildLambdaLayers/nodejs/buildlayer.sh

+114
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,114 @@
1+
#!/bin/bash
2+
set -o pipefail
3+
# Copyright [2019]-[2019] Amazon.com, Inc. or its affiliates. All Rights Reserved.
4+
# SPDX-License-Identifier: Apache-2.0
5+
#
6+
# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file
7+
# except in compliance with the License. A copy of the License is located at
8+
#
9+
# http://aws.amazon.com/apache2.0/
10+
#
11+
# or in the "license" file accompanying this file. This file is distributed on an "AS IS"
12+
# BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
13+
# License for the specific language governing permissions and limitations under the License.
14+
#
15+
# --------------------
16+
# Purpose: Build layers to allow for module testing the latest libraries rapidly or for diagnostics such as
17+
# X-Ray. Please test thoroughly if this suits your use-case, and if you have any issues report via git issues
18+
#
19+
# Note: This script can be used for any NodeJS version you choose to by specifying the version in the
20+
# bash script agrguments
21+
#
22+
# Example requirements.txt file:
23+
#
24+
# aws-sdk
25+
# aws-xray-sdk
26+
#
27+
# Example of requirements file specifying versions:
28+
#
29+
# [email protected]
30+
# [email protected]
31+
#
32+
# Requirements to allow this script to upload layer and/or update lambda are AWSCLI & jq.
33+
# --------------------
34+
35+
# Check if required args are passed on cmd line
36+
if ([ -f "$1" ] && [ "$2" != "" ]); then
37+
echo "Starting to build package:"
38+
else
39+
echo "ERR: Please provide the requirements file as a minimum."
40+
echo ""
41+
echo "Usage: buildlayer.sh requirements.txt node-version [lambda-version-name] [function-name]"
42+
echo ""
43+
echo " - requirements.txt is the list of python packages you wish to add to the layer. "
44+
echo " - node-version is the version of runtime the modules are downloaded for, eg 10.16.3 "
45+
echo " * https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html for the runtime currently supported."
46+
echo " - lambda-version-name (optional) if you want to upload the zip as a new version of an existing layer or as a new layer if no layer yet exists."
47+
echo " - function-name (optional) if you want to add the layer to the \$latest of a lambda function."
48+
echo ""
49+
echo "Example: buildlayer.sh requirements.txt 10.16.3 mylambda-version-name some-function-name"
50+
echo ""
51+
echo "Exiting."
52+
exit 1
53+
fi
54+
55+
# Clean out virt env
56+
if [ -d "nodejs" ]; then
57+
echo "Removing old nodejs folder."
58+
rm -r nodejs
59+
fi
60+
if [ -d "v-env" ]; then
61+
echo "Removing old v-env folder."
62+
rm -r v-env
63+
fi
64+
65+
dateofbackup=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
66+
# Backup previous layer built
67+
if [ -f layer.zip ]; then
68+
echo "Copy layer.zip to layer-"$dateofbackup".zip"
69+
cp layer.zip layer-backup-$dateofbackup.zip
70+
rm layer.zip
71+
else
72+
echo "No previous layer file, no local backup occurred."
73+
fi
74+
75+
echo "NodeJS virtual environment start:"
76+
# Build for a particular version
77+
# Check versions in our documentation: https://docs.aws.amazon.com/lambda/latest/dg/lambda-runtimes.html
78+
# and node's past versions: https://nodejs.org/en/download/releases/
79+
nodeenv v-env --node="$2" --requirements="$1"
80+
81+
# Activate virtual environment
82+
echo "Activate virtual environment:"
83+
. v-env/bin/activate
84+
85+
# Start check Node version
86+
echo "Node Version:"
87+
node --version
88+
89+
# Install modules per the provided requirements file and create folder structure needed for layer
90+
mkdir -p nodejs/node_modules
91+
cp -R v-env/lib/node_modules/ nodejs/node_modules/
92+
93+
echo "Deactivate virtual enviroment:"
94+
deactivate_node
95+
96+
echo "Zip packages to layer.zip:"
97+
zip -r9qdgds256k layer.zip nodejs/
98+
ls -l layer.zip
99+
100+
# Publish layer if asked to
101+
if [ "$3" != "" ]; then
102+
echo "Adding new layer/version:" $3
103+
arn=$(aws lambda publish-layer-version --layer-name $3 --zip-file fileb://layer.zip | jq '.LayerVersionArn' -r)
104+
echo "Layer pubished: " $arn
105+
# Update function if asked to
106+
if [ "$3" != "" ]; then
107+
echo "Updating Lambda function:" $4
108+
aws lambda update-function-configuration --function-name $4 --layers $arn | jq
109+
else
110+
exit 0
111+
fi
112+
else
113+
exit 0
114+
fi

‎Lambda/BuildLambdaLayers/nodejs/requirements.txt

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
aws-sdk
2+
aws-xray-sdk

‎Lambda/BuildLambdaLayers/python/LICENSE.txt

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
# Copyright 2019-2019 Amazon.com, Inc. or its affiliates. All Rights Reserved.
2+
#
3+
# This file is licensed under the Apache License, Version 2.0 (the "License").
4+
# You may not use this file except in compliance with the License. A copy of the
5+
# License is located at
6+
#
7+
# http://aws.amazon.com/apache2.0/
8+
#
9+
# This file is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS
10+
# OF ANY KIND, either express or implied. See the License for the specific
11+
# language governing permissions and limitations under the License.

‎Lambda/BuildLambdaLayers/python/README.md

+33
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,33 @@
1+
# Lambda - Build Layers Shell Script - Python
2+
3+
This bash script will help with downloading modules and libraries using a python virtual environment, zip and optionally upload the layer as a new version and associate with a Lambda Function if desired.
4+
5+
## Usage
6+
7+
'''bash
8+
./buildlayer.sh requirements.txt [lambda-version-name] [function-name]
9+
'''
10+
11+
- requirements.txt is the list of python packages you wish to add to the layer.
12+
- lambda-version-name (optional) if you want to upload the zip as a new version of an existing layer or as a new layer if no layer yet exists.
13+
- function-name (optional) if you want to add the layer to the $lastest of a lambda function. This updates the Lambda immediately!
14+
15+
This script will use the credentials and region configured with the AWS CLI if available. If not, AWSCLI it will ask for an AWS access key and secret key.
16+
17+
## Requirements
18+
* If you wish the script to upload the Lambda layer and deploy the layer to a Lambda function you will need [jq](https://stedolan.github.io/jq/) and zip.
19+
* python 2.7 will need [virtualenv](https://pip.pypa.io/en/stable/installing/)
20+
* python 3.x will need [venv](https://docs.python.org/3/library/venv.html)
21+
* [pip](https://virtualenv.pypa.io/en/latest/)
22+
* [AWSCLI](https://docs.aws.amazon.com/cli/latest/userguide/install-macos.html) will need to be installed and configured if you wish to upload layers and/or associate layers with Lambda functions.
23+
24+
## Permissions
25+
The following User/Role IAM permissions are required in your AWS Account to upload the layer and associate the layer with a Lambda function:
26+
* Layer Development and Use [https://docs.aws.amazon.com/lambda/latest/dg/access-control-identity-based.html#permissions-user-layer]
27+
* This script does NOT assign resource policy's to the Lambda layer, you may need to consider adding permissions in certain scenarios. [https://docs.aws.amazon.com/lambda/latest/dg/configuration-layers.html#configuration-layers-permissions]
28+
29+
### NOTE: Please test these sample scripts thoroughly to see if they suit your use case.
30+
31+
32+
33+
Work Hard, Have Fun, Make History

‎Lambda/BuildLambdaLayers/python/buildlayer.sh

+105
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,105 @@
1+
#!/bin/bash
2+
set -euo pipefail
3+
# Copyright [2019]-[2019] Amazon.com, Inc. or its affiliates. All Rights Reserved.
4+
# SPDX-License-Identifier: Apache-2.0
5+
#
6+
# Licensed under the Apache License, Version 2.0 (the "License"). You may not use this file
7+
# except in compliance with the License. A copy of the License is located at
8+
#
9+
# http://aws.amazon.com/apache2.0/
10+
#
11+
# or in the "license" file accompanying this file. This file is distributed on an "AS IS"
12+
# BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
13+
# License for the specific language governing permissions and limitations under the License.
14+
#
15+
# --------------------
16+
# Purpose: Build layers to allow for module testing the latest libraries rapidly or for diagnostics such as
17+
# X-Ray. Please test thoroughly if this suits your use-case, and if you have any issues report via git issues
18+
#
19+
# Note: This script can be used for boto and boto3 / py2/3 as needed by changing #'s out as needed below.
20+
#
21+
# Example requirements.txt file:
22+
#
23+
# boto3
24+
# aws-xray-sdk
25+
#
26+
# Example of requirements file specifying versions:
27+
#
28+
# aws-xray-sdk==2.4.2
29+
# boto3==1.9.243
30+
#
31+
# Requirements to allow this script to upload layer and/or update lambda are AWSCLI & jq.
32+
# --------------------
33+
34+
# Check if arg passed on cmd line
35+
if [ -f "$1" ]; then
36+
echo "Starting to build package:"
37+
else
38+
echo "ERR: Please provide the requirements file as a minimum."
39+
echo ""
40+
echo "Usage: buildlayer.sh requirements.txt [lambda-version-name] [function-name]"
41+
echo ""
42+
echo " - requirements.txt is the list of python packages you wish to add to the layer. "
43+
echo " - lambda-version-name (optional) if you want to upload the zip as a new version of an existing layer or as a new layer if no layer yet exists."
44+
echo " - function-name (optional) if you want to add the layer to the \$latest of a lambda function."
45+
echo ""
46+
exit 1
47+
fi
48+
49+
# Clean out virt env
50+
echo "Removing old virtual env/python folder"
51+
if [ -d "python" ]; then
52+
rm -r python
53+
fi
54+
if [ -d "v-env" ]; then
55+
rm -r v-env
56+
fi
57+
58+
dateofbackup=$(date -u +"%Y-%m-%dT%H:%M:%SZ")
59+
# Backup previous layer built
60+
if [ -f layer.zip ]; then
61+
echo "Copy layer.zip to layer-"$dateofbackup".zip"
62+
cp layer.zip layer-backup-$dateofbackup.zip
63+
rm layer.zip
64+
else
65+
echo "No previous layer file, no local backup occurred."
66+
fi
67+
68+
echo "Python virtual environment start:"
69+
# Python3.x
70+
# Alternative is to use virtualenv -p /usr/bin/python3 v-env
71+
python3 -m venv v-env
72+
# Can comment out the line above and remove the # below to allow Python2.7 virtual environments if you have python 2.7 installed
73+
#python2 -m virtualenv v-env
74+
75+
# Activate virtual environment
76+
source v-env/bin/activate
77+
78+
# Install modules per the provided requirements file
79+
echo "Download Python Packages..."
80+
pip install -q --no-cache-dir -r $1 -t python/
81+
82+
echo "End virtual env..."
83+
# Deactive virtual environment
84+
deactivate
85+
86+
echo "Zip packages..."
87+
# Zip the packages up
88+
zip -r9qdgds256k layer.zip python/
89+
ls -l layer.zip
90+
91+
# Publish layer if asked to
92+
if [ "$2" != "" ]; then
93+
echo "Adding new layer or new layer version: "$2
94+
arn=$(aws lambda publish-layer-version --layer-name $2 --zip-file fileb://layer.zip | jq '.LayerVersionArn' -r)
95+
echo "Layer pubished: "$arn
96+
# Update function if asked to
97+
if [ "$3" != "" ]; then
98+
echo "Updating Lambda function: "$3
99+
aws lambda update-function-configuration --function-name $3 --layers $arn | jq
100+
else
101+
exit 0
102+
fi
103+
else
104+
exit 0
105+
fi

‎Lambda/BuildLambdaLayers/python/requirements.txt

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
aws-xray-sdk
2+
boto3

0 commit comments

Comments
 (0)