Skip to content

Commit 0e12d94

Browse files
baolsenbaolsen
authored
feat!: Lambda deployment options. Use pre-built package by default. And deprecate python3.6. (#46)
* feat: Deployment options. Use pre-built package by default. * chore!: Deprecate python3.6 runtime
1 parent a76aafb commit 0e12d94

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

47 files changed

+472
-153
lines changed

‎.gitattributes‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
######################################
2+
### NORMALIZATION (convert crlf => lf)
3+
# https://adaptivepatchwork.com/2012/03/01/mind-the-end-of-your-line/
4+
######################################
5+
#
6+
# NB: WINDOWS USERS, please set:
7+
# git config --global core.autocrlf input
8+
#
9+
# Default is to normalize all text files
10+
* text=auto
11+
12+
# These files are explicitly text files and should be normalized
13+
*.tf text=auto
14+
*.tfvars text=auto
15+
*.yml text=auto
16+
17+
# These files are explicitly text files and should not be modified
18+
19+
# These files are explicitly binary
20+
21+
######################################

‎.github/.pre-commit-config.yaml‎ renamed to ‎.github/DISABLED.pre-commit-config.yaml‎

File renamed without changes.

‎.github/workflows/check-deployments.yml‎

Lines changed: 71 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,71 @@
1+
name: Check deployments
2+
3+
on:
4+
workflow_dispatch:
5+
pull_request:
6+
branches:
7+
- main
8+
- master
9+
- develop
10+
11+
defaults:
12+
# Set shell for steps inside containers (default=sh)
13+
run:
14+
shell: bash
15+
16+
jobs:
17+
#--------------------------------------------------------------
18+
# COLLECT DEPLOYMENTS
19+
#--------------------------------------------------------------
20+
collect:
21+
name: 🍱 collect deployments
22+
# For public repos use runs-on: ubuntu-latest
23+
# For private repos use runs-on: self-hosted
24+
runs-on: ${{ vars.RUNS_ON }}
25+
outputs:
26+
matrix: ${{ steps.set-matrix.outputs.matrix }}
27+
steps:
28+
- uses: actions/checkout@v3
29+
- id: set-matrix
30+
run: |
31+
MATRIX=$(cat build_targets.json | jq -c)
32+
echo $MATRIX
33+
cat <<< matrix=$MATRIX >> $GITHUB_OUTPUT
34+
cat $GITHUB_OUTPUT
35+
36+
#--------------------------------------------------------------
37+
# CHECK DEPLOYMENTS
38+
#--------------------------------------------------------------
39+
check-deployments:
40+
needs: [collect]
41+
name: 🔍 check deployment
42+
# For public repos use runs-on: ubuntu-latest
43+
# For private repos use runs-on: self-hosted
44+
runs-on: ${{ vars.RUNS_ON }}
45+
strategy:
46+
matrix: ${{ fromJson(needs.collect.outputs.matrix) }}
47+
steps:
48+
- uses: actions/checkout@v3
49+
50+
# To test whether the deployment package is up to date,
51+
# we only check the application code (excluding dependencies).
52+
- run: |
53+
NAME=${{ matrix.name }}
54+
RUNTIME=${{ matrix.runtime }}
55+
DEPLOYMENT=deployment-$NAME-$RUNTIME.zip
56+
#
57+
echo NAME=$NAME RUNTIME=$RUNTIME
58+
#
59+
cp $DEPLOYMENT ${DEPLOYMENT}_original
60+
cd $NAME
61+
zip -r ../$DEPLOYMENT .
62+
cd ..
63+
#
64+
# Compare length and names of files in Zip.
65+
# Building in Docker doesn't work, some files are still different.
66+
[[ -f $DEPLOYMENT ]] || { echo "Deployment file not found."; exit 1; }
67+
diff \
68+
<(unzip -vqq $DEPLOYMENT | awk '{$2=""; $3=""; $4=""; $5=""; $6=""; print}' | sort -k3 -f) \
69+
<(unzip -vqq ${DEPLOYMENT}_original | awk '{$2=""; $3=""; $4=""; $5=""; $6=""; print}' | sort -k3 -f)
70+
FILES_CHANGED=$?
71+
echo FILES_CHANGED=$FILES_CHANGED

‎.github/workflows/pre-commit-and-tests.yml‎

Lines changed: 14 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -29,8 +29,8 @@ jobs:
2929
name: 💍 pre-commit
3030
# For public repos use runs-on: ubuntu-latest
3131
# For private repos use runs-on: self-hosted
32-
runs-on: ubuntu-latest
33-
container: bjorncloudandthings/terraform-aws-github:latest
32+
runs-on: ${{ vars.RUNS_ON }}
33+
container: ${{ vars.CONTAINER }}
3434
steps:
3535
- uses: actions/checkout@v3
3636
- uses: actions/setup-python@v4
@@ -41,11 +41,7 @@ jobs:
4141
run: |
4242
python -m venv .venv
4343
source .venv/bin/activate
44-
if [ -f requirements.txt ]; then
45-
pip install -r requirements.txt
46-
else
47-
pip install pre-commit
48-
fi
44+
pip install pre-commit
4945
- name: pre-commit run
5046
run: |
5147
source .venv/bin/activate
@@ -54,12 +50,16 @@ jobs:
5450
fi
5551
echo ...running developer pre-commit checks...
5652
pre-commit run --config .pre-commit-config.yaml --all-files --show-diff-on-failure
57-
# tfsec and checkov are temporarily disabled.
58-
# echo ...running remote pre-commit checks...
59-
# pre-commit run --config .github/.pre-commit-config.yaml --all-files --show-diff-on-failure
53+
if [[ -f .github/.pre-commit-config.yaml ]]; then
54+
echo ...running remote pre-commit checks...
55+
pip install checkov
56+
pre-commit run --config .github/.pre-commit-config.yaml --all-files --show-diff-on-failure
57+
fi
6058
- name: cat pre-commit log
6159
if: failure()
62-
run: cat ~/.cache/pre-commit/pre-commit.log
60+
run: |
61+
LOG_FILE=~/.cache/pre-commit/pre-commit.log
62+
[ -f $LOG_FILE ] && cat $LOG_FILE
6363
6464
#--------------------------------------------------------------
6565
# TESTS
@@ -69,8 +69,8 @@ jobs:
6969
name: ✅ tests
7070
# For public repos use runs-on: ubuntu-latest
7171
# For private repos use runs-on: self-hosted
72-
runs-on: ubuntu-latest
73-
container: bjorncloudandthings/terraform-aws-github:latest
72+
runs-on: ${{ vars.RUNS_ON }}
73+
container: ${{ vars.CONTAINER }}
7474
steps:
7575
- uses: actions/checkout@v3
7676
- uses: actions/setup-python@v4
@@ -88,7 +88,7 @@ jobs:
8888
run: |
8989
python -m venv .venv
9090
source .venv/bin/activate
91-
pip install -r requirements.txt
91+
pip install -r tests/requirements.txt
9292
- name: Run
9393
run: |
9494
source .venv/bin/activate

‎.github/workflows/release-please.yml‎

Lines changed: 0 additions & 45 deletions
This file was deleted.

‎.github/workflows/release.yml‎

Lines changed: 68 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,68 @@
1+
name: release
2+
3+
# Overview ref: https://github.com/googleapis/release-please
4+
# Configuration ref: https://github.com/google-github-actions/release-please-action
5+
6+
on:
7+
workflow_dispatch:
8+
push:
9+
branches:
10+
- main
11+
- master
12+
- develop
13+
14+
defaults:
15+
# Set shell for steps inside containers (default=sh)
16+
run:
17+
shell: bash
18+
19+
jobs:
20+
release:
21+
# For public repos use runs-on: ubuntu-latest
22+
# For private repos use runs-on: self-hosted
23+
runs-on: ${{ vars.RUNS_ON }}
24+
outputs:
25+
release_created: ${{ steps.release-please.outputs.release_created }}
26+
tag_name: ${{ steps.release-please.outputs.tag_name }}
27+
steps:
28+
- name: release-please
29+
uses: google-github-actions/release-please-action@v3
30+
id: release-please
31+
with:
32+
release-type: simple
33+
# These bumps are honoured only if there is an
34+
# initial tag of v0.1.0 . Create it manually if needed.
35+
#
36+
# BREAKING CHANGE only bumps semver minor if version < 1.0.0
37+
bump-minor-pre-major: true
38+
# feat commits bump semver patch instead of minor if version < 1.0.0
39+
bump-patch-for-minor-pre-major: true
40+
41+
release-assets:
42+
needs: [release]
43+
if: needs.release.outputs.release_created || github.event_name == 'workflow_dispatch'
44+
runs-on: ${{ vars.RUNS_ON }}
45+
container: ${{ vars.CONTAINER }}
46+
steps:
47+
- uses: actions/checkout@v3
48+
with:
49+
fetch-depth: 0
50+
#--------------------------------------------
51+
# Upload deployment as a release asset
52+
#--------------------------------------------
53+
- name: ⬆️ upload release assets
54+
env:
55+
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
56+
run: |
57+
TAG_NAME=
58+
if [[ ${{ github.event_name == 'workflow_dispatch' }} == true ]]; then
59+
# Get max tag matching: vX.Y.Z
60+
TAG_NAME=$(git tag | grep '^v[0-9.]\+$' | sort | tail -n1)
61+
else
62+
TAG_NAME=${{ needs.release.outputs.tag_name }}
63+
fi
64+
echo TAG_NAME=$TAG_NAME
65+
[[ -z "$TAG_NAME" ]] && { echo "TAG_NAME is empty" ; exit 1; }
66+
#
67+
gh config set prompt disabled
68+
gh release upload $TAG_NAME deployment-*.zip

‎.gitignore‎

Lines changed: 11 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -33,9 +33,6 @@ override.tf.json
3333
.terraform.lock.hcl
3434
lambda.zip
3535
terraform.tfvars
36-
.env
37-
.idea
38-
__pycache__
3936

4037
# Testing artefacts
4138
tf_resources.json
@@ -44,8 +41,18 @@ test-rsa.*
4441
*.tar.gz
4542
test.auto.tfvars*
4643

47-
# VSCode
44+
# IDEs
4845
*.code-workspace
46+
.env
47+
.idea
4948

5049
# Apple
5150
.DS_Store
51+
52+
# Lambda packaging
53+
.package
54+
55+
# Python
56+
__pycache__
57+
.venv
58+
.pytest_cache

‎.tfsec-config.yml‎

Lines changed: 4 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -14,5 +14,7 @@
1414

1515
---
1616
exclude:
17-
# Add IDs of checks to ignore them
18-
# - aws-sns-topic-encryption-use-cmk
17+
# Add IDs of checks to ignore them
18+
# - aws-sns-topic-encryption-use-cmk
19+
# Without full tracing enabled it is difficult to trace the flow of logs
20+
- aws-lambda-enable-tracing

‎README.md‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -50,21 +50,24 @@ Full contributing [guidelines are covered here](.github/contributing.md).
5050
| <a name="input_included_accounts"></a> [included\_accounts](#input\_included\_accounts) | List of accounts that be scanned to manual actions. If empty will scan all accounts. | `list(string)` | `[]` | no |
5151
| <a name="input_included_users"></a> [included\_users](#input\_included\_users) | List of emails that be scanned to manual actions. If empty will scan all emails. | `list(string)` | `[]` | no |
5252
| <a name="input_lambda_memory_size"></a> [lambda\_memory\_size](#input\_lambda\_memory\_size) | The amount of memory for Lambda to use | `number` | `"128"` | no |
53-
| <a name="input_lambda_runtime"></a> [lambda\_runtime](#input\_lambda\_runtime) | The lambda runtime to use | `string` | `"python3.8"` | no |
53+
| <a name="input_lambda_runtime"></a> [lambda\_runtime](#input\_lambda\_runtime) | The lambda runtime to use. One of: `["python3.9", "python3.8", "python3.7"]` | `string` | `"python3.8"` | no |
5454
| <a name="input_log_retention_in_days"></a> [log\_retention\_in\_days](#input\_log\_retention\_in\_days) | Number of days to keep CloudWatch logs | `number` | `14` | no |
5555
| <a name="input_message_format"></a> [message\_format](#input\_message\_format) | Where do you want to send this message? slack or msteams | `string` | `"slack"` | no |
5656
| <a name="input_naming_prefix"></a> [naming\_prefix](#input\_naming\_prefix) | Resources will be prefixed with this | `string` | `"clickops-notifier"` | no |
57+
| <a name="input_s3_bucket"></a> [s3\_bucket](#input\_s3\_bucket) | S3 bucket for deployment package. | `string` | `null` | no |
58+
| <a name="input_s3_key"></a> [s3\_key](#input\_s3\_key) | S3 object key for deployment package. Otherwise, defaults to `var.naming_prefix/local.deployment_filename`. | `string` | `null` | no |
5759
| <a name="input_standalone"></a> [standalone](#input\_standalone) | Deploy ClickOps in a standalone account instead of into an entire AWS Organization. Ideal for teams who want to monitor ClickOps in only their accounts where it is not instrumented at an Organizational level. | `bool` | `false` | no |
5860
| <a name="input_subcription_filter_distribution"></a> [subcription\_filter\_distribution](#input\_subcription\_filter\_distribution) | The method used to distribute log data to the destination. By default log data is grouped by log stream, but the grouping can be set to random for a more even distribution. This property is only applicable when the destination is an Amazon Kinesis stream. Valid values are "Random" and "ByLogStream". | `string` | `"Random"` | no |
5961
| <a name="input_tags"></a> [tags](#input\_tags) | Tags to add to resources in addition to the default\_tags for the provider | `map(string)` | `{}` | no |
62+
| <a name="input_upload_deployment_to_s3"></a> [upload\_deployment\_to\_s3](#input\_upload\_deployment\_to\_s3) | If `true`, the deployment package within this module repo will be copied to S3. If `false` then the S3 object must be uploaded separately. Ignored if `s3_bucket` is null. | `bool` | `true` | no |
6063
| <a name="input_webhook"></a> [webhook](#input\_webhook) | The webhook URL for notifications. https://api.slack.com/messaging/webhooks | `string` | n/a | yes |
6164

6265
----
6366
### Modules
6467

6568
| Name | Source | Version |
6669
|------|--------|---------|
67-
| <a name="module_clickops_notifier_lambda"></a> [clickops\_notifier\_lambda](#module\_clickops\_notifier\_lambda) | terraform-aws-modules/lambda/aws | 3.2.1 |
70+
| <a name="module_clickops_notifier_lambda"></a> [clickops\_notifier\_lambda](#module\_clickops\_notifier\_lambda) | terraform-aws-modules/lambda/aws | 4.9.0 |
6871

6972
----
7073
### Outputs
@@ -96,6 +99,7 @@ Full contributing [guidelines are covered here](.github/contributing.md).
9699
|------|------|
97100
| [aws_cloudwatch_log_subscription_filter.this](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/cloudwatch_log_subscription_filter) | resource |
98101
| [aws_s3_bucket_notification.bucket_notification](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_bucket_notification) | resource |
102+
| [aws_s3_object.deployment](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/s3_object) | resource |
99103
| [aws_sqs_queue.bucket_notifications](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue) | resource |
100104
| [aws_sqs_queue_policy.bucket_notifications](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/sqs_queue_policy) | resource |
101105
| [aws_ssm_parameter.slack_webhook](https://registry.terraform.io/providers/hashicorp/aws/latest/docs/resources/ssm_parameter) | resource |

‎build.sh‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
#!/bin/bash
2+
3+
set -e
4+
5+
TARGETS=$(jq -c '.include[]' build_targets.json)
6+
7+
while IFS= read -r line; do
8+
NAME=$(echo "$line" | jq -r '.name')
9+
RUNTIME=$(echo "$line" | jq -r '.runtime')
10+
bash ./build_one.sh $NAME $RUNTIME
11+
done <<< "$TARGETS"

0 commit comments

Comments
 (0)