Skip to content

Commit 7b9e362

Browse files
baolsenbaolsen
authored
feat: Ability to send ClickOps to Firehose (#45)
1 parent 5bb738c commit 7b9e362

File tree

9 files changed

+626
-27
lines changed

9 files changed

+626
-27
lines changed

‎README.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,7 @@ Full contributing [guidelines are covered here](.github/contributing.md).
4545
| <a name="input_excluded_scoped_actions"></a> [excluded\_scoped\_actions](#input\_excluded\_scoped\_actions) | A list of service scoped actions that will not be alerted on. Format {{service}}.amazonaws.com:{{action}} | `list(string)` | `[]` | no |
4646
| <a name="input_excluded_scoped_actions_effect"></a> [excluded\_scoped\_actions\_effect](#input\_excluded\_scoped\_actions\_effect) | Should the existing exluded actions be replaces or appended to. By default it will append to the list, valid values: APPEND, REPLACE | `string` | `"APPEND"` | no |
4747
| <a name="input_excluded_users"></a> [excluded\_users](#input\_excluded\_users) | List of email addresses will not be reported on when practicing ClickOps. | `list(string)` | `[]` | no |
48+
| <a name="input_firehose_delivery_stream_name"></a> [firehose\_delivery\_stream\_name](#input\_firehose\_delivery\_stream\_name) | Kinesis Firehose delivery stream name to output ClickOps events to. | `string` | `null` | no |
4849
| <a name="input_iam_role_arn"></a> [iam\_role\_arn](#input\_iam\_role\_arn) | Existing IAM role ARN for the lambda. Required if `create_iam_role` is set to `false` | `string` | `null` | no |
4950
| <a name="input_included_accounts"></a> [included\_accounts](#input\_included\_accounts) | List of accounts that be scanned to manual actions. If empty will scan all accounts. | `list(string)` | `[]` | no |
5051
| <a name="input_included_users"></a> [included\_users](#input\_included\_users) | List of emails that be scanned to manual actions. If empty will scan all emails. | `list(string)` | `[]` | no |

‎examples/delivery_stream/README.md‎

Lines changed: 277 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,277 @@
1+
<!-- BEGIN_TF_DOCS -->
2+
----
3+
## main.tf
4+
```hcl
5+
terraform {
6+
required_version = ">= 0.14.0"
7+
8+
required_providers {
9+
aws = {
10+
source = "hashicorp/aws"
11+
version = "4.9.0"
12+
}
13+
14+
random = {
15+
source = "hashicorp/random"
16+
version = "3.4.3"
17+
}
18+
}
19+
}
20+
21+
provider "aws" {
22+
region = "eu-west-1"
23+
}
24+
25+
locals {
26+
tags = {
27+
usage = "clickops-testing"
28+
run = random_pet.run_id.id
29+
}
30+
31+
naming_prefix = "clickops-test-basic-${random_pet.run_id.id}"
32+
}
33+
34+
resource "random_pet" "run_id" {
35+
keepers = {
36+
# Generate a new pet name
37+
run_id = var.run_id
38+
}
39+
}
40+
41+
#---------------------------------------
42+
# Cloudtrail infrastructure - standalone
43+
#---------------------------------------
44+
# S3 bucket
45+
module "logs_bucket" {
46+
source = "trussworks/logs/aws"
47+
version = "~> 14"
48+
49+
s3_bucket_name = local.naming_prefix
50+
51+
allow_cloudtrail = true
52+
force_destroy = true
53+
}
54+
55+
# Cloudtrail
56+
locals {
57+
naming_prefix_cloudtrail = "${local.naming_prefix}-cloudtrail"
58+
naming_prefix_firehose = "${local.naming_prefix}-firehose"
59+
}
60+
module "aws_cloudtrail" {
61+
source = "trussworks/cloudtrail/aws"
62+
version = "~> 4"
63+
64+
s3_bucket_name = module.logs_bucket.aws_logs_bucket
65+
66+
trail_name = local.naming_prefix_cloudtrail
67+
iam_policy_name = local.naming_prefix_cloudtrail
68+
iam_role_name = local.naming_prefix_cloudtrail
69+
70+
cloudwatch_log_group_name = local.naming_prefix_cloudtrail
71+
log_retention_days = 30
72+
}
73+
74+
#---------------------------------------
75+
# ClickOps module
76+
#---------------------------------------
77+
module "clickops_notifications" {
78+
source = "../.."
79+
80+
standalone = true
81+
82+
naming_prefix = local.naming_prefix
83+
84+
webhook = "https://fake.com"
85+
message_format = "slack"
86+
87+
tags = local.tags
88+
89+
# cloudtrail_bucket_name = aws_s3_bucket.clickops_cloudtrail.id
90+
cloudtrail_log_group = local.naming_prefix_cloudtrail
91+
92+
firehose_delivery_stream_name = aws_kinesis_firehose_delivery_stream.extended_s3_stream.name
93+
94+
depends_on = [
95+
module.aws_cloudtrail
96+
]
97+
98+
}
99+
100+
#---------------------------------------
101+
# Delivery stream infrastructure
102+
#---------------------------------------
103+
resource "aws_kinesis_firehose_delivery_stream" "extended_s3_stream" {
104+
name = local.naming_prefix
105+
destination = "extended_s3"
106+
107+
extended_s3_configuration {
108+
role_arn = aws_iam_role.firehose.arn
109+
bucket_arn = aws_s3_bucket.firehose.arn
110+
111+
compression_format = "UNCOMPRESSED"
112+
113+
buffer_size = 64
114+
buffer_interval = 300
115+
116+
# Hive-style dynamic partitioning by recipientAccountId and awsRegion
117+
# https://docs.aws.amazon.com/firehose/latest/dev/dynamic-partitioning.html
118+
119+
dynamic_partitioning_configuration {
120+
enabled = "true"
121+
}
122+
123+
prefix = join("/", [
124+
"data",
125+
"recipientAccountId=!{partitionKeyFromQuery:recipientAccountId}",
126+
"awsRegion=!{partitionKeyFromQuery:awsRegion}",
127+
"year=!{timestamp:yyyy}", "month=!{timestamp:MM}", "day=!{timestamp:dd}", "hour=!{timestamp:HH}",
128+
""
129+
])
130+
error_output_prefix = join("/", [
131+
"errors",
132+
"year=!{timestamp:yyyy}", "month=!{timestamp:MM}", "day=!{timestamp:dd}", "hour=!{timestamp:HH}",
133+
"!{firehose:error-output-type}",
134+
""
135+
])
136+
137+
processing_configuration {
138+
enabled = "true"
139+
140+
# DeAggreagate records
141+
processors {
142+
type = "RecordDeAggregation"
143+
parameters {
144+
parameter_name = "SubRecordType"
145+
parameter_value = "JSON"
146+
}
147+
}
148+
149+
# Calculate partition variables
150+
processors {
151+
type = "MetadataExtraction"
152+
parameters {
153+
parameter_name = "JsonParsingEngine"
154+
parameter_value = "JQ-1.6"
155+
}
156+
parameters {
157+
parameter_name = "MetadataExtractionQuery"
158+
parameter_value = "{recipientAccountId:.recipientAccountId, awsRegion:.awsRegion}"
159+
}
160+
}
161+
162+
# Append new line between output records
163+
processors {
164+
type = "AppendDelimiterToRecord"
165+
}
166+
}
167+
}
168+
}
169+
170+
resource "aws_s3_bucket" "firehose" {
171+
bucket = local.naming_prefix_firehose
172+
}
173+
174+
resource "aws_s3_bucket_acl" "firehose_bucket_acl" {
175+
bucket = aws_s3_bucket.firehose.id
176+
acl = "private"
177+
}
178+
179+
resource "aws_iam_role" "firehose" {
180+
name = local.naming_prefix_firehose
181+
182+
assume_role_policy = jsonencode({
183+
Version = "2012-10-17"
184+
Statement = [
185+
{
186+
Action = "sts:AssumeRole"
187+
Effect = "Allow"
188+
Sid = ""
189+
Principal = {
190+
Service = "firehose.amazonaws.com"
191+
}
192+
}
193+
]
194+
})
195+
}
196+
197+
resource "aws_iam_role_policy" "firehose" {
198+
name = local.naming_prefix_firehose
199+
role = aws_iam_role.firehose.id
200+
policy = data.aws_iam_policy_document.firehose.json
201+
}
202+
203+
data "aws_iam_policy_document" "firehose" {
204+
statement {
205+
actions = [
206+
"s3:AbortMultipartUpload",
207+
"s3:GetBucketLocation",
208+
"s3:GetObject",
209+
"s3:ListBucket",
210+
"s3:ListBucketMultipartUploads",
211+
"s3:PutObject"
212+
]
213+
214+
resources = [
215+
aws_s3_bucket.firehose.arn,
216+
"${aws_s3_bucket.firehose.arn}/*"
217+
]
218+
}
219+
}
220+
```
221+
----
222+
223+
## Documentation
224+
225+
----
226+
### Inputs
227+
228+
| Name | Description | Type | Default | Required |
229+
|------|-------------|------|---------|:--------:|
230+
| <a name="input_run_id"></a> [run\_id](#input\_run\_id) | Used to ensure resources are unique | `string` | n/a | yes |
231+
232+
----
233+
### Modules
234+
235+
| Name | Source | Version |
236+
|------|--------|---------|
237+
| <a name="module_aws_cloudtrail"></a> [aws\_cloudtrail](#module\_aws\_cloudtrail) | trussworks/cloudtrail/aws | ~> 4 |
238+
| <a name="module_clickops_notifications"></a> [clickops\_notifications](#module\_clickops\_notifications) | ../.. | n/a |
239+
| <a name="module_logs_bucket"></a> [logs\_bucket](#module\_logs\_bucket) | trussworks/logs/aws | ~> 14 |
240+
241+
----
242+
### Outputs
243+
244+
No outputs.
245+
246+
----
247+
### Providers
248+
249+
| Name | Version |
250+
|------|---------|
251+
| <a name="provider_aws"></a> [aws](#provider\_aws) | 4.9.0 |
252+
| <a name="provider_random"></a> [random](#provider\_random) | 3.4.3 |
253+
254+
----
255+
### Requirements
256+
257+
| Name | Version |
258+
|------|---------|
259+
| <a name="requirement_terraform"></a> [terraform](#requirement\_terraform) | >= 0.14.0 |
260+
| <a name="requirement_aws"></a> [aws](#requirement\_aws) | 4.9.0 |
261+
| <a name="requirement_random"></a> [random](#requirement\_random) | 3.4.3 |
262+
263+
----
264+
### Resources
265+
266+
| Name | Type |
267+
|------|------|
268+
| [aws_iam_role.firehose](https://registry.terraform.io/providers/hashicorp/aws/4.9.0/docs/resources/iam_role) | resource |
269+
| [aws_iam_role_policy.firehose](https://registry.terraform.io/providers/hashicorp/aws/4.9.0/docs/resources/iam_role_policy) | resource |
270+
| [aws_kinesis_firehose_delivery_stream.extended_s3_stream](https://registry.terraform.io/providers/hashicorp/aws/4.9.0/docs/resources/kinesis_firehose_delivery_stream) | resource |
271+
| [aws_s3_bucket.firehose](https://registry.terraform.io/providers/hashicorp/aws/4.9.0/docs/resources/s3_bucket) | resource |
272+
| [aws_s3_bucket_acl.firehose_bucket_acl](https://registry.terraform.io/providers/hashicorp/aws/4.9.0/docs/resources/s3_bucket_acl) | resource |
273+
| [random_pet.run_id](https://registry.terraform.io/providers/hashicorp/random/3.4.3/docs/resources/pet) | resource |
274+
| [aws_iam_policy_document.firehose](https://registry.terraform.io/providers/hashicorp/aws/4.9.0/docs/data-sources/iam_policy_document) | data source |
275+
276+
----
277+
<!-- END_TF_DOCS -->

0 commit comments

Comments
 (0)