You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
[unifiedlogs] Add ingest pipeline, ECS field mappings, and custom field definitions (elastic#17644)
* [unifiedlogs] Add ingest pipeline, ECS field mappings, and custom field definitions
Add structured field parsing for the macOS Unified Logs integration.
The raw ndjson output from the log CLI was previously stored as a single
message string with zero field extraction, making structured queries and
detection rules impossible.
Changes:
- Add default ingest pipeline that parses raw ndjson into ECS and custom fields
- Add ECS field definitions (process.*, user.id, dll.*, log.level, event.*)
- Add unified_log.* custom fields (subsystem, category, event_type, etc.)
- Add apple_event.* custom fields for Apple Event enrichment
- Add sample_event.json
- Update docs with pipeline documentation and security monitoring predicates
- Bump version to 0.5.0
Resolveselastic#17643
Made-with: Cursor
* [unifiedlogs] Update changelog with actual PR link
Made-with: Cursor
* [unifiedlogs] Regenerate docs/README.md via elastic-package build
The CI check requires docs/README.md to match the auto-generated output
from the field YAML definitions. Regenerated using elastic-package build.
Made-with: Cursor
* [unifiedlogs] Fix event.category and event.type to use ECS array format
ECS defines event.category and event.type as arrays. Updated the ingest
pipeline Painless script and sample_event.json to use array values.
Static tests now pass.
Made-with: Cursor
* [unifiedlogs] Fix Painless variable scoping error in apple_event_enrichment script
Rename params to paramList to avoid a variable [params] is already
defined compile error. When the YAML folded scalar is delivered to
Painless as a single-line string, block-scoped variables inside if
bodies are treated as top-level declarations and collide.
Made-with: Cursor
* Fix NPE in message_type assignment and split remove processor for empty fields
Add null check for ctx.unified_log before assigning message_type to prevent
NullPointerException when no other unified_log fields were populated earlier
in the pipeline. Split the combined remove processor into separate processors
so boot_uuid and timezone are independently removed only when each is empty.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
---------
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
### Recommended Predicates for Security Monitoring
53
+
54
+
To collect security-relevant events while managing volume, use these predicates:
55
+
56
+
```
57
+
subsystem=="com.apple.appleevents" AND (eventMessage CONTAINS "event={" OR eventMessage CONTAINS "reply={")
58
+
```
59
+
60
+
```
61
+
subsystem=="com.apple.TCC" AND category=="access" AND (eventMessage CONTAINS "AUTHREQ_CTX" OR eventMessage CONTAINS "Denied" OR eventMessage CONTAINS "publishAccessChangedEvent")
62
+
```
63
+
64
+
```
65
+
subsystem=="com.apple.loginwindow.logging" AND eventMessage CONTAINS "performAutolaunch"
66
+
```
67
+
68
+
Enable **debug** level logging in the integration configuration to capture
69
+
Apple Event and TCC debug messages.
70
+
26
71
## Fields Mapping
27
72
28
73
In addition to the fields specified below, this integration includes the ECS Dynamic Template. Any field that follow the ECS Schema will get assigned the correct index field mapping and does not need to be added manually.
### Recommended Predicates for Security Monitoring
53
+
54
+
To collect security-relevant events while managing volume, use these predicates:
55
+
56
+
```
57
+
subsystem=="com.apple.appleevents" AND (eventMessage CONTAINS "event={" OR eventMessage CONTAINS "reply={")
58
+
```
59
+
60
+
```
61
+
subsystem=="com.apple.TCC" AND category=="access" AND (eventMessage CONTAINS "AUTHREQ_CTX" OR eventMessage CONTAINS "Denied" OR eventMessage CONTAINS "publishAccessChangedEvent")
62
+
```
63
+
64
+
```
65
+
subsystem=="com.apple.loginwindow.logging" AND eventMessage CONTAINS "performAutolaunch"
66
+
```
67
+
68
+
Enable **debug** level logging in the integration configuration to capture
69
+
Apple Event and TCC debug messages.
70
+
26
71
## Fields Mapping
27
72
28
73
In addition to the fields specified below, this integration includes the ECS Dynamic Template. Any field that follow the ECS Schema will get assigned the correct index field mapping and does not need to be added manually.
@@ -32,11 +77,55 @@ In addition to the fields specified below, this integration includes the ECS Dyn
32
77
| Field | Description | Type |
33
78
|---|---|---|
34
79
|@timestamp| Event timestamp. | date |
80
+
| apple_event.decoded_payloads | Decoded ASCII text extracted from utxt (Unicode text) hex payloads in the Apple Event. Values are truncated to 16 characters by the Unified Log system. Useful for identifying short indicators like parameter values and script fragments. | keyword |
81
+
| apple_event.direction | The direction of the Apple Event. One of self (sendToSelf), remote (sendToModernProcess), or reply (Reply of SendToSelf or reply=\{\}). | keyword |
82
+
| apple_event.mute | Set to true when an Apple Event set volume command includes mute=true in its payload. Indicates a volume mute operation, which is a known pre-indicator of macOS stealer malware. | boolean |
83
+
| apple_event.parameters | List of four-character parameter codes present in the Apple Event payload (e.g. htxt for hidden text, dtxt for display text, prmp for prompt). These codes identify the types of data carried in the event. | keyword |
84
+
| apple_event.return_id | The return ID of the Apple Event, used to correlate requests with their replies. Extracted from returnID=N in the log message. | keyword |
85
+
| apple_event.target_process | The target process of the Apple Event, extracted from the target='psn' block in the log message. May contain process name, PID, or PSN identifiers. | keyword |
86
+
| apple_event.type_code | The four-character Apple Event class and ID pair (e.g. syso,dlog for display dialog, aevt,stvl for set volume, Jons,gClp for get clipboard). Extracted from the event=\{X,Y\} or reply=\{X,Y\} pattern in the log message. | keyword |
35
87
| cloud.image.id | Image ID for the cloud instance. | keyword |
36
88
| cloud.project.id | Name of the project in Google Cloud. | keyword |
37
89
| data_stream.dataset | Data stream dataset. | constant_keyword |
38
90
| data_stream.namespace | Data stream namespace. | constant_keyword |
39
91
| data_stream.type | Data stream type. | constant_keyword |
92
+
| dll.name | Name of the library. This generally maps to the name of the file on disk. | keyword |
93
+
| dll.path | Full file path of the library. | keyword |
94
+
| ecs.version | ECS version this event conforms to. `ecs.version` is a required field and must exist in all events. When querying across multiple indices -- which may conform to slightly different ECS versions -- this field lets integrations adjust to the schema version of the events. | keyword |
| event.category | This is one of four ECS Categorization Fields, and indicates the second level in the ECS category hierarchy. `event.category` represents the "big buckets" of ECS categories. For example, filtering on `event.category:process` yields all events relating to process activity. This field is closely related to `event.type`, which is used as a subcategory. This field is an array. This will allow proper categorization of some events that fall in multiple categories. | keyword |
97
+
| event.dataset | Name of the dataset. If an event source publishes more than one type of log or events (e.g. access log, error log), the dataset is used to specify which one the event comes from. It's recommended but not required to start the dataset name with the module name, followed by a dot, then the dataset name. | keyword |
98
+
| event.kind | This is one of four ECS Categorization Fields, and indicates the highest level in the ECS category hierarchy. `event.kind` gives high-level information about what type of information the event contains, without being specific to the contents of the event. For example, values of this field distinguish alert events from metric events. The value of this field can be used to inform how these kinds of events should be handled. They may warrant different retention, different access control, it may also help understand whether the data is coming in at a regular interval or not. | keyword |
99
+
| event.module | Name of the module this data is coming from. If your monitoring agent supports the concept of modules or plugins to process events of a given source (e.g. Apache logs), `event.module` should contain the name of this module. | keyword |
100
+
| event.provider | Source of the event. Event transports such as Syslog or the Windows Event Log typically mention the source of an event. It can be the name of the software that generated the event (e.g. Sysmon, httpd), or of a subsystem of the operating system (kernel, Microsoft-Windows-Security-Auditing). | keyword |
101
+
| event.type | This is one of four ECS Categorization Fields, and indicates the third level in the ECS category hierarchy. `event.type` represents a categorization "sub-bucket" that, when used along with the `event.category` field values, enables filtering events down to a level appropriate for single visualization. This field is an array. This will allow proper categorization of some events that fall in multiple event types. | keyword |
40
102
| host.containerized | If the host is a container. | boolean |
41
103
| host.os.build | OS build information. | keyword |
42
104
| host.os.codename | OS codename, if any. | keyword |
105
+
| host.os.family | OS family (such as redhat, debian, freebsd, windows). | keyword |
106
+
| host.os.type | Use the `os.type` field to categorize the operating system into one of the broad commercial families. If the OS you're dealing with is not listed as an expected value, the field should not be populated. Please let us know by opening an issue with ECS, to propose its addition. | keyword |
107
+
| log.level | Original log level of the log event. If the source of the event provides a log level or textual severity, this is the one that goes in `log.level`. If your source doesn't specify one, you may put your event transport's severity here (e.g. Syslog severity). Some examples are `warn`, `err`, `i`, `informational`. | keyword |
108
+
| message | For log events the message field contains the log message, optimized for viewing in a log viewer. For structured logs without an original message field, other fields can be concatenated to form a human-readable summary of the event. If multiple messages exist, they can be combined into one message. | match_only_text |
109
+
| process.executable | Absolute path to the process executable. | keyword |
110
+
| process.executable.text | Multi-field of `process.executable`. | match_only_text |
111
+
| process.name | Process name. Sometimes called program name or similar. | keyword |
112
+
| process.name.text | Multi-field of `process.name`. | match_only_text |
113
+
| process.pid | Process id. | long |
114
+
| process.thread.id | Thread ID. | long |
115
+
| tags | List of keywords used to tag each event. | keyword |
116
+
| unified_log.activity_id | The activity identifier for correlating related log entries. Corresponds to the ndjson `activityIdentifier` field. | long |
117
+
| unified_log.boot_uuid | The boot UUID identifying the system boot session. Corresponds to the ndjson `bootUUID` field. | keyword |
118
+
| unified_log.category | The category within the subsystem (e.g. access, connection). Corresponds to the ndjson `category` field. | keyword |
119
+
| unified_log.event_type | The Unified Log event type (e.g. logEvent, activityCreateEvent, traceEvent). Corresponds to the ndjson `eventType` field. | keyword |
120
+
| unified_log.format_string | The os_log format string used to compose the message. Useful for grouping events by log template. Corresponds to the ndjson `formatString` field. | keyword |
121
+
| unified_log.mach_timestamp | The Mach absolute time timestamp. Corresponds to the ndjson `machTimestamp` field. | long |
122
+
| unified_log.message_type | The original message severity before normalization to log.level. One of Default, Info, Debug, Error, or Fault. Corresponds to the ndjson `messageType` field. | keyword |
123
+
| unified_log.parent_activity_id | The parent activity identifier. Corresponds to the ndjson `parentActivityIdentifier` field. | long |
124
+
| unified_log.process.uuid | The UUID of the process image (Mach-O binary). Corresponds to the ndjson `processImageUUID` field. | keyword |
125
+
| unified_log.sender.program_counter | The program counter value of the sender at the time the log entry was emitted. Corresponds to the ndjson `senderProgramCounter` field. | long |
126
+
| unified_log.sender.uuid | The UUID of the sender image (shared library or framework). Corresponds to the ndjson `senderImageUUID` field. | keyword |
127
+
| unified_log.subsystem | The subsystem that emitted the log entry (e.g. com.apple.appleevents, com.apple.TCC). Corresponds to the ndjson `subsystem` field. | keyword |
128
+
| unified_log.timezone | The timezone name where the log was recorded. Corresponds to the ndjson `timezoneName` field. | keyword |
129
+
| unified_log.trace_id | The trace identifier for correlating log entries across processes. Stored as keyword because the value can exceed the long integer range. Corresponds to the ndjson `traceID` field. | keyword |
130
+
| user.id | Unique identifier of the user. | keyword |
0 commit comments