Skip to content

Commit 25eeccd

Browse files
committed
docs: reflect enrichment, TLS, and detached sources in architecture
The README ASCII diagram and assets/architecture.mmd predated three unreleased features already documented in CHANGELOG.md: post-evaluation enrichment (#134), server-side TLS for the daemon API listener (#128), and detached dynamic sources (#135). Update both surfaces so they match what the next release ships. README prose and ASCII diagram: - Add an Enrichment bullet to the streaming-runtime section listing the four primitives (template, lookup, http, command), the kind-aware ${detection.*} / ${correlation.*} namespaces, scope filter, HTTP response cache, and the on_error policies. - Insert an enrichment/ block in the rsigma-runtime ASCII diagram between RuntimeEngine and io/, and route the flow through it. - Mention TLS termination (mTLS, cert hot-reload) on the shared API listener in the io/ block. - Mention DaemonSourceRegistry (external --source + pipeline-embedded deprecated, collision-error) in the sources/ block. - Note the enrichments map in the EvaluationResult output box and the Output bullet of the prose. Mermaid diagram (assets/architecture.mmd): - Add 'enrichments' to the RuleHeader line in the OUTPUT node. - Add a new RENRICH node and wire RENG --> RENRICH --> RIO. - Mention TLS termination in the RIO node. - Mention DaemonSourceRegistry, --source, and the collision-error semantics in the RSRC node. Verified with mmdc: assets/architecture.mmd renders cleanly.
1 parent bcecc09 commit 25eeccd

2 files changed

Lines changed: 26 additions & 7 deletions

File tree

README.md

Lines changed: 22 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -402,7 +402,8 @@ When running as a streaming detection engine, `rsigma-eval` feeds into `rsigma-r
402402
- **Input:** Format adapters parse raw log lines (JSON, syslog, logfmt\*, CEF\*, plain text, with auto-detection) into `EventInputDecoded`. EVTX\* files are parsed directly from binary via `EvtxFileReader`. Sources include stdin, HTTP POST, NATS JetStream, and OTLP\* (HTTP protobuf/JSON and gRPC).
403403
- **Dynamic sources:** `SourceResolver` fetches data from files, commands, HTTP APIs, and NATS subjects. Resolved values are injected into pipelines via `TemplateExpander`. A `SourceCache` (in-memory + optional SQLite) provides fallback data. `RefreshScheduler` manages auto-refresh (interval, file watch, NATS push, on-demand). Extraction supports jq, JSONPath, and CEL. `DaemonSourceRegistry` unifies sources from external files (`--source`) and pipeline-embedded declarations with collision-error semantics.
404404
- **Processing:** `LogProcessor` runs batch evaluation with parallel detection and sequential correlation. `RuntimeEngine` wraps `Engine` and `CorrelationEngine` with rule loading and `ArcSwap` hot-reload.
405-
- **Output:** Sinks write evaluation results to stdout, files, or NATS as one flat JSON object per result. Multiple sinks can run in fan-out. The output type is `EvaluationResult` (a composition of `RuleHeader` + `ResultBody::Detection|Correlation`), carrying rule title, id, level, tags, matched selections, field matches, aggregated values, and optionally the triggering events.
405+
- **Enrichment:** `EnrichmentPipeline` runs between the engine and the sinks, injecting context (asset info, IP reputation, identity, GeoIP, KEV flags, runbook URLs, ...) into each result's `RuleHeader.enrichments` map. Four primitives (`template`, `lookup`, `http`, `command`) compose into recipes. Kind-aware template namespaces (`${detection.*}` for detection-kind enrichers, `${correlation.*}` for correlation-kind) are validated at config-load time. Optional HTTP response cache, scope filtering by rule glob, tag set, and severity, and `on_error` policies (`skip`, `null`, `drop`).
406+
- **Output:** Sinks write evaluation results to stdout, files, or NATS as one flat JSON object per result. Multiple sinks can run in fan-out. The output type is `EvaluationResult` (a composition of `RuleHeader` + `ResultBody::Detection|Correlation`), carrying rule title, id, level, tags, the `enrichments` map written by the enrichment pipeline, matched selections, field matches, aggregated values, and optionally the triggering events.
406407

407408
Feature-gated items are marked with \* in the diagram.
408409

@@ -485,6 +486,9 @@ Feature-gated items are marked with \* in the diagram.
485486
│ ↓ raw line → EventInputDecoded │
486487
│ │
487488
│ sources/ ──> dynamic pipelines: │
489+
│ DaemonSourceRegistry: external │
490+
│ (--source) + pipeline-embedded │
491+
│ (deprecated), collision-error │
488492
│ SourceResolver (HTTP, command, │
489493
│ file, NATS subjects) │
490494
│ TemplateExpander (${source.*}) │
@@ -505,8 +509,21 @@ Feature-gated items are marked with \* in the diagram.
505509
│ RuntimeEngine ──> wraps Engine + │
506510
│ CorrelationEngine with rule loading │
507511
│ │
512+
│ enrichment/ ──> post-eval pipeline: │
513+
│ primitives: template, lookup, │
514+
│ http, command │
515+
│ kind-aware namespaces: │
516+
│ ${detection.*}, ${correlation.*} │
517+
│ scope filter (rules, tags, levels) │
518+
│ HTTP response cache, on_error policy │
519+
│ └──> writes RuleHeader.enrichments │
520+
│ between engine and sinks │
521+
│ │
508522
│ io/ ──> EventSource (stdin, HTTP, NATS) │
509523
│ OTLP* (HTTP + gRPC) │
524+
│ TLS* termination (mTLS, cert │
525+
│ hot-reload) on shared API │
526+
│ listener │
510527
│ Sink (stdout, file, NATS) │
511528
│ DLQ (failed events) │
512529
└──────────────────────────────────────────┘
@@ -515,9 +532,10 @@ Feature-gated items are marked with \* in the diagram.
515532
516533
┌────────────────────────┐
517534
│ EvaluationResult │──> rule title, id, level, tags,
518-
│ = RuleHeader + │ matched selections, field matches,
519-
│ ResultBody:: │ aggregated values, optional events
520-
│ Detection / │
535+
│ = RuleHeader (incl. │ enrichments map, matched
536+
│ enrichments) + │ selections, field matches,
537+
│ ResultBody:: │ aggregated values, optional
538+
│ Detection / │ events
521539
│ Correlation │
522540
└────────────────────────┘
523541
```

assets/architecture.mmd

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ flowchart TD
22
YAML["YAML input"]
33
SERDE["yaml_serde"]
44
EVENTS["Log events"]
5-
OUTPUT["EvaluationResult\nRuleHeader: title · id · level · tags\nResultBody: Detection (matched fields/selections) ·\nCorrelation (correlation_type · group_key · aggregated_value)"]
5+
OUTPUT["EvaluationResult\nRuleHeader: title · id · level · tags · enrichments\nResultBody: Detection (matched fields/selections) ·\nCorrelation (correlation_type · group_key · aggregated_value)"]
66
QUERIES["SQL · SPL · KQL · Lucene"]
77
EDITOR["Editor diagnostics + code actions"]
88
LEGEND["* = feature-gated\n** = requires daachorse-index feature"]
@@ -45,8 +45,9 @@ flowchart TD
4545
RINPUT["input/ format adapters:\nJSON · syslog · logfmt* · CEF* · EVTX*\nplain text · auto-detect\nraw line → EventInputDecoded"]
4646
RINPUT --> RPROC["LogProcessor\nbatch evaluation\nArcSwap hot-reload (rules + pipelines)\nMetricsHook · EventFilter"]
4747
RPROC --> RENG["RuntimeEngine\nwraps Engine + CorrelationEngine\nwith rule loading"]
48-
RENG --> RIO["io/\nEventSource (stdin · HTTP · NATS)\nOTLP* (HTTP + gRPC)\nSink (stdout · file · NATS) · DLQ"]
49-
RSRC["sources/ (dynamic pipelines)\nSourceResolver: HTTP · command · file · NATS\nTemplateExpander · SourceCache (SQLite TTL)\nRefreshScheduler: interval · watch · push\nSIGHUP · NATS control · includes\nextract: jq · JSONPath · CEL"]
48+
RENG --> RENRICH["enrichment/ post-eval pipeline\nprimitives: template · lookup · http · command\nkind-aware: ${detection.*} · ${correlation.*}\nscope filter · HTTP response cache · on_error\nwrites RuleHeader.enrichments"]
49+
RENRICH --> RIO["io/\nEventSource (stdin · HTTP · NATS)\nOTLP* (HTTP + gRPC)\nTLS* termination (mTLS · cert hot-reload)\non shared API listener\nSink (stdout · file · NATS) · DLQ"]
50+
RSRC["sources/ (dynamic pipelines)\nDaemonSourceRegistry: external (--source) +\npipeline-embedded (deprecated) · collision-error\nSourceResolver: HTTP · command · file · NATS\nTemplateExpander · SourceCache (SQLite TTL)\nRefreshScheduler: interval · watch · push\nSIGHUP · NATS control · includes\nextract: jq · JSONPath · CEL"]
5051
end
5152

5253
YAML -->|"Raw YAML Value"| SERDE

0 commit comments

Comments
 (0)