Skip to content

Rule exemplars and rsigma rule test #343

Description

@mostafa

Summary

Embed machine-verifiable example events in Sigma rules under rsigma.exemplars (expected match/no-match), verify them with a new rsigma rule test command, lint shape checks, an MCP tool, and ADS validation integration so each rule carries its own regression evidence.

Motivation

Backtest works against external corpora, but per-rule true-positive and true-negative evidence does not travel with the rule. Embedding exemplars makes the rule the single reviewable artifact for Detection-as-Code diffs and CI.

Proposed approach

  • Typed extraction in the parser for exemplar event and correlation sequence payloads.
  • Static lint for shape and rule-kind mismatches (never executes exemplars).
  • Runner in the evaluator that compiles rules and asserts each exemplar outcome.
  • rsigma rule test with pipeline support, text/json output, and --fail-on-missing for CI.
  • MCP test_exemplars tool; a well-formed expect-match exemplar can satisfy ADS validation when prose is absent.

Out of scope

  • Replacing corpus-based rule backtest.
  • Auto-generating exemplars from production traffic (separate capture work).

Tasks

  • Parser types and lint rules for exemplars
  • Exemplar runner in the evaluator
  • rsigma rule test CLI and goldens
  • MCP tool and ADS integration
  • Docs: custom-attributes, cli/rule/test, lint catalogue

References

  • Custom attributes and ADS modules in rsigma-parser
  • rule backtest expectations schema for output conventions
  • MCP tool-router pattern in rsigma-mcp

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestrustPull requests that update rust code

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions