Flat execution plan / bytecode on top of IR

[mostafa]

Status

Feasibility done: deferred. The base IR shipped (crate, lowering, eval + convert wired through the HIR, optimization passes, serialization cache). A measurement spike then found that a flat execution plan / bytecode VM is not worth building now, because the condition tree-walk it would replace is not the bottleneck.

Summary

After the base IR landed, the idea was to add a flat execution-plan / bytecode lowering for hotter eval paths. Measurement shows the payoff is not there yet, so this is parked with a clear revisit trigger.

Feasibility findings

Isolated single-rule evaluator microbench (evaluate_rule directly, bypassing the engine's RuleIndex / bloom / cross-rule-AC prefilters):

• Baseline single-selection rule: ~201 ns/event.
• ~198 ns per added selection, dominated by the detection HashMap lookup, the event field lookup, the matcher call, the matched_selections allocation, and the post-match collect_field_matches second pass. The And/Or/Not recursion a bytecode VM would replace is a rounding error.
• Duplicate-reference re-evaluation is real but cache-hot (~145 ns) and rare in real rules.
• The post-match second pass runs only on a match, which is rare once the prefilters prune the candidate set.

Conclusion: a bytecode rewrite is a large, high-risk change to a correct, fast hot path (with load-bearing wire-shape quirks) for a gain the numbers do not support.

Higher-leverage alternatives (do these first, if eval throughput ever demands it)

1. Capture field matches during the first eval pass to drop the post-match collect_field_matches re-evaluation (wire-shape sensitive; only pays on matching events).
2. Resolve detection-name lookups to integer slot indices at compile time (a flat plan without a VM).
3. Memoize per-event detection results (helps duplicate refs / selectors; low expected payoff per the spike).

Only pursue the full bytecode VM after 1-3 land and profiling of a realistic post-prefilter workload still attributes meaningful time to condition-walk dispatch.

Constraints any evaluator change must preserve

• matched_selections is not deduplicated; Not records inner matched identifiers into the same vector.
• Selector counting (Any/All/Count(n)), vacuous all of <pattern> = true, them skipping _-prefixed names.
• Array-scope object matching and Conditional bodies; MatchDetailLevel::{Off,Summary,Full} output; the 16-entry keyword-field cap.
• explain_rule is a separate recording evaluator that must stay in parity.
• Regression gate: match/no-match + EvaluationResult wire shape identical to the tree-walk across the SigmaHQ corpus and fixtures (never CompiledRule structural equality).

Out of scope

• Shipping before the base IR phases (now moot: they shipped).
• Cross-language bytecode compatibility.

Tasks (revised)

• Revisit trigger: profile a realistic post-prefilter workload; only proceed if condition-walk dispatch is a meaningful share.
• Alternative 1: drop the post-match second pass by capturing matches during the first pass.
• Alternative 2: slot-index detection lookups at compile time.
• Alternative 3: per-event detection-result memoization.
• Only then: opcode/shape design, compile_to_program lowering, MatchVm, equivalence tests, Criterion gate.

References

• Eval hot path (crates/rsigma-eval/src/compiler/mod.rs, matcher/, explain/) and the eval benches.
• The base of Internal IR layer (rsigma-ir) #346 shipped.