feat: add RCAEval RE2/RE3 as a downloadable eval corpus (#78)

[leo-aa88]

Summary

Wires RCAEval (MIT — phamquiluan/RCAEval) into the eval harness from #77, giving raglogs its first labeled corpus. Unblocked now that #77 is merged.

RCAEval ships one directory per failure case named {benchmark}{system}_{service}_{fault}_{instance} (e.g. re2ob_adservice_cpu_1), each with inject_time.txt (the fault-injection Unix timestamp — exactly the trigger label the harness needs) and logs.csv (time, service, message). Because the case name encodes the ground-truth root-cause service and fault type, labels come from the directory name + inject_time.txt — no Parquet-metadata dependency.

What's here

• src/eval/rcaeval.py — pure, unit-tested converters: parse the case name, logs.csv → raglogs JSONL (inferring a level so error lines drive clustering and the baseline), inject_time → trigger label, a documented window policy around injection, and case.yaml assembly.
• scripts/eval_data.py (make eval-data) — snapshot_download of re2*/re3* into gitignored data/, then converts each case into data/eval-cases/rcaeval/{re2,re3}/. --suite/--limit for quick subsets. The ~3.4 GB data is never committed.
• huggingface-hub added to the dev extra; the eval README documents scoring RE2 and RE3 separately.

The expected finding

RE2's faults (CPU/memory/disk/network stress) don't write a log line announcing themselves, so raglogs' 12 hand-written trigger patterns will match nothing — every RE2 case caps at "trigger: none identified". That near-zero RE2 trigger score is not a bug in the dataset; it's the finding, and it directly motivates the trigger-redesign issue (#82). RE3 (code-level faults, visible as stack traces) is where raglogs should be competitive.

Testing

• make lint → All checks passed
• make test-unit → 947 passed (17 new, including that converted output loads back as a valid harness case)

Conversion is fully unit-tested; the download and first real numbers come from running make eval-data (needs network + huggingface_hub) then raglogs eval --cases data/eval-cases/rcaeval/re2.

Closes #78

🤖 Generated with Claude Code

[leo-aa88]

APPROVE

LGTM, with one thing to verify before trusting the RE2/RE3 root-cause numbers (inline). Clean, pure, well-tested wiring of RCAEval into the #77 harness — reviewed at head 4f6b2f8.

CONFIRMED — labels come from the directory name + inject_time.txt, no Parquet dependency, and the data is never committed. .gitignore covers data/; the diff adds no corpus files (only code + tests + README); snapshot_download is scoped to re2*/re3* into gitignored data/ and gated behind an explicit make eval-data (not run in CI).

CONFIRMED — conversion is correct and round-trips. convert_logs_csv handles header or positional time,service,message, skips unparseable-timestamp rows, and infers level so error lines drive clustering/baseline; parse_inject_time/_parse_log_time guard ms-vs-s epochs and fall back to ISO; build_case_yaml emits exactly the shape the #77 loader expects (no logs key → defaults to the logs.jsonl that convert_case writes). 17 unit tests, including one that loads the converted output back as a valid harness case; lint clean.

Honest framing of the RE2 result. The "RE2 trigger score ≈ 0 because stress faults don't announce themselves in logs" is a legitimate expected finding that motivates #82, not a dataset bug — good to call it out up front and to score RE2/RE3 separately.

One MINOR/QUESTION inline on parse_case_dir_name: the positional single-token fault assumption may mislabel the root-cause service if RCAEval uses any compound fault names — worth confirming against the real directory listing, since it's the metric the corpus is meant to feed.

VERDICT

Correct, well-tested, boundary-respecting (no committed data, download gated), and honestly scoped (harness + converter now; first numbers from make eval-data). No BLOCKING/MAJOR findings. Merge it — just confirm the fault-name parsing against actual RCAEval dirs (or make it vocabulary-aware) before relying on RE2/RE3 root-cause accuracy. (Comment rather than Approve — GitHub blocks self-approval on your own PR.)

[leo-aa88]

MINOR / QUESTION — fault is extracted positionally as a single token (parts[-2]), with everything between system and fault lumped into service. If any RCAEval case name uses a compound fault, the first fault token silently leaks into the root-cause service label.

Worked example, assuming a dir like re2ob_adservice_packet_loss_1:

parts   = [re2ob, adservice, packet, loss, 1]
fault   = parts[-2]          -> "loss"
service = "_".join(parts[1:-2]) -> "adservice_packet"   # ← wrong root-cause label

That would make root_cause.service wrong for those cases, silently depressing the root-cause-accuracy metric — the exact number the corpus exists to produce.

What makes this a real QUESTION rather than a nit: trigger_type_for_fault is written and unit-tested against a compound fault (trigger_type_for_fault("packet_loss") == "dependency"), yet the parser can only ever hand it a single trailing token — so the two halves of the module disagree about whether faults are compound. Either faults are always single-token (then the packet_loss test exercises an unreachable input, and this is fine) or they aren't (then service labels are corrupted for those cases).

Please confirm against the actual RCAEval directory names. If compound faults exist, make the fault extraction vocabulary-aware — peel a known fault suffix (cpu|mem|disk|io|socket|delay|latency|loss|network|code, longest match from the right) and treat the remainder as the service — so service is always correct regardless of token count. I couldn't verify the naming here (the 3.4 GB download isn't runnable in review), so flagging rather than asserting.