feat(evaluation) 3/15: indexing throughput aggregation

[Ahmath-Gadji]

Part 3 of 15 of the split of #811. Targets eval/02-testset-parsing (#813).

What

indexing_metrics — folds the per-file timings the worker collects (part 9) into throughput figures: files/min, MB/s, p50 / p95, and a per-extension breakdown. Pure; the worker measures, this decides what the measurements mean.

Notable

Two definitions worth calling out, both pinned by tests:

• Throughput divides by measured wall clock, not by the sum of the per-file durations. Summing overstates speed the moment files are indexed concurrently. (Indexing is sequential today — see the caveat carried over from feat(evaluation): admin System tab for indexing speed and RAG quality #811 — but the metric should not silently become wrong when it isn't.)
• Percentiles are nearest-rank (ceil). statistics.quantiles needs two points and interpolates, so it cannot describe a one-file run at all; and round breaks ties to even, which picks the wrong observation whenever fraction * n lands on an odd integer — p50 of two files returned the slower one. test_p50_of_an_even_sample_takes_the_lower_middle is the regression test.

Files that failed to index are counted in files_total / files_failed and then excluded from every rate and percentile — a failure is not a fast file.

Review follow-up

bytes_total now counts every file the run attempted (CodeRabbit). It previously summed only the successful samples, which made it disagree with files_total sitting right beside it — and the run detail page renders this field as "Corpus size", so a run where a large PDF failed to parse would have reported a corpus smaller than the one the admin uploaded, with nothing explaining the gap.

Split into the two questions it was conflating: total_bytes (all samples → bytes_total) and indexed_bytes (successes only → megabytes_per_second). Rates and percentiles were already computed off the successes and are unchanged. The failure-accounting test now uses distinct non-zero sizes and asserts both directions, so it fails against the old code.

Testing

7 unit tests: wall-clock throughput, failure accounting, single-file and even-sample percentiles, zero-division, extension bucketing. ruff, format check and the layer-import guard pass.

The rest of metrics.py — folding promptfoo's output into retrieval and answer quality — lands in part 4.

Summary by CodeRabbit

• New Features

  • Added indexing evaluation metrics, including file counts, processed bytes, throughput, duration percentiles, and extension-based breakdowns.
  • Exposed indexing metrics through the evaluation package.

• Bug Fixes

  • Failed files are excluded from success-based performance calculations.
  • Zero-duration evaluations now return safe zero rates without errors.

• Tests

  • Added coverage for throughput, failures, percentiles, zero-time handling, and extension grouping.

[coderabbitai]

Warning

Review limit reached

@EnjoyBacon7, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 59 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: e9ed74ce-f513-470f-ab9f-1ba99a9547f3

📥 Commits

Reviewing files that changed from the base of the PR and between 7e09c04 and e573a8a.

📒 Files selected for processing (3)

• openrag/core/evaluation/__init__.py
• openrag/core/evaluation/metrics.py
• tests/unit/core/evaluation/test_metrics.py

📝 Walkthrough

Walkthrough

Adds an indexing metrics module that aggregates file samples into throughput, duration, percentile, and extension statistics, exports it through the evaluation package, and tests successful and failed indexing scenarios.

Changes

Indexing metrics

Layer / File(s)	Summary
Implement indexing metric aggregation openrag/core/evaluation/metrics.py, openrag/core/evaluation/__init__.py	Adds nearest-rank percentiles, wall-time-based throughput, successful-sample filtering, extension-level means, failure counts, and public package exports.
Validate indexing metric behavior tests/unit/core/evaluation/test_metrics.py	Tests throughput, failure handling, percentile selection, zero wall time, and lowercased extension aggregation.

Estimated code review effort: 3 (Moderate) | ~20 minutes

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly reflects the main change: adding evaluation indexing throughput aggregation.

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch eval/03-indexing-metrics

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Ahmath-Gadji]

@coderabbitai review

[coderabbitai]

✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

[Ahmath-Gadji]

@coderabbitai full review

[coderabbitai]

✅ Action performed

Full review finished.

[coderabbitai]

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@openrag/core/evaluation/metrics.py`:
- Around line 44-46: Update the metric aggregation around succeeded, durations,
and total_bytes so bytes_total includes size_bytes from all samples, while
retaining a separate successful-byte total for MB/s calculations and excluding
failed samples from rates and percentiles. Extend the failure-accounting test
with a failed sample having nonzero size and assert that its bytes contribute to
bytes_total.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 71ffb8fb-b6cb-468e-9026-eab16c5f30e5

📥 Commits

Reviewing files that changed from the base of the PR and between 18bd516 and 7e09c04.

📒 Files selected for processing (3)

• openrag/core/evaluation/__init__.py
• openrag/core/evaluation/metrics.py
• tests/unit/core/evaluation/test_metrics.py

[hedhoud]

Looks good to me. The metrics are calculated consistently, failed files are handled correctly, and the focused tests pass locally.