Skip to content

fix: preserve multiclass per-source predictions - #139

Merged
dylanuys merged 2 commits into
devfrom
fix/multiclass-per-source
Aug 26, 2026
Merged

dylanuys merged 2 commits into
devfrom
fix/multiclass-per-source

Conversation

@dylanuys

Copy link
Copy Markdown
Contributor

Summary

Preserve prediction counts for every supported media class in the bookkeeping summaries emitted after a benchmark run. Video predictions labeled semisynthetic or rendered were absent from per_source_accuracy, which made downstream reporting show partial totals and misleading 0/0 rows.

Scope

This is a reporting/bookkeeping-only change. It does not change:

  • model inference or preprocessing
  • dataset discovery, sampling, or evaluation coverage
  • ground-truth labels or predicted labels
  • MCC, Brier, SN34, benchmark score, ranking, or pass/fail outcomes
  • benchmark execution behavior or runtime characteristics

Those metrics are calculated from the complete result rows before per_source_accuracy is assembled. This patch only stops the final summary payload from discarding prediction-count buckets for labels 2 and 3.

Changes

  • include semisynthetic and rendered counts in per-dataset prediction distributions
  • carry all four counts into per_source_accuracy
  • add a regression test covering the complete per-dataset-to-per-source reporting path

Existing runs

Existing scores and outcomes do not require correction. Their lossy per_source_accuracy bookkeeping can be rebuilt from stored parquet records without rerunning inference.

Validation

  • pytest -q tests/unit/test_multiclass_metrics.py — 18 passed
  • Ruff checks passed

dylanuys and others added 2 commits August 24, 2026 17:56
Release experimental taxonomy, scoring, and download updates
@dylanuys
dylanuys merged commit 585a7d8 into dev Aug 26, 2026
3 checks passed
@dylanuys
dylanuys deleted the fix/multiclass-per-source branch August 26, 2026 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant