Skip to content

curation: rank measurements_latest by the episode's own recorded_at - #52

Merged
kstonekuan merged 1 commit into
Hebbian-Robotics:mainfrom
chintondutta:fix/51-measurements-latest-repair-on-read
Aug 20, 2026
Merged

curation: rank measurements_latest by the episode's own recorded_at#52
kstonekuan merged 1 commit into
Hebbian-Robotics:mainfrom
chintondutta:fix/51-measurements-latest-repair-on-read

Conversation

@chintondutta

Copy link
Copy Markdown
Contributor

Fixes #51.

The gap

#47 made episodes/<file_stem>.parquet the single source of truth for recorded_at on a given append -- but only for the episodes table itself. measurements_latest still ranks rows by its own recorded_at column:

CREATE VIEW measurements_latest AS
SELECT * EXCLUDE (row_rank) FROM (
    SELECT *, row_number() OVER (
        PARTITION BY episode_id, key
        ORDER BY recorded_at DESC, run_fingerprint DESC
    ) AS row_rank FROM measurements
) WHERE row_rank = 1

A repair pass that wins the episodes race can still crash before reaching measurements (Catalog.append_episode's repair loop in src/hflow/catalog.py). Because exists(episodes/...) is the only thing a later retry checks before early-returning, that dependent's recorded_at is never revisited -- it can be permanently older or newer than the episode it belongs to. measurements_latest and episodes_latest can then disagree about which run is "latest" for the same episode_id, reproducing the class of bug #47 fixed, just from a narrower crash window instead of a live race.

Fix

Join measurements to episodes_raw on (episode_id, run_fingerprint) and rank -- and report -- recorded_at from the episode side, the one column create-if-absent guarantees a single writer for. episodes_latest itself needed no change (it already IS the authoritative source). The inner join also means a run whose episodes file hasn't landed yet (an append still mid-flight or crashed before completing) is invisible in measurements_latest, matching append_episode's own idiom that episodes existing is what proves an append complete.

Testing

New regression test test_measurements_latest_ranks_by_the_owning_episodes_recorded_at (tests/test_catalog_curation.py): records an older and a newer run of the same episode, then corrupts the newer run's measurements file to carry an older recorded_at than the older run's (simulating the crash-mid-repair window). Confirmed it fails against pre-fix code (picks the stale 1.0 instead of 2.0) and passes with the fix.

uv run pytest tests/test_catalog_curation.py -q   # 19 passed
uv run pytest -q                                  # unrelated: tests/test_ffmpeg.py fails/errors in this sandbox (no ffmpeg/ffprobe on PATH); everything else passes: 298 passed, 3 skipped
uv run ruff check --fix
uv run ruff format
uv run ty check

No stored-format changes -- this only changes which recorded_at a measurements_latest row reports and how it's ranked, never what's stored on disk.

A repair pass that wins the episodes create-if-absent race can still crash
before reaching the measurements table; a later retry early-returns on
exists(episodes) and never revisits it, leaving that table's recorded_at
stale forever (Hebbian-Robotics#51). measurements_latest ranked by its own recorded_at
column, so it could then disagree with episodes_latest about which run is
newest for the same episode. Join to episodes_raw and rank (and report) off
its recorded_at instead -- the one column create-if-absent guarantees a
single writer for.

@kstonekuan kstonekuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @chintondutta. Another self-found correctness bug with a precise writeup, a fix that targets the actual invariant, and a regression test that locks it in. That is three for three.

What I validated locally:

  • Full quality gate is clean (ruff check, ruff format --check, ty check) and all 307 tests pass, including the new test_measurements_latest_ranks_by_the_owning_episodes_recorded_at.
  • Reproduced your pre-fix failure by checking main's curation.py into the branch: the test picks the stale 1.0 exactly as you described, and passes with the fix.
  • Confirmed the wide episodes view builds its measurement columns from measurements_latest, so everyday queries inherit the correction too.

What I especially like: ranking off the one column that create-if-absent guarantees a single writer for, instead of trying to catch every crash window at write time. The inner join dropping runs whose episodes file has not landed is the right call and matches append_episode's own "episodes existing proves the append completed" idiom. The comment in the view explaining all of that will save the next reader a lot of archaeology.

Merging now.

One ask before your next batch: please stick to one open pull request at a time, and leave the good first issues for newcomers. The bot closed the seven you opened yesterday for that reason, and none of the work is lost, but opening them all at once crowds out first-time contributors and outruns our review bandwidth. Honestly, starter issues are beneath what you are doing here. Bugs like #44, #46, and #51 are worth far more to the project, and if you want a fresh hunting ground, try running HFlow against a real corpus like Egocentric-10K and telling us what breaks, what is slow, or what is awkward. Come say hi on Discord if you have not already.

@kstonekuan
kstonekuan merged commit aa00fe8 into Hebbian-Robotics:main Aug 20, 2026
5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

catalog: crash-mid-repair after winning episodes race can leave dependents permanently stale

2 participants