Skip to content

feat(api): editorial_score + sentiment_score (schema v2)#361

Merged
FlyM1ss merged 13 commits into
mainfrom
feat/score-field-rename
Jul 15, 2026
Merged

feat(api): editorial_score + sentiment_score (schema v2)#361
FlyM1ss merged 13 commits into
mainfrom
feat/score-field-rename

Conversation

@FlyM1ss

@FlyM1ss FlyM1ss commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator

Disambiguates the two unrelated score fields (spec: Docs/superpowers/specs/2026-07-14-score-field-disambiguation-design.md).

  • items: disk+wire editorial_score, news-story v2 — strict, legacy batches poison-pill (404)
  • signals: artifact v2 sentiment_score, pinned by new signals-v2.schema.json; the view normalizes historical v1 so the wire is uniformly v2, including ?as_of reads
  • three-copy gate edited same-commit; parity corpus widened + mutation-tested both ways
  • LLM prompt/reply key, PROMPT_VERSION, diagnostics.scores_damped and ATL-internal names unchanged

Merge gate: ATL PR-1 (Open-Finance-Lab/AgenticTrading#112, sentiment_score fallback) must be merged and deployed first. Merging here auto-deploys Heartbeat + Django concurrently — run the plan's Task 9 runbook right after (it was corrected on main in 0581d9a: the manual kick can exit 0 without writing a batch, and the rollback is symmetric for items but not for signals).

Tests: Heartbeat 172 OK; backend 780 passed / 1 skipped / 40 subtests.

Beyond the plan, three defects in the plan's own code were found and fixed — see the review notes below.

🤖 Generated with Claude Code

https://claude.ai/code/session_012Hfd5GY7STjojT7kSrZSqF

FlyM1ss and others added 9 commits July 15, 2026 00:25
…tory v2)

The items/news pipeline's newsworthiness score is now editorial_score,
disambiguating it from the signals pipeline's sentiment score. Task 3
renamed the producer; this renames the reader side to match.

Both copies of the input trust boundary move together in this commit —
Heartbeat's validation_gate and its port in signals_views._validate_items —
so the two gates never disagree about what input is valid. Legacy score-only
batches are a missing required field: they poison the batch (404), a strict
cutover with no compat shim.

Parity guard extended with two strictness cases and mutation-tested: a
one-sided flip of either copy's REQUIRED_FIELDS fails the parity suite.

signals-fixture.json regenerated: the artifact embeds the generator VERSION,
which this commit bumps. Exactly one line changes (.generator) — the other 42
fields are identical, confirming the rename moves no signal values.

The signals writer's own score (LLM reply key, diagnostics.scores_damped,
artifact entries) is untouched — that is a separate contract.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Hfd5GY7STjojT7kSrZSqF
Historical v1 disk artifacts (score) are normalized to v2
(sentiment_score) at the response boundary, whether served as latest
or via ?as_of, so `score` never reaches the wire. Entry dicts are
copied before mutation since they're shared references into the
per-request memoized artifact.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Hfd5GY7STjojT7kSrZSqF
Normalize per entry regardless of the artifact's schema_version, and drop a
stray score rather than let it overwrite an authoritative sentiment_score.
The view is the only boundary — nothing validates artifacts at runtime.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Hfd5GY7STjojT7kSrZSqF
The banner described the items rename while citing only §4.2/§4.4, neither of
which defines it; §4.1's pinned items contract is where `score` ->
`editorial_score` actually lands. §4.3's LLM reply key is called out as
deliberately unchanged so it is not "fixed" later.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Hfd5GY7STjojT7kSrZSqF
The validator keyed only on artifact identity, so a wire-shape change left it
byte-identical while the body changed — a conditional request could 304 a
client into a stale v1 body carrying `score`, permanently for ?as_of reads.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Hfd5GY7STjojT7kSrZSqF
If-Modified-Since alone bypasses the ETag entirely (Django conditional
response step 4), so a frozen generated_at would 304 a client into a stale v1
body despite the salted ETag. Native-v2 artifacts keep the validator.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Hfd5GY7STjojT7kSrZSqF
@FlyM1ss

FlyM1ss commented Jul 14, 2026

Copy link
Copy Markdown
Collaborator Author

Deviations from the plan (all deliberate, all reviewed)

Three of the plan's own instructions turned out to be defective. Each was verified against the code before deviating:

  1. Task 6's normalizer, hardened. The plan's code only renamed when "score" in entry and "sentiment_score" not in entry, and gated the whole loop behind schema_version != 2 — leaving two paths for score to reach the wire (an entry with both keys; a v2-labelled artifact with a stray score). Neither is reachable from today's writer, but there is no runtime schema validation in the view, so the invariant rested on an assumption. Now entries normalize unconditionally and setdefault makes the v2 value authoritative.
  2. Task 4 regenerated signals-fixture.json despite its brief saying not to. The brief was self-contradictory: the fixture embeds "generator": "news_signals.py/<VERSION>" and test_committed_fixture_matches_regeneration rebuilds it live, so the mandated VERSION bump reds that test on its own. Obeying literally would have committed a red suite. Diff is one line; the other 42 leaf fields being unchanged independently proves the items rename moves no signal values.
  3. Task 7's amendment banner cites §4.1. The plan cited only §4.2/§4.4 — inherited from the design doc's own Amends: line — but §4.1 is where the items field it describes actually lives.

Two defects the final whole-branch review caught (fixed here: 3b35c7c, ad9c05d)

Neither was visible to any per-task review, since each saw only one commit.

  • _etag keyed only on artifact identity. This branch changes the body-generating function for an unchanged artifact, so pre/post-deploy ETags were byte-identical while bodies differed — an If-None-Match revalidation would 304 a client into a stale v1 body carrying score. Not self-healing: a poison-pilled run writes no artifact, so generated_at freezes for ~24h; for ?as_of it froze permanently. Fixed by salting with the wire version.
  • If-Modified-Since bypasses the ETag entirely (Django get_conditional_response step 4 has a not if_none_match_etags guard), so the salt couldn't reach that path. Fixed by withholding Last-Modified from artifacts the normalizer rewrites; native-v2 artifacts keep it, and the gate self-retires.

Accepted residual: Cache-Control: public, max-age=300 lets a shared cache serve a v1 body unrevalidated for ≤5 min. No validator can fix that; bounded, and ATL PR-1's fallback covers it.

Follow-up before ATL PR-2 (which deletes that fallback): grep ATL for If-None-Match/ETag. If ATL revalidates conditionally, the fallback is what's absorbing any stale score body today — and a stale body after PR-2 would KeyError-drop silently, the same signature as the earlier AF→ATL wire mismatch.

FlyM1ss and others added 4 commits July 15, 2026 02:42
The page said Last-Modified is sent "only on unfiltered responses", which a
reader takes as "always, when unfiltered". Since ad9c05d it is also withheld
from artifacts predating the rename — permanently for ?as_of reads — so the
sentence under-described when the header is absent. Also record that
_SIGNALS_WIRE_SCHEMA_VERSION now gates three things, not one.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_012Hfd5GY7STjojT7kSrZSqF
§4.2 and §9 still pointed contract tests at `signals-v1.schema.json`, which
this branch made false: v1 pins the superseded `score`, so it rejects current
output rather than validating it. §4.4's `Headers` bullet — the contract
`signals_views.py` names in its own docstring — still described both
conditional validators in their pre-rename shape, missing the ETag version
salt and the `Last-Modified` schema gate.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pbs5wnQZhTi9CQ3pb14vDo
`_load_artifact` validates that `signals` is a dict but never each entry's
type, so a corrupt entry does reach the wire normalizer, whose `isinstance`
guard was reachable but untested. It closes two separate raises, because
`"score" not in entry` means something different per type: a TypeError on an
int, and a *substring* test on a str — so an entry merely containing "score"
slips through into `dict(entry)` and raises ValueError. Either would 500 an
endpoint whose every failure path is contracted to be a 404.

Also records why `_items_etag` needs no schema salt, unlike `_etag`: the items
poison pill makes every pre-rename batch unservable, so no stale-representation
window exists for a salt to close.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pbs5wnQZhTi9CQ3pb14vDo
Nothing has loaded it since 8e40d22 repointed both suites at v2, so an
unannotated in-tree schema reads as the live contract. A `$comment` records
that it describes what is *stored*, not what is served: pre-rename artifacts
stay on disk and are still reachable via `?as_of`, but the view normalizes
them up to v2 at the wire. `$comment` is a draft-07 annotation, so this cannot
change a validation outcome.

The `window_hours` comment in `test_news_signals.py` still named v1 after the
same commit repointed `SCHEMA_PATH`; v2 pins the same floor, so it now names
no version.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Pbs5wnQZhTi9CQ3pb14vDo
@FlyM1ss FlyM1ss merged commit 6c4d191 into main Jul 15, 2026
6 checks passed
@FlyM1ss FlyM1ss deleted the feat/score-field-rename branch July 15, 2026 06:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant