feat(api): add /api/news/items/ raw news-items endpoint (#359)

* test(api): add TDD suite for /api/news/items/ (Phase B, ATL integration)

17 cases pinning the raw news-items response contract: newest-by-mtime
selection, limit clamping [1,200], conditional GET/ETag variance, poison-pill
vs drop-story validation semantics, and security parity (control/bidi strip,
NEWS_DATA marker, field caps). Written red against the not-yet-existing
news_items view/RAW_ITEMS_DIR setting.

* feat(api): add /api/news/items/ raw news-items endpoint

Phase B of the ATL news-signals integration plan: serves the newest
items-*.jsonl batch, ungated (no subject/roundup/LLM gating — that stays
news_signals-only). Validation/sanitization (_validate_items/_clean_text) is
a byte-faithful port of Heartbeat/news_signals.py's validation_gate/
clean_text, since that script is deliberately isolated and not importable
from Django. Same decorator stack, fail-closed 404 contract, and per-request
memoization pattern as news_signals. Also extends test_api_auth.py with
dedicated 401/dev-mode/valid-bearer coverage for the new route.

Suite: 738 passed, 1 skipped (pre-existing).

* chore(deploy): mount heartbeat digests dir for /api/news/items/

Sibling of the existing signals :ro mount + env: without this the endpoint
404s in prod (RAW_ITEMS_DIR unset -> unconfigured -> fail-closed).

* fix(api): fail-closed on corrupt tickers + no re-stat in conditional-GET + regex port fidelity

- Guard the tickers comprehension in _validate_items against non-string
  elements (AttributeError -> 500) and non-list values (char-iteration ->
  silently wrong tickers), dropping only the bad entries per the module's
  never-500 contract.
- Thread the newest batch's mtime through the _load_items memo instead of
  re-stat()ing in _items_etag/_items_last_modified, closing a TOCTOU window
  where a pruned batch could 500 instead of 404.
- Restore double-backslash regex escapes in _CONTROL_RE/_LINEBREAK_RE to
  match Heartbeat/news_signals.py verbatim (ASCII-only .pattern).

* feat(api): items endpoint speaks news-story v1 (headline/url + schema_version)

* fix(news): harden the items gate and pin the port to its source

Review findings on #359:

- clean_text/validation_gate: non-str required fields drop the story
  (same stance as the numeric parse) instead of raising; clean_text is
  now total. Fixes a live crash in news_signals.py, where a corrupt
  "tickers":[123] raised AttributeError past process_batch's
  ValueError-only except and aborted the whole sweep.
- Both copies (Heartbeat + the Django port) fixed together and pinned by
  Heartbeat/tests/test_port_parity.py, which AST-extracts the ported
  region and compares behaviour without importing Django. Corpus is
  mutation-proven: it goes red on the drift it claims to catch.
- _MAX_ITEMS_FILE_MB no longer hardcoded — settings.RAW_ITEMS_MAX_FILE_MB
  reads SIGNALS_MAX_FILE_MB, so one operator knob moves both readers.
- signals_views.py added to heartbeat-tests.yml pull_request paths only:
  the push trigger also gates the deploy job (paths scope the run, not
  the job), so listing it there would redeploy the droplet on a
  Django-only merge.
- Coverage for the ?limit Last-Modified suppression, the recency upper
  bound, and the blank-line skip — each mutation-verified.

VERSION -> 2026-07-14.1 (deployed behaviour changed); fixture regenerated.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(api): document GET /api/news/items/

- api_reference: endpoints-table row + a News Items section (limit
  clamp, no as_of, fail-closed 404, ETag/Last-Modified variance). Notes
  that its `score` is the editorial score, not the [-1,1] sentiment
  score of the same name under /api/signals/news/.
- project_structure: signals_views.py hosts both news endpoints now.
- pipeline design spec: SIGNALS_MAX_FILE_MB has a second reader.

Sphinx build clean (0 warnings).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: FlyM1ss

.github/workflows/backend-deploy.yml

@@ -313,7 +313,7 @@ jobs:
             cat > "$OVERRIDE_DIR/override.conf" <<EOF
             [Service]
             ExecStart=
-            ExecStart=/usr/bin/podman run --name ${SYSTEMD_UNIT} --replace --rm --cap-drop=ALL --cap-add=NET_ADMIN --cap-add=CHOWN --cap-add=SETUID --cap-add=SETGID --cap-add=SETPCAP --pids-limit=1024 --read-only --tmpfs=/tmp:rw,size=512m,mode=1777 --tmpfs=/app/staticfiles:rw,mode=0755 --tmpfs=/home/fingpt:rw,mode=0755 --cgroups=split --sdnotify=conmon -d --memory=1.7g --memory-swap=2g --network fingpt-net -v /home/deploy/fingpt/runtime:/app/runtime:U,Z -v /home/deploy/fingpt/heartbeat/signals:/app/signals:ro,Z --env SIGNALS_DIR=/app/signals --publish 127.0.0.1:8000:8000 --env-file /home/deploy/fingpt/envs/.env.production --env REDIS_URL=redis://fingpt-redis:6379/0 ${REMOTE_IMAGE}
+            ExecStart=/usr/bin/podman run --name ${SYSTEMD_UNIT} --replace --rm --cap-drop=ALL --cap-add=NET_ADMIN --cap-add=CHOWN --cap-add=SETUID --cap-add=SETGID --cap-add=SETPCAP --pids-limit=1024 --read-only --tmpfs=/tmp:rw,size=512m,mode=1777 --tmpfs=/app/staticfiles:rw,mode=0755 --tmpfs=/home/fingpt:rw,mode=0755 --cgroups=split --sdnotify=conmon -d --memory=1.7g --memory-swap=2g --network fingpt-net -v /home/deploy/fingpt/runtime:/app/runtime:U,Z -v /home/deploy/fingpt/heartbeat/signals:/app/signals:ro,Z --env SIGNALS_DIR=/app/signals -v /home/deploy/fingpt/heartbeat/digests:/app/digests:ro,Z --env RAW_ITEMS_DIR=/app/digests --publish 127.0.0.1:8000:8000 --env-file /home/deploy/fingpt/envs/.env.production --env REDIS_URL=redis://fingpt-redis:6379/0 ${REMOTE_IMAGE}
             EOF
 
             systemctl --user daemon-reload

.github/workflows/heartbeat-tests.yml

@@ -7,10 +7,23 @@ on:
     paths:
       - "Heartbeat/**"
       - ".github/workflows/heartbeat-tests.yml"
+      # Main/backend/api/signals_views.py is deliberately NOT listed here,
+      # only on pull_request below: this trigger also gates the deploy job,
+      # whose sole condition is the main-ref check (paths scope the workflow
+      # RUN, not a job), so a Django-only merge would redeploy the droplet's
+      # heartbeat scripts — silently shipping any held-back rollout on a
+      # commit that touched no Heartbeat file.
   pull_request:
     paths:
       - "Heartbeat/**"
       - ".github/workflows/heartbeat-tests.yml"
+      # The backend vendors a port of news_signals.py's validation gate
+      # (clean_text/validation_gate -> _clean_text/_validate_items) and
+      # Heartbeat/tests/test_port_parity.py is what pins the two copies
+      # together, so a port-side edit must run this suite. PR-time is the
+      # right gate: the point is to make drift impossible to MERGE, and
+      # deploy skips on PR refs anyway.
+      - "Main/backend/api/signals_views.py"
   workflow_dispatch:
 
 env:

Docs/source/api_reference.rst

@@ -571,6 +571,9 @@ the signed ``fingpt_sessionid`` cookie. Callers may pass an optional
    * - GET
      - ``/api/signals/news/``
      - Latest news→sentiment signals artifact
+   * - GET
+     - ``/api/news/items/``
+     - Raw news stories from the newest Heartbeat batch
 
 All share the ``API_RATE_LIMIT`` budget (``429 {"error": "rate_limited"}``
 when exceeded). The chat endpoints can also return ``503 {"error": "busy"}``
@@ -727,6 +730,44 @@ an ``ETag`` validator and ``Cache-Control: public, max-age=300``.
 variants are ETag-only, so conditional requests for them must use
 ``If-None-Match``.
 
+News Items
+~~~~~~~~~~
+
+``GET /api/news/items/`` serves the **raw news stories** of the newest
+Heartbeat batch — the corpus the signals above are derived from, before any
+LLM scoring. Like ``/api/signals/news/`` it is an integration surface for
+external consumers; the browser extension does not call it.
+
+**Query parameters:**
+
+- ``limit=N`` — how many stories to return, newest first. Clamped to
+  ``[1, 200]``; defaults to ``50`` when absent. A non-integer value returns
+  ``400 {"error": "bad_limit"}``.
+
+There is no ``as_of`` here: this endpoint always reads the single newest
+batch.
+
+**Response (200):** ``{schema_version, items, count, batch}``, where ``batch``
+names the source file and each entry of ``items`` is
+``{guid, headline, url, source, published, description, tickers, score}``.
+``published`` is epoch seconds.
+
+.. note::
+
+   ``score`` on this endpoint is the pipeline's **editorial** score — how
+   newsworthy the story is, the gate that decides which stories become
+   sentiment candidates (``SIGNALS_MIN_EDITORIAL_SCORE``). It is *not* the
+   ``[-1, 1]`` sentiment score of the same name under ``/api/signals/news/``.
+   The two fields share a name and nothing else.
+
+``404 {"error": "no_items"}`` when no batch exists, or when the newest one is
+unreadable or validates to zero stories — the endpoint never falls back to an
+older batch, and never returns a ``500``. Responses carry an ``ETag`` and
+``Cache-Control: public, max-age=300``. ``Last-Modified`` is sent only on the
+default-limit variant — an explicit ``?limit`` slices the batch differently,
+so those variants are ETag-only and conditional requests for them must use
+``If-None-Match``.
+
 ---
 
 Usage Examples

Docs/source/project_structure.rst

@@ -38,7 +38,7 @@ Backend Structure
    ├── api/                          # REST API layer
    │   ├── views.py                  # Main API endpoints
    │   ├── openai_views.py           # OpenAI-compatible API endpoints
-   │   ├── signals_views.py          # News-signals endpoint (GET /api/signals/news/)
+   │   ├── signals_views.py          # News endpoints (GET /api/signals/news/, GET /api/news/items/)
    │   ├── middleware/               # CORS and custom middleware
    │   ├── utils/                    # API utility functions
    │   ├── apps.py                   # Django app configuration

Docs/superpowers/specs/2026-07-06-news-to-signals-pipeline-design.md

@@ -197,7 +197,7 @@ A machine-readable JSON Schema ships at `Heartbeat/schemas/signals-v1.schema.jso
 
 ## 5. Configuration (the v1 tuning surface)
 
-Env vars read by `news_signals.py` (module constants as defaults). This is the surface future user-facing tuning builds on:
+Env vars read by `news_signals.py` (module constants as defaults). This is the surface future user-facing tuning builds on. All are pipeline-only except `SIGNALS_MAX_FILE_MB`, which the Django API reads too (see its row):
 
 | Var | Default | Meaning |
 |-----|---------|---------|
@@ -209,7 +209,7 @@ Env vars read by `news_signals.py` (module constants as defaults). This is the s
 | `SIGNALS_THRESHOLD` | `0.20` | ± threshold for bullish/bearish label (40/60 band, empirically backed; was 0.15) |
 | `SIGNALS_DAMP_CAP` | `0.7` | Max \|score\| when under-corroborated |
 | `SIGNALS_DAMP_MIN_ARTICLES` | `2` | Corroboration needed for \|score\| > damp cap |
-| `SIGNALS_MAX_FILE_MB` | `10` | Reject oversized items files |
+| `SIGNALS_MAX_FILE_MB` | `10` | Reject oversized items files. **Two readers as of 2026-07-14:** `news_signals.py` (via `load_config`) and the Django API (via `settings.RAW_ITEMS_MAX_FILE_MB`, which `GET /api/news/items/` enforces when validating a batch). Deliberately one operator knob — raising it for the pipeline without raising it for the API would 404 a batch the pipeline happily accepted. Set it in `.env.production` alongside the heartbeat's own env file; the two defaults are pinned together by `Heartbeat/tests/test_port_parity.py`. |
 | `SIGNALS_STALENESS_ALERT_H` | `20` | Canary threshold (§6-C). Tuned, not arbitrary: the daily canary check runs 2 h after the daily beat, so a single fully-missed day leaves the newest artifact ~25.5 h old at the *next* day's check — a 30 h threshold would not cross that (it silently absorbs one entire missed day, only firing after a second consecutive miss); 20 h does. |
 
 ## 6. Failure policy (every mode decided)

Heartbeat/.env.heartbeat.example

@@ -36,6 +36,9 @@ DISCORD_CHANNEL_ID=
 # SIGNALS_DAMP_CAP=0.7
 # SIGNALS_DAMP_MIN_ARTICLES=2
 # SIGNALS_MAX_FILE_MB=10
+# NOTE: also read by the Django API (settings.RAW_ITEMS_MAX_FILE_MB) for
+# GET /api/news/items/ — this knob now has two readers. Update both env
+# files together, or the API will 404 a batch the pipeline accepted.
 # Rolling retention cap: keep only the N most recent signals-*.json artifacts;
 # older ones are pruned after each successful sweep. Bounds signals/ growth.
 # SIGNALS_KEEP_N=14

Heartbeat/README.md

@@ -14,10 +14,14 @@ fallback), writes a digest log, and posts it to a Discord channel.
 
 The production droplet is 1 vCPU / 2 GB RAM with no pip on the host, so
 `news_heartbeat.py` is a **single stdlib-only Python file** — deploying is
-copying one file; running costs ~30 MB RSS. It is fully decoupled from the
-`fingpt-api` container. Summaries are short, attributed, and always link out to
-the source article (Yahoo ToS posture: orchestration, not raw-data
-redistribution).
+copying one file; running costs ~30 MB RSS. It still runs as an independent
+systemd service and never imports or depends on the Django app — the coupling
+is one-directional: the `fingpt-api` container mounts its `digests/` output
+read-only (as `RAW_ITEMS_DIR`) to serve `GET /api/news/items/`, and
+`Heartbeat/tests/test_port_parity.py` pins the API's vendored copy of the
+validation gate to this script. Summaries are short, attributed, and always
+link out to the source article (Yahoo ToS posture: orchestration, not
+raw-data redistribution).
 
 ## Run
 

Heartbeat/news_signals.py

@@ -24,7 +24,7 @@
 from datetime import date, datetime, timezone
 from pathlib import Path
 
-VERSION = "2026-07-11.2"
+VERSION = "2026-07-14.1"
 SCHEMA_VERSION = 1
 PROMPT_VERSION = 1
 
@@ -45,6 +45,10 @@
 REQUIRED_FIELDS = ("guid", "title", "link", "source", "published", "score")
 FIELD_CAPS = {"title": 500, "description": 5000, "link": 2000, "source": 200,
               "guid": 200}
+# Required fields that must be strings. A malformed type drops the story — same
+# stance as the numeric parse in validation_gate — so a corrupt field can never
+# reach clean_text as a non-str and can never poison the whole batch.
+TEXT_REQUIRED_FIELDS = ("guid", "title", "link", "source")
 # Caps for the LLM/exception-derived output fields, pinned by the signals-v1
 # schema's maxLength values (headline/source/url are covered by FIELD_CAPS:
 # they pass through from the validated input). Same single-source-of-truth
@@ -109,9 +113,13 @@ def load_env_file(path):
 
 
 def clean_text(s, cap):
+    # non-str (incl. None) collapses to "": the gate must never raise on a
+    # malformed field type. Required-field types are checked in validation_gate;
+    # this keeps optional and LLM-derived callers total on their own.
+    s = s if isinstance(s, str) else ""
     # line boundaries first — CONTROL_RE would strip \v/\f/\x1c-\x1e to
     # nothing and fuse the words they separated
-    s = _LINEBREAK_RE.sub(" ", unicodedata.normalize("NFC", s or ""))
+    s = _LINEBREAK_RE.sub(" ", unicodedata.normalize("NFC", s))
     s = CONTROL_RE.sub("", s)
     s = s.replace("NEWS_DATA", "")  # marker token can never come from the feed
     return s[:cap]
@@ -161,7 +169,8 @@ def load_config():
 
 def validation_gate(path, max_file_mb):
     """Input trust boundary (spec §7.1). Batch-level defects raise ValueError
-    (poison pill, §6.1); a bad `published` drops only that story."""
+    (poison pill, §6.1); a bad `published`, a malformed numeric, or a
+    non-str TEXT_REQUIRED_FIELDS value drops only that story."""
     st = path.stat()
     if st.st_size > max_file_mb * 1024 * 1024:
         raise ValueError(f"file exceeds {max_file_mb}MB")
@@ -181,14 +190,23 @@ def validation_gate(path, max_file_mb):
             continue  # malformed numeric types: drop the story, keep the batch
         if not (lo <= published <= hi):
             continue  # forged/insane epoch: drop the story, keep the batch
+        if not all(isinstance(story[f], str) for f in TEXT_REQUIRED_FIELDS):
+            continue  # malformed text types: drop the story, keep the batch
         story["published"] = published
         story["title"] = clean_text(story["title"], FIELD_CAPS["title"])
         story["description"] = clean_text(story.get("description", ""),
                                           FIELD_CAPS["description"])
         story["source"] = clean_text(story["source"], FIELD_CAPS["source"])
-        story["guid"] = clean_text(str(story["guid"]), FIELD_CAPS["guid"])
-        story["link"] = clean_text(str(story["link"]), FIELD_CAPS["link"])
-        story["tickers"] = [t.upper() for t in story.get("tickers", [])]
+        story["guid"] = clean_text(story["guid"], FIELD_CAPS["guid"])
+        story["link"] = clean_text(story["link"], FIELD_CAPS["link"])
+        # non-list/non-string tickers dropped, never crashed: a corrupt
+        # "tickers":[123] must not .upper() -> AttributeError, which the only
+        # caller (process_batch's ValueError-only except) would not catch and
+        # which would abort the whole sweep; a bare "tickers":"AAPL" must not
+        # char-iterate to ['A','A','P','L'].
+        raw_tickers = story.get("tickers", [])
+        story["tickers"] = ([t.upper() for t in raw_tickers if isinstance(t, str)]
+                            if isinstance(raw_tickers, list) else [])
         stories.append(story)
     return stories
 

Heartbeat/tests/fixtures/signals-fixture.json

@@ -2,7 +2,7 @@
   "schema_version": 1,
   "profile": "default",
   "generated_at": "2026-07-06T15:00:00+00:00",
-  "generator": "news_signals.py/2026-07-11.2",
+  "generator": "news_signals.py/2026-07-14.1",
   "model": "gpt-4o-mini",
   "prompt_version": 1,
   "source_items": "items-fixture.jsonl",

Heartbeat/tests/test_news_signals.py

@@ -83,6 +83,27 @@ def test_clean_text_caps_and_handles_none(self):
         self.assertEqual(ns.clean_text(None, 5), "")
         self.assertEqual(ns.clean_text("x" * 10, 5), "xxxxx")
 
+    def test_clean_text_is_total_non_str_types_collapse_to_empty(self):
+        # D1: clean_text's isinstance guard replaces the old "s or ''"
+        # fallback and must never raise on a non-str input.
+        import news_signals as ns
+        for bad in (True, 123, [], {}):
+            with self.subTest(bad=bad):
+                self.assertEqual(ns.clean_text(bad, 100), "")
+
+    def test_clean_text_no_regression_for_every_input_the_old_fallback_handled(self):
+        # D1 strict-superset proof: "s or ''" returned "" for every falsy
+        # input and passed through every truthy str unchanged (modulo the
+        # existing cleanup). Pin the exact old behavior for the falsy/edge
+        # inputs that worked before this change, so a future edit can't
+        # silently regress them.
+        import news_signals as ns
+        self.assertEqual(ns.clean_text("", 100), "")
+        self.assertEqual(ns.clean_text(None, 100), "")
+        self.assertEqual(ns.clean_text(0, 100), "")
+        self.assertEqual(ns.clean_text(False, 100), "")
+        self.assertEqual(ns.clean_text("0", 100), "0")
+
     def test_clean_text_collapses_embedded_newlines_and_tabs(self):
         # A multi-line feed title must not let forged-looking lines reach
         # logs or artifact consumers: headline/rationale/source/guid are
@@ -257,6 +278,58 @@ def test_link_hygiene_strips_bidi(self):
         stories = ns.validation_gate(p, 10)
         self.assertNotIn("\u202e", stories[0]["link"])
 
+    def test_non_string_ticker_elements_are_dropped_not_crashed(self):
+        # F2b regression: [t.upper() for t in tickers] would raise
+        # AttributeError on a non-str element ('int' object has no attribute
+        # 'upper'). run_sweep's ONLY except clause around process_batch is
+        # `except ValueError as exc:  # poison pill` (news_signals.py, the
+        # run_sweep loop) \u2014 an AttributeError is NOT a ValueError, so it
+        # would propagate uncaught and abort the whole sweep instead of just
+        # dropping this one story. Must never raise.
+        bad = make_story(guid="bad", tickers=[123, "msft", None, "nvda"])
+        ok = make_story(guid="ok")
+        p = write_items(self.td, [bad, ok])
+        stories = ns.validation_gate(p, 10)
+        self.assertEqual([s["guid"] for s in stories], ["bad", "ok"])
+        self.assertEqual(stories[0]["tickers"], ["MSFT", "NVDA"])
+
+    def test_bare_string_and_non_list_tickers_become_empty_list(self):
+        # A bare "tickers":"AAPL" must not char-iterate to ['A','A','P','L'].
+        cases = [
+            ("str", "AAPL"),
+            ("none", None),
+            ("dict", {"a": 1}),
+        ]
+        for name, val in cases:
+            with self.subTest(tickers=name):
+                p = write_items(self.td, [make_story(guid=name, tickers=val)],
+                                 name=f"items-{name}.jsonl")
+                stories = ns.validation_gate(p, 10)
+                self.assertEqual(stories[0]["tickers"], [])
+
+    def test_non_string_required_text_field_drops_story_not_batch(self):
+        # F2a's Heartbeat twin: title/source/guid/link must be str, else the
+        # story is dropped (D2) rather than reaching clean_text with a type
+        # that would raise inside unicodedata.normalize.
+        bad_title = make_story(guid="bad-title", title=True)      # JSON bool
+        bad_source = make_story(guid="bad-source", source=123)    # JSON number
+        bad_guid = make_story(guid=456, title="fine")
+        bad_link = make_story(guid="bad-link", link=789)
+        ok = make_story(guid="ok")
+        p = write_items(self.td, [bad_title, bad_source, bad_guid, bad_link, ok])
+        stories = ns.validation_gate(p, 10)
+        self.assertEqual([s["guid"] for s in stories], ["ok"])
+
+    def test_non_string_description_blanks_field_but_keeps_story(self):
+        # description is optional and only reaches clean_text's own isinstance
+        # guard (D1) \u2014 it must NOT be dropped like a TEXT_REQUIRED_FIELDS
+        # violation, just blanked.
+        s = make_story(guid="ok", description=True)
+        p = write_items(self.td, [s])
+        stories = ns.validation_gate(p, 10)
+        self.assertEqual([s["guid"] for s in stories], ["ok"])
+        self.assertEqual(stories[0]["description"], "")
+
 
 class TestSubjectGate(unittest.TestCase):
     def test_symbol_token_in_headline_is_subject(self):