fix(health): add finite stale-content grace (#7577)

* docs(health): plan stale-content grace

* test(health): reproduce stale-content grace gap

* test(docs): define graced stale-content summary

* test(monitor): reproduce stale-content grace alert

* fix(health): add finite stale-content grace

* test(health): reproduce per-entity grace gap

* test(health): cover count-driven grace clock

* Address PR review feedback (#7577)

Extend stale-content grace to per-entity checks and keep count-driven grace on one stable Redis deadline.

* fix(health): make stale-content grace a genuine one-shot window

Review of #7577 found the grace was not finite for two reachable source
classes, and that its production wiring had no end-to-end coverage.

The deadline was re-derived from the live observation on every sweep. A feed
that keeps publishing while staying just past its budget therefore advanced
its own deadline forever and could never reach the warning bucket, and a
source that changed which evidence shape applied was granted a second window
without ever recovering. Both alarms were silent for the whole time, since the
scheduled monitor's ceiling was satisfied too.

Redis is now the single source of truth for the deadline in both modes. Every
graced source claims one anchor with HSETNX and republishes that stored value,
so the window cannot slide or be re-granted. The claim runs after
classification and is gated on the status a key actually received, so a
higher-precedence failure (REDIS_PARTIAL, EMPTY, STALE_SEED) can no longer
burn the source's single anchor while publishing no grace at all.

Also from the review:

- Split the grace pipeline by urgency. Only the claim decides what the
  response publishes; the recovery cleanup now goes through ctx.waitUntil
  instead of adding a round trip to nearly every sweep.
- Give the grace hash a refreshed TTL so retired registry names and
  per-deploy preview keys reap themselves.
- Drop the seedMetaByName precompute, which was provably identical to the
  fallback it bypassed.
- Collapse the three copy-pasted deadline branches into one declared table.
- Restore the #3845 provenance comments and document why the window is
  source-agnostic.
- Re-bound the docs gate on staleContent (<= ok + warn, still true) instead
  of dropping the check.
- Give the monitor ceiling explicit clock-skew slack, and keep graced entries
  visible in its report rather than filtering them into silence.

Tests: the helpers were the only thing covered, so the wiring could have been
deleted green. Adds handleHealth-level coverage for the claim, the
fail-closed path, and the cached-snapshot guard.

Author: koala73

api/health.js

@@ -57,7 +57,7 @@ Primary health endpoint. Checks all Redis-backed data keys and seed freshness me
 }
 ```
 
-`summary` fields: `total` is one entry per probed key, and grows as panels are added — read it from your own response rather than from this page. `warn` **excludes** on-demand-empty keys — those are surfaced separately as `onDemandWarn` so they don't drive the overall verdict to `WARNING`. `staleContent` is a **subset** of `warn` (fresh seeder but the upstream feed stopped advancing), and `rolloutPending` is another **subset** of `warn` (a newly deployed schema still inside its bounded deploy-before-cron window). Only `crit` (`EMPTY`/`EMPTY_DATA`) drives `DEGRADED`/`UNHEALTHY`.
+`summary` fields: `total` is one entry per probed key, and grows as panels are added — read it from your own response rather than from this page. `warn` **excludes** on-demand-empty keys — those are surfaced separately as `onDemandWarn` so they don't drive the overall verdict to `WARNING`. `staleContent` counts all `STALE_CONTENT` diagnostics. It can be greater than `warn` while a source is inside its three-hour `staleContentGraceUntil` window. The source stays visible in `problems`, but it counts in `ok` until that deadline. `rolloutPending` is a **subset** of `warn` (a newly deployed schema still inside its bounded deploy-before-cron window). Only `crit` (`EMPTY`/`EMPTY_DATA`) drives `DEGRADED`/`UNHEALTHY`.
 
 With `?compact=1`, the `checks` object is replaced by `problems` containing only non-OK keys.
 
@@ -97,7 +97,7 @@ Keys are grouped into three tiers that determine alert severity:
 | `SOURCE_BLOCKED` | Green | The Japan MOD adapter proved no transport path reaches the publisher while retaining fresh reviewed records — either an upstream HTTP 403 after a successful proxy CONNECT (`HTTP_403`), or a target-scoped proxy CONNECT refusal corroborated by a successful control tunnel (`PROXY_TARGET_FORBIDDEN`). Uncorroborated CONNECT refusals, stale, empty, or unreviewed states still fail closed |
 | `COVERAGE_MARGIN_LOW` | Green | Every configured pool clears its floor, but at least one is within `poolCountMargin` of it (`scorecardFiveFactor`, margin 10). Informational, and deliberately not a warning: this probe's health minimums equal the producer's publication floors, so a passing cohort sitting a few records above the line is the normal steady state. Per-pool headroom is on `poolCoverageMargin` for every status, in the full payload (`?compact=1` omits `checks`); an actual breach is still `COVERAGE_PARTIAL` |
 | `STALE_SEED` | Warn | Data present but `seed-meta` age exceeds `maxStaleMin` |
-| `STALE_CONTENT` | Warn | Seeder is fresh but the upstream content stopped advancing (frozen feed); counted in `summary.staleContent` |
+| `STALE_CONTENT` | Warn after grace | Seeder is fresh but the upstream content stopped advancing or has no usable timestamp. It is counted in `summary.staleContent` and remains in `problems` during the three-hour grace. `staleContentGraceUntil` gives the deadline. At and after that time, it counts in `summary.warn` as usual. |
 | `COVERAGE_PARTIAL` | Warn | Aggregate records or a required subgroup is below the key's declared coverage floor (for example, 139/174 PortWatch countries or an empty prediction-market pool) |
 | `COVERAGE_DEGRADED` | Warn | The producer's own coverage diagnostics are missing, or its completion ratio is below the key's `minSuccessRate` (for example, a consumer-price market that completed 4 of 12 retailer pages) |
 | `ROLLOUT_PENDING` | Warn | A newly deployed schema whose producer has not reached its first scheduled run yet. **Bounded**: the health runtime persists a deployment-relative `rolloutPendingUntil` deadline in Redis, and the state becomes `EMPTY` (crit) once that passes or once the producer writes its durable activation marker. Counted in `summary.rolloutPending` |

docs/health-endpoints.mdx

@@ -56,7 +56,7 @@ description: "World Monitor 提供两个健康检查端点用于持续监控数
 }
 ```
 
-`summary` 字段：`total` 是每个探测键的一个条目，并随面板增加而增长 —— 请以你自己响应中的数值为准，而非本页示例。`warn` **不包括**按需为空的键 —— 这些键作为 `onDemandWarn` 单独显示，因此它们不会将总体判定推向 `WARNING`。`staleContent` 是 `warn` 的一个**子集**（种子器新鲜但上游订阅源停止推进），`rolloutPending` 是 `warn` 的另一个**子集**（新部署的 schema 仍处于其有界的「部署先于定时任务」窗口内）。只有 `crit`（`EMPTY`/`EMPTY_DATA`）会驱动 `DEGRADED`/`UNHEALTHY`。
+`summary` 字段：`total` 是每个探测键的一个条目，并随面板增加而增长 —— 请以你自己响应中的数值为准，而非本页示例。`warn` **不包括**按需为空的键 —— 这些键作为 `onDemandWarn` 单独显示，因此它们不会将总体判定推向 `WARNING`。`staleContent` 统计所有 `STALE_CONTENT` 诊断。在数据源处于三小时的 `staleContentGraceUntil` 宽限期内时，该值可以大于 `warn`。数据源仍显示在 `problems` 中，但在截止时间前计入 `ok`。`rolloutPending` 是 `warn` 的一个**子集**（新部署的 schema 仍处于其有界的「部署先于定时任务」窗口内）。只有 `crit`（`EMPTY`/`EMPTY_DATA`）会驱动 `DEGRADED`/`UNHEALTHY`。
 
 使用 `?compact=1` 时，`checks` 对象会被 `problems` 替换，仅包含非 OK 的键。
 
@@ -82,7 +82,7 @@ consumer-prices-core 服务会将每个市场的抓取覆盖率快照写入 `con
 | `OK_CASCADE` | 绿色 | 键为空但级联组中的同级键有数据（例如，战区态势后备链） |
 | `NOT_CONFIGURED` | 绿色 | 本部署从未为该可选数据源适配器配置凭据（生产者写入 `sourceState: "unavailable"`，例如未配置 `SAM_GOV_API_KEY` 的全球招标 SAM.gov 适配器）。这不是故障，也不计入问题；添加凭据后的首次运行即转为 `OK` |
 | `STALE_SEED` | 警告 | 数据存在但 `seed-meta` 年龄超过 `maxStaleMin` |
-| `STALE_CONTENT` | 警告 | 种子器新鲜但上游内容停止推进（冻结的订阅源）；计入 `summary.staleContent` |
+| `STALE_CONTENT` | 宽限期后为警告 | 种子器新鲜，但上游内容停止推进或没有可用时间戳。三小时宽限期内仍计入 `summary.staleContent` 并显示在 `problems` 中。`staleContentGraceUntil` 给出截止时间。到达或超过该时间后，它照常计入 `summary.warn`。 |
 | `COVERAGE_PARTIAL` | 警告 | 总记录数或必需子组低于该键声明的覆盖率下限（例如 139/174 个 PortWatch 国家，或某个预测市场池为空） |
 | `COVERAGE_DEGRADED` | 警告 | 生产者自身的覆盖率诊断缺失，或其完成率低于该键的 `minSuccessRate`（例如某个消费价格市场仅完成 12 个零售商页面中的 4 个） |
 | `ROLLOUT_PENDING` | 警告 | 新部署的 schema，其生产者尚未到达首次计划运行。**有界**：健康检查运行时在 Redis 中持久化相对于部署时间的 `rolloutPendingUntil` 截止时间；一旦该时间过去，或生产者写入其持久激活标记，状态即变为 `EMPTY`（严重）。计入 `summary.rolloutPending` |

docs/plans/2026-09-03-001-fix-stale-content-grace-plan.md

@@ -64,19 +64,69 @@ export function isOnDemandProblem(problem) {
 // this, a single bad `rolloutPendingUntil` — a registry bug, a bad merge, a
 // tampered response — would silence these keys in CI effectively forever.
 export const MAX_ROLLOUT_WINDOW_MS = 24 * 60 * 60 * 1000;
+// Mirrors STALE_CONTENT_GRACE_MS in api/health.js (the publisher), plus slack.
+// tests/seed-freshness-monitor.test.mjs pins the relationship so the two cannot
+// silently drift apart.
+//
+// The slack is load-bearing, not padding. The publisher stamps a deadline at
+// most exactly one window ahead of ITS clock; this gate compares that value
+// against a DIFFERENT machine's clock. With a bare 3h ceiling the entire
+// tolerance for skew is network latency, so a CI runner a few seconds behind
+// Vercel would reject a perfectly legitimate grace and report it as blocking.
+// Five minutes absorbs realistic skew while still refusing the thing the
+// ceiling exists to refuse: a registry bug or tampered response claiming a
+// window far longer than the publisher can legally mint.
+export const STALE_CONTENT_GRACE_SKEW_SLACK_MS = 5 * 60 * 1000;
+export const MAX_STALE_CONTENT_GRACE_MS = 3 * 60 * 60 * 1000 + STALE_CONTENT_GRACE_SKEW_SLACK_MS;
 
-export function isRolloutPendingProblem(problem, now = Date.now()) {
-  if (problem?.status !== 'ROLLOUT_PENDING') return false;
-  const until = Date.parse(problem?.rolloutPendingUntil ?? '');
+function hasActiveBoundedDeadline(raw, now, maxWindowMs) {
+  const until = Date.parse(raw ?? '');
   if (!Number.isFinite(until)) return false;
-  if (until - now > MAX_ROLLOUT_WINDOW_MS) return false;
+  if (until - now > maxWindowMs) return false;
   return now < until;
 }
 
+export function isRolloutPendingProblem(problem, now = Date.now()) {
+  if (problem?.status !== 'ROLLOUT_PENDING') return false;
+  return hasActiveBoundedDeadline(problem.rolloutPendingUntil, now, MAX_ROLLOUT_WINDOW_MS);
+}
+
+export function isStaleContentGraceProblem(problem, now = Date.now()) {
+  if (problem?.status !== 'STALE_CONTENT') return false;
+  return hasActiveBoundedDeadline(
+    problem.staleContentGraceUntil,
+    now,
+    MAX_STALE_CONTENT_GRACE_MS,
+  );
+}
+
+/**
+ * Sources that are diagnosed STALE_CONTENT but still inside their bounded grace.
+ *
+ * These are deliberately NOT operational failures yet — that is the whole point
+ * of the grace — but filtering them out of the run entirely would make a green
+ * report indistinguishable from one where nothing is wrong at all. Surfacing
+ * them separately keeps "three feeds are mid-grace, alerting at 14:05Z" visible
+ * to whoever reads the run.
+ */
+export function findGracedStaleContent(payload, now = Date.now()) {
+  return Object.entries(payload.problems ?? {})
+    .filter(([, problem]) => isStaleContentGraceProblem(problem, now))
+    .map(([name, problem]) => ({
+      name,
+      status: problem?.status ?? 'UNKNOWN',
+      graceUntil: problem?.staleContentGraceUntil ?? null,
+    }));
+}
+
 export function findOperationalProblems(payload, now = Date.now()) {
   validateCompactHealthPayload(payload);
   return Object.entries(payload.problems ?? {})
-    .filter(([, problem]) => !isOnDemandProblem(problem) && !isRolloutPendingProblem(problem, now))
+    .filter(([, problem]) => (
+      !isOnDemandProblem(problem)
+      && !isRolloutPendingProblem(problem, now)
+      && !isStaleContentGraceProblem(problem, now)
+    ))
     .map(([name, problem]) => ({
       name,
       status: problem?.status ?? 'UNKNOWN',
@@ -430,6 +480,11 @@ async function main() {
   if (outputPath) writeFileSync(outputPath, `${JSON.stringify(observation, null, 2)}\n`);
   const { report } = observation;
   for (const line of report.info) console.log(line);
+  // Non-blocking, but never silent: a green run should still say which feeds
+  // are mid-grace and when they start counting as warnings.
+  for (const graced of findGracedStaleContent(payload)) {
+    console.log(`in grace: ${graced.name} (${graced.status}, alerting at ${graced.graceUntil})`);
+  }
   for (const line of report.errors) console.error(line);
   if (report.failed) process.exitCode = 1;
 }

docs/zh/health-endpoints.mdx

@@ -1622,8 +1622,11 @@ function healthSummaryDocSources(pages = null) {
 //   - `total` equals the registry size. True of any response, whatever its status.
 //   - the buckets sum to `total`. `summary.warn` is already net of onDemandWarn
 //     (api/health.js computes `realWarnCount = counts.warn - counts.onDemandWarn`),
-//     and staleContent/rolloutPending are documented SUBSETS of warn, so the
-//     partition is exactly ok + warn + onDemandWarn + crit. This is what caught
+//     and rolloutPending is a documented SUBSET of warn, so the partition is
+//     exactly ok + warn + onDemandWarn + crit. staleContent is diagnostic: a
+//     graced entry counts in `ok`, so it is NOT a subset of warn — but every
+//     STALE_CONTENT entry still lands in exactly one of ok/warn, which keeps
+//     `staleContent <= ok + warn` a real bound worth enforcing. This is what caught
 //     the pre-#6300 api-platform.mdx body, which showed a concrete "HEALTHY"
 //     alongside 5 warns.
 function validateHealthSummaryDocs(stats, docs = null) {
@@ -1663,12 +1666,20 @@ function validateHealthSummaryDocs(stats, docs = null) {
           `${where}: ok + warn + onDemandWarn + crit = ${partition}, which must equal total (${counts.total})`,
         );
       }
-      for (const subset of ['staleContent', 'rolloutPending']) {
-        if (counts[subset] > counts.warn) {
-          failures.push(
-            `${where}: ${subset} (${counts[subset]}) is documented as a subset of warn (${counts.warn})`,
-          );
-        }
+      if (counts.rolloutPending > counts.warn) {
+        failures.push(
+          `${where}: rolloutPending (${counts.rolloutPending}) is documented as a subset of warn (${counts.warn})`,
+        );
+      }
+      // staleContent is no longer a subset of warn (a graced entry counts in
+      // ok), but it is still bounded: STATUS_COUNTS maps STALE_CONTENT to warn
+      // and healthStatusBucket only ever overrides that to ok, so an example
+      // claiming more stale-content diagnoses than there are ok+warn keys is
+      // arithmetically impossible and must not ship.
+      if (counts.staleContent > counts.ok + counts.warn) {
+        failures.push(
+          `${where}: staleContent (${counts.staleContent}) exceeds ok + warn (${counts.ok + counts.warn}), which is impossible`,
+        );
       }
     });
   }

scripts/check-seed-freshness.mjs

@@ -186,11 +186,47 @@ describe('/api/health probed-key count doc gate (#6300)', () => {
     );
   });
 
-  it('catches a subset counter exceeding the warn bucket it is a subset of', () => {
+  it('allows graced stale-content diagnostics outside warn but keeps rolloutPending inside warn', () => {
+    assert.deepEqual(
+      validateHealthSummaryDocs(REAL_STATS, {
+        'docs/health-endpoints.mdx': docWithSummary({
+          total: REAL_TOTAL,
+          ok: REAL_TOTAL - 2,
+          warn: 2,
+          staleContent: 5,
+        }),
+      }),
+      [],
+    );
+
     const failures = validateHealthSummaryDocs(REAL_STATS, {
-      'docs/health-endpoints.mdx': docWithSummary({ total: REAL_TOTAL, ok: REAL_TOTAL - 2, warn: 2, staleContent: 5 }),
+      'docs/health-endpoints.mdx': docWithSummary({
+        total: REAL_TOTAL,
+        ok: REAL_TOTAL - 2,
+        warn: 2,
+        rolloutPending: 5,
+      }),
     });
-    assert.ok(failures.some((f) => /staleContent \(5\) is documented as a subset/.test(f)), failures.join(' | '));
+    assert.ok(failures.some((f) => /rolloutPending \(5\) is documented as a subset/.test(f)), failures.join(' | '));
+  });
+
+  it('still rejects a staleContent count larger than the buckets it can occupy', () => {
+    // Grace freed staleContent from being a subset of warn, but not from
+    // arithmetic: every STALE_CONTENT entry lands in ok or warn, so a documented
+    // example claiming more than ok + warn is impossible and must fail the gate.
+    const failures = validateHealthSummaryDocs(REAL_STATS, {
+      'docs/health-endpoints.mdx': docWithSummary({
+        total: REAL_TOTAL,
+        ok: 3,
+        warn: 2,
+        crit: REAL_TOTAL - 5,
+        staleContent: 6,
+      }),
+    });
+    assert.ok(
+      failures.some((f) => /staleContent \(6\) exceeds ok \+ warn \(5\)/.test(f)),
+      failures.join(' | '),
+    );
   });
 
   it('fails when a known page silently drops its example', () => {