fix(health): add finite stale-content grace - #7577
Merged
Merged
Conversation
Contributor
|
Preview deployment for your docs. Learn more about Mintlify Previews.
💡 Tip: Enable Automations to automatically generate PRs for you. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 6277901ad3
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Extend stale-content grace to per-entity checks and keep count-driven grace on one stable Redis deadline.
Review of #7577 found the grace was not finite for two reachable source classes, and that its production wiring had no end-to-end coverage. The deadline was re-derived from the live observation on every sweep. A feed that keeps publishing while staying just past its budget therefore advanced its own deadline forever and could never reach the warning bucket, and a source that changed which evidence shape applied was granted a second window without ever recovering. Both alarms were silent for the whole time, since the scheduled monitor's ceiling was satisfied too. Redis is now the single source of truth for the deadline in both modes. Every graced source claims one anchor with HSETNX and republishes that stored value, so the window cannot slide or be re-granted. The claim runs after classification and is gated on the status a key actually received, so a higher-precedence failure (REDIS_PARTIAL, EMPTY, STALE_SEED) can no longer burn the source's single anchor while publishing no grace at all. Also from the review: - Split the grace pipeline by urgency. Only the claim decides what the response publishes; the recovery cleanup now goes through ctx.waitUntil instead of adding a round trip to nearly every sweep. - Give the grace hash a refreshed TTL so retired registry names and per-deploy preview keys reap themselves. - Drop the seedMetaByName precompute, which was provably identical to the fallback it bypassed. - Collapse the three copy-pasted deadline branches into one declared table. - Restore the #3845 provenance comments and document why the window is source-agnostic. - Re-bound the docs gate on staleContent (<= ok + warn, still true) instead of dropping the check. - Give the monitor ceiling explicit clock-skew slack, and keep graced entries visible in its report rather than filtering them into silence. Tests: the helpers were the only thing covered, so the wiring could have been deleted green. Adds handleHealth-level coverage for the claim, the fail-closed path, and the cached-snapshot guard.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
STALE_CONTENTno longer makes production health non-healthy at the exact content-freshness boundary. The diagnostic stays visible for a fixed three-hour grace period, withstaleContentGraceUntilshowing the deadline. At and after that deadline, the existing warning behavior applies without a new public status.Timestamped content derives its deadline from the observed content boundary. Content with no usable timestamp claims one Redis deadline with
HSETNX, so repeated fresh seeder metadata cannot renew the grace. Redis errors fail closed and keep the warning.newestItemAtnewestItemAt + maxContentAgeMin + 3 hoursnewestItemAtImplementation plan
Type of change
Affected areas
/api/*)Validation
The pre-change production response at
2026-09-03T02:49:35.461ZshowedWARNING,warn: 2, andstaleContent: 2.diseaseOutbreakshad fresh seeder metadata and content 170 minutes beyond its nine-day budget.temporalAnomalieshad four-minute seeder metadata andcontentAgeMin: null.Proof-first tests failed in six health assertions before the API implementation. The documentation contract and scheduled monitor tests also failed before their consumers were aligned.
npm run typecheck:api: passednpm run lint:boundaries: passednpm run docs:check: passednpm run test:data: 29,487 passed, 0 failed, 35 skippedgit diff --check: passedPost-deploy monitoring and validation
https://worldmonitor.app/api/health?compact=1after deployment and through the next health sweep.problemsasSTALE_CONTENT, include a futurestaleContentGraceUntil, incrementstaleContent, and not incrementwarn.warnand can make the top-level statusWARNING.[health]logs. It must ignore only active, parseable deadlines of at most three hours.Failure signals are a moving missing-timestamp deadline, grace longer than three hours, a monitor warning before the deadline, or no warning at and after expiry. Roll back the deployment if any signal occurs. The versioned Redis hash is safe to leave in place after rollback.
The timestamp-based disease-outbreak grace was already about 170 minutes old at reproduction time, so it can be expired by deployment. The temporal-anomaly grace starts on its first production health evaluation after deployment. Production acceptance remains unproved until this PR is merged and deployed.
Checklist
npm run typecheck:api)Documentation alignment
api/health.jsis the only writer and reader ofhealth:stale-content-grace:v1; the monitor consumes only the public deadline.