Repository navigation
Lifecycle events in Web UI; identity-based restart detection - #257
Merged
Merged
Conversation
added 3 commits
May 19, 2026 13:31
…ristic Sidebar.tsx toasted "pg_doorman restarted" every time overview totals fell, but the totals are summed across the live pool set. RELOAD that removes dynamic pools and the dynamic-pool GC both drop pools from the map, so the sum falls without the process going anywhere. The operator at the UI saw a restart toast on every routine reload. Identity-based detection replaces it. OverviewDto carries started_at_ms next to pid; the new useProcessIdentity hook toasts once when either field changes between polls. Counter rollback now just skips a rate-tick — no toast, no misleading rate spike on the next sample either. The events ring (src/admin/events.rs) grows two new targets, PROCESS_START and CONFIG_VALIDATION_ERROR, and acquires three new push sites: at the end of run_server setup, in the SIGHUP reload error path, and in the admin RELOAD / /api/admin/reload error paths. A new useLifecycleEvents hook polls /api/events incrementally and toasts CONFIG_VALIDATION_ERROR as a red error toast — the deploy step that quietly fails is the one the operator needs to see. A persistent LifecycleBanner surfaces shutdown_in_progress and migration_in_progress because banners do not vanish like toasts. Two BDD scenarios pin the surface contract: /api/events emits a PROCESS_START entry on boot, and /api/overview carries started_at_ms alongside pid. What is NOT in this PR (follow-up): DYNAMIC_POOL_DROP / POOL_MIGRATION / BINARY_UPGRADE_BEGIN/END events, the operator events drawer on Overview, and per-target colour mapping in the Sparkline annotations.
The first commit of this PR landed identity-based restart detection and basic lifecycle events but left four operator-visible holes that the DevOps review flagged: - CONFIG_VALIDATION_ERROR was a 10-second toast — easy to miss after the operator alt-tabs to a terminal. Now it is a persistent red banner that stays until a successful RELOAD clears it. The backend rate-limits the push to 1/s/target so a SIGHUP loop with a bad config no longer fills the 1024-entry ring with duplicates. - LifecycleBanner went stale silently — TanStack Query kept the last successful /api/overview for 5 minutes after the pooler died, so "draining" stayed on screen long after pg_doorman was gone. Now the banner switches to an "unreachable, last contact 23s ago" state when the dataUpdatedAt timestamp exceeds 15s or the query errors. - isRealRestart compared only pid and started_at_ms. On a host with 30+ days uptime a PID recycle plus an identical lazy-read start time could in principle slip through. Adding uptime_seconds < prev closes that loophole — a restarted process always has a lower uptime than the cached one. - SIGHUP that re-parsed identically used to leave no trace. Now it emits a RELOAD entry with "config unchanged" so the audit timeline has one event per signal, no more. Tailwind tokens text-warning-strong / text-accent-strong did not exist in tailwind.css and were silently dropped; the banner is now on the base hues (text-warning, text-accent, text-danger). A separate cleanup hoists STARTED_AT_MS from process.rs to app/server.rs so both overview and process collectors share one LazyLock instead of duplicating the conversion. /api/events and /api/overview now ship Cache-Control: no-store so intermediate proxies cannot collapse two consecutive polls into the same response. A new unit test covers the rate-limit collapse behaviour (rate_limited_collapses_burst_per_target).
Cargo.toml + Cargo.lock move to 3.10.0 alongside the existing 3.10.0 changelog header. Two new subsections cover what landed since the 3.10.0 stub was opened: - Eviction visibility for prepared-statement caches (PR #256). - Web UI lifecycle events: identity-based restart detection, PROCESS_START / CONFIG_VALIDATION_ERROR ring entries, persistent banner for shutdown / migration / validation error / unreachable (this PR).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
OverviewDto.started_at_msnext topid, the newuseProcessIdentityhook toasts once per real pid/start-time change, counter rollback just skips a rate-tick.PROCESS_STARTandCONFIG_VALIDATION_ERRORinsrc/admin/events.rs, with push sites inrun_server, the SIGHUP reload error path, and the admin RELOAD //api/admin/reloaderror paths. The newuseLifecycleEventshook polls/api/eventsincrementally and surfacesCONFIG_VALIDATION_ERRORas a red toast — the deploy step that quietly fails is the one the operator needs to see.LifecycleBannersurfacesshutdown_in_progressandmigration_in_progressas a persistent strip (banners don't vanish like toasts).Why
Operators were getting "restarted" toasts on routine reloads and dynamic-pool churn. The toast was wrong: the process was still up (sidebar
pidanduptimeconfirmed it), only the counter sum legitimately fell because pools left the live set. Counter-based restart detection is a category error — pid + start-time is the source of truth, the same waypgbouncer_exporterandhatopdo it.What is NOT in this PR (follow-up)
DYNAMIC_POOL_DROP/POOL_MIGRATION/BINARY_UPGRADE_BEGIN/ENDeventsTest plan
cargo fmt --checkcleancargo clippy --lib --tests --bins -- --deny warningscleancargo test --lib: 1135 pass, 1 pre-existing flaky (web::metrics::tests::test_prometheus_server_basic— fails 3/3 on clean master locally, not our regression)npm run typecheck+npm run lint+npm run build(frontend dist regenerated and committed)@web-uimatrix tag:/api/eventscarriesPROCESS_START,/api/overviewcarriesstarted_at_mscfg errorchip persists on next page navigation; RELOAD via psql → no restart toast; restart pg_doorman → restart toast fires once🤖 Generated with Claude Code