Skip to content

Fix front-coalesce query blow-up and statement_timeout coverage gaps - #264

Merged
SiteRelEnby merged 1 commit into
mainfrom
fix/front-coalesce-scaling
Aug 18, 2026
Merged

Fix front-coalesce query blow-up and statement_timeout coverage gaps#264
SiteRelEnby merged 1 commit into
mainfrom
fix/front-coalesce-scaling

Conversation

@SiteRelEnby

Copy link
Copy Markdown
Contributor

What

Fixes a production-stability bug where computing "fronting since" on a large imported front history could take the whole database down, and closes two statement-timeout coverage gaps that turned a 30-second query failure into a multi-minute outage.

The query

With coalesce-contiguous-fronts enabled, the walk that finds how far back a member's unbroken fronting run extends was a recursive CTE joining prev.ended_at = chain.started_at with UNION ALL. That enumerates paths, not states: on an imported history whose switch boundaries share timestamps (PluralKit exports round to the second, so thousands collide), every step matches multiple predecessors and the intermediate set grows multiplicatively per level. The depth cap bounded depth, not width. A genuine ~14k-front imported history recently blew this up into ~19 GB of Postgres query temp, filling the volume and briefly failing queries for every user.

The replacement is a set-based gaps-and-islands pass: one sorted window scan over the fronting members' own entries, where a run breaks only when the running MAX of ended_at falls strictly before the next started_at (an open front reaches infinity). Cost is O(n log n) in the member's own front count, no recursion, no width explosion.

Semantics are identical wherever fronts do not overlap, which is all live-written data - the pre-existing coalesce test suite passes unchanged. One deliberate refinement: overlapping fronts for the same member now merge into a single continuous run, where the old exact-boundary walk under-counted (a member continuously fronting across overlapping imported entries now gets the earlier, correct "since"). The walk-back depth cap is gone with nothing to need it; member_since_capped stays in the API for compatibility and is always empty.

The timeout

Two gaps, found root-causing why the incident query ran ~13 minutes when the request cap is 30 seconds:

  • The realtime stream endpoint builds its snapshots in self-managed sessions (deliberately, so no pooled connection is held for the stream's lifetime) - and those sessions never carried the per-request statement_timeout. The uncapped stream twin of the coalesce query is what kept spilling long after the /current copy had been correctly killed at 30 s.
  • get_db applied SET LOCAL statement_timeout once per session; SET LOCAL is transaction-scoped, so the first mid-request commit silently dropped the cap for the rest of that request.

Sessions now opt into the cap via session.info, and an after_begin listener re-applies SET LOCAL at the start of every transaction. A new request_session() context manager carries the request-tier cap for paths that cannot use Depends(get_db), and the stream endpoint's three sessions use it. Background jobs keep their separate (default unlimited) ceiling - verified live: capped sessions hold the cap across a commit, bare job sessions stay at 0.

Tests

New regression fixture reproducing the incident shape, miniaturised: 4 duplicate fronts per level x 12 levels, all sharing exact boundary timestamps (~16.7M chain paths under the old query; instant under the new one), asserting the true chain start, a wall-clock bound, and an empty capped list. Plus overlap-merge, dense-history-with-a-real-gap, and member-in-two-open-fronts cases. Full behavioural config green: 1783 passed.

Docs

Changelog entries for both fixes, and SELFHOSTING.md gains a short "Postgres safety limits" section recommending temp_file_limit + log_temp_files, so a self-hosted instance's worst case is one failed query rather than a full data volume.

The coalesce-contiguous-fronts walk-back was a recursive CTE joining
prev.ended_at = chain.started_at with UNION ALL - it enumerates paths,
not states, so an imported history whose switch boundaries share
timestamps (PluralKit exports are second-rounded) fans out
multiplicatively per level. A genuine ~14k-front imported history
detonated it into ~19 GB of Postgres query temp on 2026-08-13, filling
the data volume and briefly failing queries for everyone. The depth cap
bounded depth, not width, so it never helped.

Replace it with a set-based gaps-and-islands pass: one sorted window
scan over the fronting members' own entries, a run breaking only where
the running MAX of ended_at falls strictly before the next started_at
(open fronts reach infinity). Cost is O(n log n) in the member's own
front count. Results are identical wherever fronts do not overlap;
overlapping fronts for the same member now deliberately merge into one
continuous run, which the exact-boundary walk under-counted. The depth
cap is gone; member_since_capped stays in the API for compatibility and
is always empty.

Separately, the per-request statement_timeout had two coverage gaps:
the SSE stream endpoint builds its snapshots in self-managed sessions
that never carried the cap (this is how the incident query kept
spilling for ~13 minutes after its /current twin was cancelled at 30 s),
and get_db applied SET LOCAL once per session, so a mid-request commit
silently dropped the cap for the rest of the request. Sessions now opt
into the cap via session.info and an after_begin listener re-applies
SET LOCAL on every transaction; a new request_session() carries the
request-tier cap for paths that cannot use Depends(get_db), and the
stream endpoint uses it. Background jobs remain uncapped by default.

Adds a dense shared-boundary regression fixture (the incident shape,
miniaturised: 4 duplicate fronts per level x 12 levels, ~16.7M paths
under the old query) plus overlap-merge, dense-with-gap, and
multiple-open-front cases, and documents recommended Postgres
temp_file_limit / log_temp_files settings for self-hosters.
@SiteRelEnby
SiteRelEnby enabled auto-merge August 18, 2026 03:58
@SiteRelEnby
SiteRelEnby merged commit ffd81f9 into main Aug 18, 2026
6 checks passed
@SiteRelEnby
SiteRelEnby deleted the fix/front-coalesce-scaling branch August 18, 2026 04:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant