Fix front-coalesce query blow-up and statement_timeout coverage gaps - #264
Merged
Conversation
The coalesce-contiguous-fronts walk-back was a recursive CTE joining prev.ended_at = chain.started_at with UNION ALL - it enumerates paths, not states, so an imported history whose switch boundaries share timestamps (PluralKit exports are second-rounded) fans out multiplicatively per level. A genuine ~14k-front imported history detonated it into ~19 GB of Postgres query temp on 2026-08-13, filling the data volume and briefly failing queries for everyone. The depth cap bounded depth, not width, so it never helped. Replace it with a set-based gaps-and-islands pass: one sorted window scan over the fronting members' own entries, a run breaking only where the running MAX of ended_at falls strictly before the next started_at (open fronts reach infinity). Cost is O(n log n) in the member's own front count. Results are identical wherever fronts do not overlap; overlapping fronts for the same member now deliberately merge into one continuous run, which the exact-boundary walk under-counted. The depth cap is gone; member_since_capped stays in the API for compatibility and is always empty. Separately, the per-request statement_timeout had two coverage gaps: the SSE stream endpoint builds its snapshots in self-managed sessions that never carried the cap (this is how the incident query kept spilling for ~13 minutes after its /current twin was cancelled at 30 s), and get_db applied SET LOCAL once per session, so a mid-request commit silently dropped the cap for the rest of the request. Sessions now opt into the cap via session.info and an after_begin listener re-applies SET LOCAL on every transaction; a new request_session() carries the request-tier cap for paths that cannot use Depends(get_db), and the stream endpoint uses it. Background jobs remain uncapped by default. Adds a dense shared-boundary regression fixture (the incident shape, miniaturised: 4 duplicate fronts per level x 12 levels, ~16.7M paths under the old query) plus overlap-merge, dense-with-gap, and multiple-open-front cases, and documents recommended Postgres temp_file_limit / log_temp_files settings for self-hosters.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Fixes a production-stability bug where computing "fronting since" on a large imported front history could take the whole database down, and closes two statement-timeout coverage gaps that turned a 30-second query failure into a multi-minute outage.
The query
With coalesce-contiguous-fronts enabled, the walk that finds how far back a member's unbroken fronting run extends was a recursive CTE joining
prev.ended_at = chain.started_atwithUNION ALL. That enumerates paths, not states: on an imported history whose switch boundaries share timestamps (PluralKit exports round to the second, so thousands collide), every step matches multiple predecessors and the intermediate set grows multiplicatively per level. The depth cap bounded depth, not width. A genuine ~14k-front imported history recently blew this up into ~19 GB of Postgres query temp, filling the volume and briefly failing queries for every user.The replacement is a set-based gaps-and-islands pass: one sorted window scan over the fronting members' own entries, where a run breaks only when the running MAX of
ended_atfalls strictly before the nextstarted_at(an open front reaches infinity). Cost is O(n log n) in the member's own front count, no recursion, no width explosion.Semantics are identical wherever fronts do not overlap, which is all live-written data - the pre-existing coalesce test suite passes unchanged. One deliberate refinement: overlapping fronts for the same member now merge into a single continuous run, where the old exact-boundary walk under-counted (a member continuously fronting across overlapping imported entries now gets the earlier, correct "since"). The walk-back depth cap is gone with nothing to need it;
member_since_cappedstays in the API for compatibility and is always empty.The timeout
Two gaps, found root-causing why the incident query ran ~13 minutes when the request cap is 30 seconds:
statement_timeout. The uncapped stream twin of the coalesce query is what kept spilling long after the/currentcopy had been correctly killed at 30 s.get_dbappliedSET LOCAL statement_timeoutonce per session;SET LOCALis transaction-scoped, so the first mid-request commit silently dropped the cap for the rest of that request.Sessions now opt into the cap via
session.info, and anafter_beginlistener re-appliesSET LOCALat the start of every transaction. A newrequest_session()context manager carries the request-tier cap for paths that cannot useDepends(get_db), and the stream endpoint's three sessions use it. Background jobs keep their separate (default unlimited) ceiling - verified live: capped sessions hold the cap across a commit, bare job sessions stay at 0.Tests
New regression fixture reproducing the incident shape, miniaturised: 4 duplicate fronts per level x 12 levels, all sharing exact boundary timestamps (~16.7M chain paths under the old query; instant under the new one), asserting the true chain start, a wall-clock bound, and an empty capped list. Plus overlap-merge, dense-history-with-a-real-gap, and member-in-two-open-fronts cases. Full behavioural config green: 1783 passed.
Docs
Changelog entries for both fixes, and SELFHOSTING.md gains a short "Postgres safety limits" section recommending
temp_file_limit+log_temp_files, so a self-hosted instance's worst case is one failed query rather than a full data volume.