Skip to content

fix: 3.9.1 follow-up for startup_parameters - #250

Merged
vadv merged 8 commits into
masterfrom
fix/3.9.1-codex-followup-on-master
May 13, 2026
Merged

vadv merged 8 commits into
masterfrom
fix/3.9.1-codex-followup-on-master

Conversation

@vadv

@vadv vadv commented May 13, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Close the gaps that surfaced in the first deployment cycle of startup_parameters from 3.9.0: SQLSTATE choice, ParameterStatus correctness, auth_query refetch races, RBAC on the new API surfaces, and alert rules.

What operators see

  • Backend startup that fails because the merged map does not fit the StartupMessage budget now returns SQLSTATE 53400 (configuration_limit_exceeded). The previous code used 54000. Alert rules and log filters keyed on 54000 for this case need to switch.
  • PgDoormanStartupParameterPgRejection ships as severity: warning (was critical in 3.9.0). Cascade overflow stays critical. Re-check on-call routing if it pages by severity.
  • ParameterStatus messages forwarded to the client no longer overwrite operator-managed GUCs, so the value the client reads matches what the backend actually has.
  • auth_query accepts a startup_parameters column typed as native json or jsonb; the ::text cast that earlier deployments needed is gone.
  • After an MD5 password refetch, the dynamic pool is rebuilt with the fresh parameters instead of keeping the stale overlay; concurrent first-auth requests no longer race a half-initialised pool into existence.
  • /api/config and /api/pools show literal startup_parameters values only to Admin. SSO readers get the masked view consistent with the rest of the read-only surface.
  • /api/config marks general.host, general.port, web.host, web.port (plus worker_threads, unix_socket_dir, backlog) as restart-required so the SPA stops offering RELOAD on fields that need a restart.
  • New Prometheus rules cover PostgreSQL-side rejection, cascade and overlay budget overflows, malformed auth_query payloads, dedicated-mode drops, and SSO credentials arriving over insecure transport.

Why

The 3.9.0 rollout exposed three classes of issue. First, the rejection SQLSTATE conflated a pooler-side budget decision with a backend statement-cancellation code, which made log triage and alert keying ambiguous. Second, the dynamic pool path had two real races: a ParameterStatus overlay leaked across reauthentication, and concurrent first-auth callers could observe a pool created against a now-stale auth_query row. Third, the new read APIs leaked literal GUC values to SSO readers and presented bind-address fields as if they were reloadable.

The cached merge in the pool removes per-checkout allocation of the merged map and its budget decision; checkout now reads the precomputed result instead of recalculating it.

Risk and rollout

  • Breaking for alerting: operators with rules on SQLSTATE 54000 for the pooler-side budget case must move to 53400. Severity of PgDoormanStartupParameterPgRejection dropped to warning; pager routing keyed on critical for that alert will stop paging.
  • Breaking for /api/config and /api/pools readers: SSO callers no longer see literal startup_parameter values; only Admin does. Anonymous and SSO surfaces stay the same shape, only the values are masked.
  • Wire protocol, on-disk config schema, and the 3.9.0 cascade semantics (general -> pool -> auth_query) are unchanged.
  • BDD covers the new behaviour: a jsonb auth_query column applied without ::text, and /api/config returning restart-required for bind-address fields.

Test plan

  • Existing startup_parameters.feature and web-ui.feature scenarios pass on this branch.
  • New scenarios in startup-parameters.feature (jsonb column) and web-ui.feature (bind-address restart-required) pass under make test-bdd.
  • cargo test and cargo clippy --all-targets -- --deny warnings are green.
  • Operators with alert rules on SQLSTATE 54000 switched to 53400 before deploying 3.9.1.

@vadv vadv changed the title fix: 3.9.1 codex follow-up for startup_parameters fix: 3.9.1 follow-up for startup_parameters May 13, 2026
Comment thread src/auth/auth_query.rs Dismissed
Comment thread src/auth/mod.rs Dismissed
Comment thread src/auth/mod.rs Dismissed
Comment thread src/auth/mod.rs Dismissed
Comment thread src/pool/dynamic.rs Fixed
Comment thread src/pool/dynamic.rs Dismissed
Comment thread src/pool/server_pool.rs Dismissed
Comment thread src/pool/server_pool.rs Dismissed
dmitrivasilyev added 7 commits May 13, 2026 13:19
…n overlay drift

A static pool registered for the same (db, user) identifier holds `empty_overlay_hash()`. A concurrent auth_query passthrough login that fetched a non-empty overlay used to take the hash-mismatch branch in both fast and slow paths: drop_dynamic_pool is a no-op for static pools, and new_pools.remove(&identifier) does not check is_dynamic_pool. The result was a static pool silently replaced by a dynamic auth_query pool with operator-defined backend_auth and startup_parameters swapped for the passthrough version.

Add should_rebuild_for_overlay_drift(live_hash, fetched_hash, is_dynamic). The rebuild decision now requires both an overlay-hash mismatch AND that the live pool is dynamic. Both fast and slow paths consult the same predicate; backend_auth is refreshed only when the hash matches.
…rtup_parameters

PostgreSQL GUC lookup is case-insensitive, and `canonicalize_param_name` lowercases non-tracked keys before they reach the wire. The reserved protocol-extension prefix `_pq_.` was checked case-sensitively, so a configured or auth_query-supplied key like `_PQ_.foo` passed `validate_key`, then got lowercased on the cascade output and arrived at the backend as `_pq_.foo`. That bypassed the documented guard against injecting protocol-extension parameters.

Tighten the prefix check to `eq_ignore_ascii_case` on the first five bytes; both `validate` and `validate_entry` (auth_query JSON path) now reject `_PQ_.`, `_Pq_.`, and any other casing.
…_sources

`SHOW STARTUP_PARAMETERS` and `/api/pools` previously showed raw operator-written keys, while the wire-ready cascade canonicalised them via `cascade_canonical_keys`. A pool that wrote `TimeZone` over a general `timezone` showed both rows in the read surface, and the wire compare in `effective_startup_parameters_with_sources` then flagged one variant as `dropped_due_to_budget` or `stale`. That pointed the operator at RELOAD/refetch even though the runtime had merged the cascade correctly.

Canonicalise inside `resolve_with_sources` with the same function as the runtime. Layer precedence is preserved because later inserts at the same canonical key replace earlier ones, mirroring the original raw-key behaviour.
The new `@web-ui` BDD job in `.github/workflows/bdd-tests.yml` first runs in this branch and exposed that 13 of the 14 scenarios used `And output contains "..."`. cucumber-rs found no matching step definition, marked every assertion as skipped, and failed the suite. The new `/api/config` restart-required scenario never executed.

Rewrite the steps as `And the command output should contain "..."`, the phrasing declared by `command_output_should_contain` in `tests/bdd/shell_helper.rs:303`. Scenario behaviour is unchanged.
…apshot

Client startup used to compute the operator-managed key set with a second `POOLS` global lookup after `authenticate` returned `server_parameters`. A RELOAD or auth_query overlay refetch between the two reads could leave the filter inspecting one snapshot while `server_parameters` came from another. The visible symptom: `ParameterStatus` messages forwarded to the client carry values for keys that the backend session already has pinned by `startup_parameters`.

Have `authenticate` return both via a new `AuthOutcome` struct. The static path snapshots the key set from the pool that produced `server_parameters`; the auth_query path does the same for the shared (dedicated mode) and dynamic (passthrough mode) pools. `OperatorManagedKeys = Arc<HashSet<String>>` keeps the signature readable and preserves the existing `Arc` zero-copy clone.
…us-rules paths

Three documentation and CI tweaks from codex review:

- `dashboard-validation.yml` listens on `monitoring/prometheus-rules/**` so a rules-only change does not bypass smoke and ground-truth validation.
- The per-user `startup_parameters` tutorial now spells out that changes follow `auth_query.cache_ttl` rather than the next reconnect, and lists the levers operators have to roll out immediately.
- The generated `startup_parameters` reference describes the runtime reject path (SQLSTATE 53400) instead of the drop-and-continue wording. The `fields.yaml` source of truth is updated alongside the reference text in `general.md`.
…eaks

The Phase 6 fields.yaml change replaced the drop-and-continue wording with the SQLSTATE 53400 reject behaviour. The committed reference configs (`pg_doorman.toml`, `pg_doorman.yaml`) embed those descriptions verbatim, so the in-tree copies fell out of sync and the matches-file guard tests in `app::generate::annotated::tests` failed in Library Tests.
@vadv
vadv merged commit c76457c into master May 13, 2026
15 checks passed
@vadv
vadv deleted the fix/3.9.1-codex-followup-on-master branch May 13, 2026 10:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants