Shared preload libraries#45
Merged
Merged
Conversation
…ce() Phase 1 of session_preload_libraries support (see dev/docs/PRELOAD_MODES.md). - _PG_init installs the query hooks (scan/executor/ProcessUtility) in every load mode, but gates the static maintenance worker and the blob-cache shmem reservation behind process_shared_preload_libraries_in_progress. Loading via session_preload/LOAD/fmgr keeps query correctness; worker + cache stay off. - Factor the worker's per-tick loop into a shared run_maintenance_pass() / maintain_one_table(), exposed as deltax.deltax_run_maintenance() for external scheduling (e.g. pg_cron) in session mode. Replica guard + catalog check live inside the shared function. - Add real per-table error isolation: each table's maintenance runs in an internal subtransaction (run_in_subtransaction), so a compress/retention ERROR rolls back only that table and the pass continues. The worker gains this too (previously a single failure aborted the whole tick). - Serialize maintenance passes with a transaction-level advisory lock so a manual deltax_run_maintenance() can't deadlock the background worker in full mode (both run the same detach/attach/compress DDL). The §3 runtime guard against silent zero-rows on mis-scoped backends is NOT in this change and is required before session mode is exposed as safe.
Decide not to build the runtime guard against silent zero-rows in session mode. The only core-evaluated mechanism that fires on a plain SELECT without our hooks (RLS USING qual) is bypassed by superusers and table owners — exactly the pg_dump/ETL/replication/admin connections most likely to arrive without hooks — so even the buildable form fails open for the highest-stakes readers. Document it as a known limitation instead, mitigated by configuration: - scope session_preload_libraries at the DATABASE level, not per-role - ensure backup/ETL/replication tooling loads the library too - prefer full mode if every reader can't be guaranteed to load it Update the status header, motivation, the mode-comparison table, the _PG_init pseudocode (no hooks-installed sentinel), and the out-of-scope list to match.
The session-mode tests added here caught a real bug: _PG_init defined the
PGC_POSTMASTER GUCs (target_database, blob_cache_mb, blob_cache_shards)
unconditionally, and PostgreSQL FATALs ("cannot create PGC_POSTMASTER variables
after startup") when those are created outside postmaster startup. So *any*
session_preload / LOAD / fmgr load of pg_deltax crashed the backend — session
mode was completely broken.
Fix: gate the three define_*_guc calls behind
process_shared_preload_libraries_in_progress, alongside the worker + blob-cache
registration (the pg_stat_statements pattern). In session mode these knobs are
simply absent (SHOW → unrecognized parameter); USERSET/SUSET GUCs stay
unconditional and work in both modes.
Tests:
- Unit (src/worker.rs): 6 #[pg_test]s for run_in_subtransaction — commit on
success, rollback + isolate a Postgres error, capture the message, continue
after a failed unit, nested isolation; plus a run_maintenance_pass no-op smoke.
- Integration (tests/test_session_preload.py): a dedicated container started
WITHOUT shared_preload (the only way to exercise the _PG_init else branch),
covering CREATE EXTENSION without preload, no background worker registered,
session_preload installing hooks for a plain-SELECT reader, the documented
silent-empty limitation (and that LOAD fixes it), deltax_run_maintenance() in
session mode, the now-absent PGC_POSTMASTER GUCs, and parallel-scan
correctness.
Docs: PRELOAD_MODES.md §4 updated — the GUCs are absent (not merely inert) in
session mode, with the FATAL rationale.
Two gaps found reviewing the maintenance pass against multi-DB and superuser-scheduler scenarios: 1. The pass-level advisory lock used a single fixed key. Advisory locks are cluster-wide, so that serialized maintenance across *every* database — the multiple workers in a full-mode target_database list, or one pg_cron job per database — even though they touch disjoint tables (a regression vs. the pre-advisory-lock behavior). Fold MyDatabaseId into the low 32 bits (maintenance_lock_key) so mutual exclusion is confined to a single database. 2. run_maintenance_pass ran the maintenance SQL (now(), pg_tables, operators, casts — all unqualified) under the caller's search_path. The worker pins search_path because it runs as superuser; the SQL-callable path didn't, so a superuser pg_cron job was exposed to search_path shadowing. Issue SET LOCAL search_path = pg_catalog, pg_temp at the top of every pass — covers both callers and reverts at txn end (no leak into the caller's session). Tests: a pg_test asserting the lock key = tag<<32 | current DB oid, and an integration test asserting deltax_run_maintenance() preserves the caller's search_path (proving SET LOCAL, not SET). Docs updated (§ background worker points 4-5).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This adds optional support for loading deltax via
session_preload_libraries.This has a number of advantages:
However there are some cons as well (at least currently):
pgcronor similar.To use:
There are more details on pros/cons and example usage in
docs/PRELOAD_MODES.md.