fix(pg-delta): suppress Wasm FDW dependents and pgmq queue triggers in supabase integration#261
Open
avallete wants to merge 13 commits into
Open
fix(pg-delta): suppress Wasm FDW dependents and pgmq queue triggers in supabase integration#261avallete wants to merge 13 commits into
avallete wants to merge 13 commits into
Conversation
…able, and user mapping dependents in supabase integration Pins the post-CLI-1470 gap: suppressing CREATE/DROP/ALTER FOREIGN DATA WRAPPER for Wasm wrappers (`clerk`, `clerk_oauth`, ...) on its own leaves dependent CREATE SERVER / CREATE FOREIGN TABLE / CREATE USER MAPPING in the diff, which breaks `supabase db reset` locally with `foreign-data wrapper "clerk_oauth" does not exist`. Refs #258
…ing dependents in supabase integration Follow-up to CLI-1470 / #258. The supabase filter previously suppressed CREATE/DROP/ALTER FOREIGN DATA WRAPPER for Wasm-based wrappers (handler/validator in `extensions.*`), but left their dependent CREATE SERVER, CREATE FOREIGN TABLE, and CREATE USER MAPPING in the diff. Local `supabase db reset` then aborts at the SERVER statement with: ERROR: foreign-data wrapper "clerk_oauth" does not exist (SQLSTATE 42704) At statement: 9 CREATE SERVER clerk_oauth_server FOREIGN DATA WRAPPER clerk_oauth OPTIONS (...); Carry the parent wrapper's handler/validator on the server, foreign_table, and user_mapping models at extract time (joined to `pg_foreign_data_wrapper` + `pg_proc`, omitted from `dataFields()` so diffs stay stable) and extend the supabase integration filter to drop these dependents when either is in `extensions.*`. Server *privilege* scope is intentionally exempt because `postgres_fdw` legitimately installs into `extensions` and its server ACL is user-declarative state (covered by the existing CLI-1469 companion test "preserves GRANT on user-owned FOREIGN SERVER"). RED output (from preceding test commit): (fail) supabase integration filter — Wasm FDW dependents > suppresses CREATE SERVER bound to extensions.* Wasm FDW expect(received).toBe(expected); Expected: false; Received: true (fail) supabase integration filter — Wasm FDW dependents > suppresses DROP FOREIGN TABLE bound to extensions.* Wasm FDW expect(received).toBe(expected); Expected: false; Received: true (fail) supabase integration filter — Wasm FDW dependents > suppresses ALTER FOREIGN TABLE bound to extensions.* Wasm FDW expect(received).toBe(expected); Expected: false; Received: true (fail) supabase integration filter — Wasm FDW dependents > suppresses DROP USER MAPPING bound to extensions.* Wasm FDW expect(received).toBe(expected); Expected: false; Received: true (fail) supabase integration filter — Wasm FDW dependents > suppresses CREATE USER MAPPING when only wrapper validator is in extensions expect(received).toBe(expected); Expected: false; Received: true (fail) supabase integration e2e (pg17) > suppresses Wasm FDW server, foreign table, and user mapping dependents expect(received).toStrictEqual(expected); received: ["CREATE USER MAPPING FOR postgres SERVER wasm_server OPTIONS (user 'remote', password '__OPTION_PASSWORD__')"] Refs #258
- Added Verdaccio configuration for local package publishing. - Introduced scripts for starting Verdaccio and publishing pg-delta locally. - Updated .gitignore to include Verdaccio storage and configuration files. - Updated package.json to include Verdaccio as a dependency and new scripts. - Changed pg-delta versioning to a local format for development. This setup facilitates easier local development and testing of the pg-delta package.
…e tables in supabase integration
Pins the case where `db pull` emits CREATE TRIGGER against
`pgmq.q_<name>` / `pgmq.a_<name>` tables that are dynamically
materialized by `select pgmq.create('<name>')`. On a healthy
install the trigger extractor's `pg_depend deptype='e'` filter
already drops these, but real-world cloud projects (e.g. pgmq
1.4.4 which is what Supabase Cloud currently ships) can lose
that row, causing `supabase db reset` to abort with
`relation "pgmq.q_<name>" does not exist`.
Refs #258
… supabase integration Follow-up to the Wasm FDW dependents fix in #258. The trigger extractor already drops user triggers on tables that pgmq records as `pg_depend deptype='e'` to the pgmq extension, but real-world cloud projects can be missing that row. The immediate motivator is pgmq `1.4.4` (the version Supabase Cloud currently ships) which never records the dependency for `pgmq.q_<name>` / `pgmq.a_<name>` tables; verified on a real cloud project where all 59 queue/archive tables show `has_ext_link = false` while every other system-schema table (`cron.job`, `pgmq.meta`, `pgsodium.key`, `vault.secrets`) is properly linked. Older pgmq installs and manual `pg_dump`/restore that loses extension deps hit the same gap. When the row is missing, `supabase db reset` aborts at the trigger statement with: ERROR: relation "pgmq.q_<name>" does not exist (SQLSTATE 42P01) CREATE TRIGGER ... AFTER INSERT ON pgmq.q_<name> ... Add a defensive name-match fallback in the supabase integration filter that drops user triggers on `pgmq.q_*` / `pgmq.a_*` regardless of pg_depend state. A pgmq upgrade on the Supabase Cloud side closes the upstream root cause; this fix covers the window until all projects upgrade. RED output (from preceding test commit): (fail) supabase integration filter — pgmq queue triggers > suppresses CREATE trigger on pgmq.q_<name> calling a public function expect(received).toBe(expected); Expected: false; Received: true (fail) supabase integration filter — pgmq queue triggers > suppresses DROP trigger on pgmq.a_<name> calling a public function expect(received).toBe(expected); Expected: false; Received: true (fail) supabase integration e2e (pg17) > suppresses user triggers on pgmq queue tables when pg_depend link is missing received: ["CREATE TRIGGER after_insert_processed_milestones_queue AFTER INSERT ON pgmq.q_processed_milestones_queue FOR EACH ROW EXECUTE FUNCTION move_data_from_queue()"] Refs #258
…ge directories - Added 'verdaccio/.verdaccio/' to ignore Verdaccio's local storage. - Ensured '.verdaccio/plugins/' is also included for completeness.
🦋 Changeset detectedLatest commit: 4399f48 The changes in this PR will be included in the next version bump. This PR includes changesets to release 1 package
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
commit: |
Contributor
There was a problem hiding this comment.
Pull request overview
This PR extends @supabase/pg-delta’s Supabase integration to prevent supabase db pull from emitting DDL that cannot be replayed by local supabase db reset, specifically for (1) dependents of platform-managed Wasm FDWs and (2) user triggers on pgmq queue/archive tables when the extension dependency link is missing. It also adds Verdaccio-based tooling to publish and consume local @supabase/pg-delta builds without waiting for npm.
Changes:
- Extract and carry FDW wrapper handler/validator metadata onto SERVER / FOREIGN TABLE / USER MAPPING models (filter-only) and use it to suppress Wasm FDW dependents in the Supabase filter (excluding SERVER privilege changes).
- Add a defensive Supabase filter rule to suppress user triggers on
pgmq.q_*/pgmq.a_*tables when the extension dependency signal is missing (e.g., pgmq 1.4.4). - Add Verdaccio config + a publish script + repo wiring (
package.json,.gitignore) for local registry-based development, plus regression coverage in unit/integration tests.
Reviewed changes
Copilot reviewed 12 out of 14 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
verdaccio/config.yaml |
Adds a Verdaccio configuration for a local registry used in pg-delta development. |
scripts/verdaccio-publish-pg-delta.ts |
Adds a script to build and publish @supabase/pg-delta to Verdaccio with a generated local version. |
packages/pg-delta/tests/integration/supabase-dsl-e2e.test.ts |
Adds e2e regressions for suppressing Wasm FDW dependents and pgmq queue-table triggers. |
packages/pg-delta/src/core/objects/foreign-data-wrapper/user-mapping/user-mapping.model.ts |
Extracts and stores wrapper handler/validator metadata for user mappings (filter support). |
packages/pg-delta/src/core/objects/foreign-data-wrapper/server/server.model.ts |
Extracts and stores wrapper handler/validator metadata for foreign servers (filter support). |
packages/pg-delta/src/core/objects/foreign-data-wrapper/foreign-table/foreign-table.model.ts |
Extracts and stores wrapper handler/validator metadata for foreign tables (filter support). |
packages/pg-delta/src/core/integrations/supabase.ts |
Extends the Supabase filter with pgmq trigger fallback suppression and Wasm FDW dependent suppression. |
packages/pg-delta/src/core/integrations/supabase.test.ts |
Adds unit tests for the new Supabase filter rules (Wasm dependents + pgmq triggers). |
packages/pg-delta/src/core/catalog.model.ts |
Preserves wrapper handler/validator metadata through catalog normalization (so it’s available to filters). |
package.json |
Wires new scripts (verdaccio:start, pg-delta:publish-local) and adds verdaccio devDependency. |
bun.lock |
Updates lockfile for Verdaccio and related dependency resolution changes. |
.gitignore |
Ignores Verdaccio local storage/auth/plugin state directories and files. |
.changeset/wasm-fdw-dependents.md |
Changeset documenting the Wasm FDW dependents suppression patch. |
.changeset/pgmq-queue-trigger-fallback.md |
Changeset documenting the pgmq queue/archive trigger suppression fallback patch. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
…W dependents
The Wasm FDW dependent suppression keys on the bare `extensions.*`
namespace, but contrib FDWs like `postgres_fdw` also install their
handler/validator into `extensions` on Supabase — and those ARE
available in the local image. A user-created `postgres_fdw` server (plus
its foreign table and user mapping) is therefore wrongly dropped from
`db pull` migrations.
Add unit + e2e regressions proving a `postgres_fdw`-backed server /
foreign table / user mapping owned by `postgres` must roundtrip, and
rework the Wasm e2e case to use a genuine `extensions.wasm_fdw_handler`
(the discriminator the `wrappers` extension actually ships) created
under `SET ROLE postgres`, so suppression is proven by the handler rule
rather than the `*/owner` deny list.
RED against the current filter:
(fail) preserves user FDW whose handler is extensions.postgres_fdw_handler
(fail) preserves CREATE SERVER when postgres_fdw handler lives in extensions
(fail) preserves CREATE FOREIGN TABLE when postgres_fdw handler lives in extensions
(fail) preserves CREATE USER MAPPING when postgres_fdw handler lives in extensions
22 pass / 4 fail
(fail) preserves user-owned postgres_fdw server, foreign table, and user mapping
- "hasForeignTable": true + "hasForeignTable": false
- "hasServer": true + "hasServer": false
https://claude.ai/code/session_01D4R9D5vitZsFSBuPvx59eE
The CLI-1470 follow-up suppressed any foreign data wrapper / server / foreign table / user mapping whose handler or validator lived under `extensions.*`. That namespace is too broad: contrib FDWs such as `postgres_fdw` also install `extensions.postgres_fdw_handler` / `extensions.postgres_fdw_validator` on Supabase, yet they are present in the local image and must roundtrip. The previous rule silently dropped user-created `postgres_fdw` servers, foreign tables, and user mappings from `db pull` migrations (server _privilege_ scope was already preserved, producing an inconsistent split where the GRANT survived but the CREATE SERVER did not). Key the suppression on the Wasm handler/validator function names (`extensions.wasm_fdw_handler` / `extensions.wasm_fdw_validator`) that the `wrappers` extension ships and that platform Wasm wrappers (`clerk`, `clerk_oauth`, …) are all declared with. This targets only the wrappers the local image cannot provision and leaves user `postgres_fdw` setups intact. The same narrowing is applied to the wrapper-level rule so a suppressed wrapper and its preserved dependents can never disagree. GREEN: the regressions from the previous commit now pass (26 pass / 0 fail at the unit level; 9 pass for tests/integration/supabase-dsl-e2e). https://claude.ai/code/session_01D4R9D5vitZsFSBuPvx59eE
`fail()` called `process.exit(1)`, which terminates synchronously without unwinding `try`/`finally`. When invoked from inside `main()`'s `try` block (build or publish failure), it skipped the `finally` that restores the working-copy version, leaving `packages/pg-delta/package.json` bumped to the throwaway `0.0.0-local.<ts>` value — contradicting the script's own "always restore, even on failure" contract. Throw from `fail()` instead; the existing top-level `main().catch(...)` logs the message and exits non-zero, and the `finally` now runs. https://claude.ai/code/session_01D4R9D5vitZsFSBuPvx59eE
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Follow-up to #258 / CLI-1470. Extends the Supabase integration filter so
db pullmigrations no longer emit DDL that localsupabase db resetcannot replay:CREATE SERVER,CREATE FOREIGN TABLE, andCREATE USER MAPPINGfor platform Wasm wrappers (clerk,clerk_oauth, …) whose handler/validator lives inextensions.*.CREATE TRIGGERonpgmq.q_*/pgmq.a_*tables materialized bypgmq.create(), when thepg_dependlink to thepgmqextension is missing (notably pgmq 1.4.4 on Supabase Cloud).Also adds Verdaccio tooling so you can publish
@supabase/pg-deltalocally and point the CLI at it without waiting for npm.Problem
After pulling a remote schema with
pg-delta,supabase db resetfailed at different statements:ERROR: foreign-data wrapper "clerk_oauth" does not existatCREATE SERVER clerk_oauth_server ...ERROR: relation "pgmq.q_processed_milestones_queue" does not existatCREATE TRIGGER after_insert_processed_milestones_queue ... ON pgmq.q_processed_milestones_queueCLI-1470 already suppressed
CREATE/DROP/ALTER FOREIGN DATA WRAPPERfor Wasm wrappers inextensions.*, but dependent server/foreign-table/user-mapping DDL still leaked through. For pgmq, the trigger extractor’spg_depend deptype='e'filter works on newer pgmq, but pgmq 1.4.4 (what Cloud ships today) never records that dependency for dynamically createdq_*/a_*tables — verified on a real project where all 59 queue/archive tables hadhas_ext_link = falsewhilecron.job,pgmq.meta,vault.secrets, etc. were linked correctly.Solution
Wasm FDW dependents
handler/validatoronto server, foreign_table, and user_mapping models at extract time (pg_foreign_data_wrapper+pg_proc), omitted fromdataFields()so diffs stay stable.^extensions\..privilegescope is not suppressed —GRANT/REVOKE ON SERVERdoes not require superuser; userpostgres_fdwservers legitimately install intoextensionsand their ACL must roundtrip (covered by existing e2e “preserves GRANT on user-owned FOREIGN SERVER”).pgmq queue triggers
function_schemais user-owned” rule: exclude triggers onpgmqtables whose name matches^[qa]_.extension_table_oidsintrigger.model.tsfor projects wherepg_dependnever recorded the extension link.Local dev (Verdaccio)
bun run verdaccio:start+bun run pg-delta:publish-localpublish@supabase/pg-deltaas0.0.0-local.<timestamp>tohttp://localhost:4873/.--write-version-to=<project>/supabase/.temp/pgdelta-versionso the CLI resolves the local build viaPGDELTA_NPM_REGISTRY.Test plan
packages/pg-delta/src/core/integrations/supabase.test.ts— Wasm FDW dependents + pgmq queue triggers (22 cases)tests/integration/supabase-dsl-e2e.test.ts— Wasm dependents, pgmq trigger with simulated missingpg_depend, auth.users trigger preserved, FDW ACL regressionsbun run format-and-lint --write --unsafe && bun run check-types && bun run knip --fixManual validation (reporter):
bun run pg-delta:publish-local --write-version-to=<project>/supabase/.temp/pgdelta-versionPGDELTA_NPM_REGISTRY=http://host.docker.internal:4873(or equivalent) +supabase db pull --diff-engine pg-deltaCREATE SERVER ... clerk_oauthorCREATE TRIGGER ... ON pgmq.q_*supabase db resetcompletes without42704/42P01on those statementsCommit structure (RED → GREEN)
test(pg-delta): add failing regression for Wasm FDW server, foreign table, and user mapping dependentsfix(pg-delta): suppress Wasm FDW server, foreign table, and user mapping dependentstest(pg-delta): add failing regression for user triggers on pgmq queue tablesfix(pg-delta): suppress user triggers on pgmq queue/archive tablesfeat: integrate Verdaccio for local pg-delta developmentchore: update .gitignorefor Verdaccio storageChangesets
.changeset/wasm-fdw-dependents.md— patch.changeset/pgmq-queue-trigger-fallback.md— patchFollow-ups / out of scope
normalizeCatalogdropsforeignTable.security_labels— latent gap masked by symmetric roundtrip normalization; needs a directextractCatalogassertion +dummy_seclabel; not fixed here.pg_depend deptype='e'onpgmq.q_*/pgmq.a_*. A Supabase Cloud pgmq upgrade would reduce reliance on the name-match fallback over time.