Skip to content

feat(db): align migration prototype with single-table RFC - #1772

Merged
Teingi merged 42 commits into
masterfrom
codex/unified-database-migrations-implementation
Oct 9, 2026
Merged

Teingi merged 42 commits into
masterfrom
codex/unified-database-migrations-implementation

Conversation

@Zxf-xufeng

@Zxf-xufeng Zxf-xufeng commented Sep 28, 2026 •

Copy link
Copy Markdown
Member

Which issue or RFC does this PR close?

Refs #1756 and implements the acceptance phase of RFC #1771. The implementation branch contains no RFC documents. #1716 remains independent; this PR does not close the remaining production rollout requirements.

Rationale for this change

Explicit database maintenance needs a trustworthy schema version and recoverable execution without adding run, step, or task control tables. Installing software remains separate from changing database structure. A successful four-table migration must not be mistaken for complete Server readiness.

What changes are included in this PR?

  • Use Alembic's single pc_schema_revision(version_num) control table. Keep durable maintenance intent, resource checksums, backup references, and recovery progress outside the business database.
  • Package frozen historical schemas and a linear p0001–p0003 chain covering four Artifact tables. Execute immutable resource snapshots and validate the actual schema and stored data before advancing the version. SQLite retains the replaced Tag table; seekdb/OceanBase change its CHECK constraint in place.
  • Provide server db-migrate status/plan/apply/verify for persistent SQLite, embedded seekdb, and OceanBase MySQL-mode databases through the configured maintenance adapter. These connections do not initialize current application metadata or open a business Repository. Plans bind target identity, schema, resources, configuration, backup policy, and maintenance scope.
  • Handle nontransactional seekdb/OceanBase DDL through exact registered transition prefixes and persistent external evidence. Interrupted adoption, committed partial DDL, and a committed version whose journal update was interrupted can resume. Unknown layouts or missing/inconsistent recovery evidence stop execution. Retries preserve the original backup decision and recovery point.
  • Hold the embedded seekdb directory lock through engine shutdown, including cross-process exclusion through canonical path aliases. Reject runtime fallback from native OS locking to soft locks. OceanBase uses a stable cluster/tenant/database identity with an explicit single-host OS coordinator lock and durable external run receipt. All four OceanBase commands require --lock-coordination single-host and one persistent --evidence-dir on the fixed maintenance host. Business Servers may remain distributed.
  • Support configured OceanBase connections through ODP without GET_LOCK(), fixed physical session IDs, or PROCESS privileges. Normal completion and successful connection closure clear the run receipt. An interrupted run blocks verification and ready/no-op shortcuts, even if its Alembic head is current. After a DBA confirms that prior remote execution and background DDL have ended, recovery requires an exact --acknowledge-previous-run token bound to the reviewed plan and rechecked under the lock. PC does not automatically establish remote quiescence; acknowledgement never bypasses schema, data, resource, backup, or original-journal checks.
  • Provide native BackupProvider implementations, unchecked manual-backup declarations, and explicit no-backup consent. SQLite Online Backup includes committed WAL. Native Fork feature detection remains separate from accepted restoration workflows; unavailable automatic backup never silently falls back. The current executor also rejects automatic table Fork before writes because same-database backup-object tracking is not yet registered.
  • Add service stop/start/restart and a separately tested maintenance controller. Stopped services retain registration with native activation suppressed. Startup validates the loaded definition, active manager state, and readiness; interruptions and cleanup errors preserve the failure and stopped-state evidence. Complete service switching remains disabled for this partial database bundle.
  • Update bilingual website migration guidance and extend the existing Python 3.11/3.14 seekdb CI job with unified runner, schema, connection, data-integrity, and interruption-recovery coverage. Preserve the incorporated fix(persistence): finish connection cleanup before a cancellation propagates #1782 SQLite transaction/cancellation fixes and their authorship.

Are there any user-facing changes?

This is Phase A acceptance. The installed bundle covers only pc_artifacts, pc_artifact_heads, pc_artifact_candidate_versions, and pc_artifact_tags. Output reports readiness_scope=registered_bundle and server_ready=false. Complete Server databases with unmanaged objects are rejected. Normal application startup and processing-migrate remain outside this bundle, and --manage-service cannot switch a business service.

All three backends now have deployment executors for registered layouts. Existing seekdb/OceanBase databases can use declared manual backup or explicitly accepted no-backup risk. Automatic Fork backup remains unavailable until release-owned recovery acceptance exists; a version number alone does not enable it. The engine feature minima are OceanBase AI Database 4.6.2 for both Fork forms, seekdb 1.1.0 for table Fork, and seekdb 1.2.0 for database Fork. Empty initialization requires no backup.

OceanBase requires access to tenant and cluster identity metadata, one fixed maintenance host, and an explicit persistent evidence directory. Minimal isolated four-table acceptance passed through an existing ODP endpoint on OceanBase AI Database 4.6.3. Normal upgrades retain one confirmation; an interrupted OceanBase run additionally requires the explicit DBA declaration described above. Complete historical baselines, legacy-task/data/projection validators, startup/SDK/Worker gates, full index coverage, and mandatory schema-change checks remain production prerequisites. No existing business database or personal service was modified during this work.

How was this change tested?

  • UV_NO_SYNC=1 UV_OFFLINE=1 PREK_NO_CONCURRENCY=1 make check: lock consistency, workflow/reference checks, pre-commit hooks, full type checks, and 33 integration-manifest tests passed.
  • Current migration, backup, connection, schema, CLI, coordinator-lock, and probe regressions on Python 3.14.6: 147 passed, 30 skipped. Service-maintenance regression: 39 passed. Optional native environments remain opt-in; these skips are not backend acceptance.
  • Real isolated embedded seekdb 1.4 (pylibseekdb 1.4.0.post1): 20 passed, including all 10 runner cases, schema reflection/data integrity, dedicated connections, and real cross-process exclusion through a symlink alias. The selection skips the OceanBase-only interrupted-run acknowledgement case.
  • Real OceanBase AI Database 4.6.3 through ODP: one disposable database with four historical Artifact tables and three sample rows. Verified coordinator exclusion across commits; unavailable automatic backup without schema changes; recovery after the first ALTER had committed, with unacknowledged retry rejected and explicit matching acknowledgement accepted; preserved data and exactly one added version table; final verification and no-op; and completed-head refusal while an unresolved run receipt remains. Wrong-token recovery was rejected and matching-token verification required no new backup. The test database was removed and its absence independently confirmed after the DROP response timed out. No existing business database or proxy/tenant setting was changed. This is not full Server, automatic remote-DDL completion detection, or Fork restoration acceptance.
  • Independent real seekdb checkpoint review also verified recovery after the p0003 CHECK removal committed before its version update.
  • Wheel build passed; the three new backend modules and all four MySQL Alembic resource files are packaged. Frozen released resources were not changed.
  • Both updated documentation pages compile as MDX. Full website lint/build was not rerun successfully because the worktree's missing dependencies require unavailable registry access. git diff --check passed.

AI usage statement

Implemented, independently reviewed, and tested with OpenAI Codex. Incorporated #1782 fixes preserve their original authors and Claude co-author attribution. Backend evidence distinguishes real execution from uncompleted production acceptance.

Zxf-xufeng and others added 18 commits September 28, 2026 20:08
…pagates

Repeatedly cancelling a transaction could keep its pool slot checked out.
Two driver-level gaps combine. SQLAlchemy can drive both a graceful close
and a stop on the same aiosqlite connection, whose stop is not idempotent
and enqueues a second request behind a worker that already exited. And a
statement cancelled mid-await leaves its native cursor held by the exception
stack, so the rollback that follows fails with "Connection closed" and the
slot is never returned.

Close the cursor tracked for the connection before retiring it, make the
driver stop idempotent per connection, interrupt the statement SQLite is
still running, and finish the whole exit inside a managed task so a repeated
cancellation cannot strand the cleanup. Only the transaction that owns the
connection runs it, so a nested exit cannot close its caller's cursors. A
clean exit stays in the caller's task, which the seekdb shutdown ordering
depends on.

Without the cursor step the new regression fails: the only pool slot is
still checked out after the tenth cancelled transaction.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…mitting part of it

Interrupting a SQLite statement rolls the whole native transaction back, so
writes that already reported success are gone while SQLAlchemy still trusts the
transaction. Keeping `is_disconnect = False` avoided discarding the connection
-- an in-memory StaticPool holds its entire database in it -- but it also let a
caller that caught the error keep writing and commit a partial result: insert
row 3, interrupt a long UPDATE, insert row 4, and only row 4 survives.

Record the interruption on the connection and refuse to commit when the
transaction exits, raising PendingRollbackError the way the unmodified baseline
does. The connection still returns to the pool, so an in-memory database
survives.

* `test_interrupted_write_cannot_commit_earlier_work[file|memory]` fails
  without the guard.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ransaction

The mark lives on the pooled connection, and the usage recorder runs its own
transaction without going through `AsyncDatabase.transaction()`, so an
interrupt recorded there was never consumed. It survived on the connection
until an unrelated transaction picked that connection up and raised
PendingRollbackError for a rollback it had nothing to do with -- which is how
this first showed up, as a topic-memory supervisor test failing on 3.11.

Clear any leftover mark when a transaction starts, so only an interrupt raised
inside it can fail it.

* `test_interrupt_marked_outside_the_transaction_does_not_fail_it` fails
  without the cleanup.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ection

Two follow-ups from review.

`connection.info` re-resolves the DBAPI connection, so once a cancelled
statement invalidated it through SQLAlchemy's normal path, reading the
interruption mark from the failure path raised PendingRollbackError and skipped
the rest of `_finish_transaction()`. That left the transaction counted as active
and kept `database.close()` from finishing. Hold the info dict itself instead;
it stays usable after the connection is gone.

The mark was also only checked before COMMIT ran. A commit cancelled while in
flight records the mark afterwards, so it survived on the pooled connection and
failed the next borrower's transaction. Clear it in a `finally` around the exit,
which covers cancellation during COMMIT or ROLLBACK too.

* `test_invalidated_connection_still_reaches_transaction_cleanup` fails without
  the first part.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
@Zxf-xufeng Zxf-xufeng changed the title feat(db): add versioned migration prototype feat(db): align migration prototype with single-table RFC Sep 30, 2026
@Zxf-xufeng
Zxf-xufeng marked this pull request as ready for review September 30, 2026 09:20
@Teingi
Teingi merged commit 6a4e799 into master Oct 9, 2026
46 checks passed
guamiao added a commit to guamiao/powercontext that referenced this pull request Oct 9, 2026
Bring in the database-migration prototype (oceanbase#1771/oceanbase#1772), loopback proxy
bypass (oceanbase#1888), topic-memory window fixes (oceanbase#1842/oceanbase#1898) and related work.

Resolve three conflicts:
- database.py: keep the split transaction/_transaction so cancellation
  cleanup still owns the connection; adopt the upstream nested-snapshot
  comment.
- test_sqlite_profile.py: keep this branch's SyncConnection/event imports
  and its invalidation + commit-mark regression tests.
- test_topic_memory_generic_api.py: adopt upstream's reworked cancellation
  e2e (completion-flush rendezvous with busy_timeout_ms=100), which retains
  the cancellation, normal/storage-failure, settlement and recovery-usage
  assertions through the public flush boundary.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants