Skip to content

fix(runtime): make concurrent OceanBase replica startup idempotent - #1957

Draft
Tsukikage7 wants to merge 2 commits into
oceanbase:masterfrom
Tsukikage7:fix/concurrent-runtime-startup
Draft

Tsukikage7 wants to merge 2 commits into
oceanbase:masterfrom
Tsukikage7:fix/concurrent-runtime-startup

Conversation

@Tsukikage7

@Tsukikage7 Tsukikage7 commented Oct 12, 2026 •

Copy link
Copy Markdown

Which issue or RFC does this PR close?

Related to #1609. Supplies the runtime cold-start dependency for the API/background Helm deployment in #1854; does not close either item.

Rationale for this change

Concurrent API/background replicas can race between schema existence checks and DDL, or lose a static authorization preset's first unique-key insert. On OceanBase CE 4.4.2.1, native VECTOR creation can additionally return error 1235 even with IF NOT EXISTS while another replica owns the column reservation and its logical index metadata is still settling.

What changes are included in this PR?

  • Use native idempotent table and ordinary-index DDL for OceanBase/seekdb. Keep specialized vector-index ownership with the native vector initializers so regular schema creation cannot discard VECTOR/HNSW semantics.
  • Make Experience, Atomic Memory and both Topic Memory full-text projections use native IF NOT EXISTS, retaining MATCH capability probes and current-schema validation.
  • Validate the exact native vector name, embedding column and requested parameters using SHOW CREATE TABLE. For the specific OceanBase duplicate-column 1235 race, wait for matching logical metadata within the tenant's existing query timeout, without replaying CREATE. Reject ordinary or incompatible indexes and unrelated errors.
  • Replay only rolled-back MySQL 1062 conflicts for static authorization presets, with the same binding and idempotency key and at most three attempts; propagate other or persistent failures.
  • Add concurrent startup, native-vector ownership, metadata validation and authorization regressions against the current Atomic Memory schema (pc_atomic_memory_current).

The branch includes current master through 329b4be, including upstream cancellation cleanup. The PR's net change is 11 source/test files; no Helm manifests, credentials or acceptance artifacts are included.

Are there any user-facing changes?

Concurrent replicas can initialize a fresh external OceanBase database reliably, while unusable native indexes still prevent startup. No public API or persisted-format change is introduced, and existing SQLite recorder retries are unchanged. Real concurrent cold-start acceptance covered OceanBase CE 4.4.2.1. The final head also passed the CI clean-migration suite on OceanBase CE 4.3.5.6; this is a separate migration/authorization/vector regression scope. Seekdb and other versions have not received concurrent cold-start acceptance for this patch.

How was this change tested?

On final head c8a0730:

  • uv run --locked python -m pytest tests/builtin/persistence/test_native_schema_creation.py tests/builtin/persistence/test_vector_initialization.py tests/builtin/persistence/test_topic_memory_oceanbase.py tests/test_access_bootstrap.py tests/builtin/persistence/test_sqlite_profile.py tests/builtin/runtime/test_model_usage_recorder.py -q: 87 passed, 1 skipped (no live database configured in this local run).
  • make check: passed, including uv run --locked prek run -a, both type-check scopes and 49 integration-manifest tests.
  • make docs-test: passed; 916 public pages and their internal links verified.
  • uv run --locked python -m pytest -q: 5060 passed, 430 skipped, 4 deselected, 1 failed in 20m43s. The sole failure is the existing tests/builtin/test_native_code.py::test_credentials_and_symlinks_are_not_read fixture attempting to git add .env under the host global Git ignore rule. The exact test passed separately with GIT_CONFIG_GLOBAL=/dev/null; user Git configuration and test source were unchanged.
  • Fork E2E workflow: completed, success on this exact head, all 5 jobs passed (Recall effort/OceanBase, Atomic Memory clean migration/OceanBase, validate, Acceptance/SQLite, Acceptance/OceanBase). Logs confirm 3 real OceanBase contract tests, 302 recall/statistics/prepared-context regressions, 40 clean migration cases without skips, 18 OceanBase Atomic Memory/HTTP regressions, and both deterministic acceptance scenarios.
  • Upstream workflows for this head are action_required, awaiting maintainer authorization; the fork run does not replace upstream required checks.

Separate real-backend acceptance was completed before the latest master merge, using the final six production-file patch on base 2058325. These production files are identical on this PR head:

  • OceanBase CE 4.4.2.1: 7 passed, no skips, covering concurrent/repeated DDL, unusable FTS rejection, preserved rows, authorization bootstrap, all three native vector indexes and real Topic Memory hybrid/detail search.
  • An isolated amd64 Kubernetes Helm source fixture loaded that exact patched runtime into two API and two background replicas: fresh-database cold start with zero restarts; authenticated HTTP/MCP and unauthenticated 401; persistence after replacing API pods; and observed leader lease transfer after deleting the identified background leader. Current Atomic Memory schema columns were verified.
  • Acceptance resources and temporary credentials were removed afterward.

Limit: the Kubernetes run used a temporary source fixture, not a newly published application image. Published-image pull/entrypoint and the unmodified chart against that image remain unverified. The real-backend and Kubernetes acceptance has not been rerun after the latest master merge; the cleaned environment was not recreated.

AI usage statement

OpenAI Codex assisted with implementation and validation. The exact model identifier was not recorded in the local task evidence.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant