ES blue-green M1: clone-first alias bootstrap (eph-1 rehearsal) - #6255
Merged
Conversation
…master) Master-compatible variant of #6164 (which targets develop). Same two management commands + tests; only adapted to master's code: - elasticsearch_dsl import path (es-py 8), not elasticsearch.dsl (es-py 9) - index_management calls without the develop-only using=/parallel=/thread_count= kwargs (#6031 refactor is not on master) - dropped the _es_shadow v9 wiring (imports elasticsearch.dsl -> ImportError on master, and unused in the default-connection setup) No logic change to the delta itself; 34 tests pass on the ES8 environment.
…nifests, bulk-copy) README runbook, mutate-data.yml / reindex-delta.yml pod manifests, and es8_to_es9_migration (Valeriya's raw-HTTP ES8->ES9 bulk copy) + _es_shadow helper. _es_shadow uses the es-py 8 import path (elasticsearch_dsl) and is not wired into the delta on master (default connection); kept only for a multi-alias shadow setup.
…ook table Renumber to 11 sequential steps, drop the duplicated delta row, and update the command-section step references (final delta is now step 9).
Keep every step the user added as its own row in the same order (delta catch-up twice: steps 4 and 8, plus final delta at 11); only fix the malformed markdown rows and renumber 1-12. No reordering, no merging.
…elasticsearch_dsl -> elasticsearch.dsl)
…gram indexes Per index: write-block -> _clone to <name>_v1 (replicas=0) -> wait active primary -> count sanity -> atomic remove_index+add alias -> unblock + restore replicas. Resumable from ES state (skip on alias, redo takeover after crash), ES-index lock, --status/--dry-run, final es_populate_delta pass sweeps freeze-window writes. create_program_indexes now creates _v1 + alias so new programs are born on the alias scheme.
…e_program_indexes Program-activation signal and rebuild_search_index no longer delete a live index: they use the new non-destructive ensure_program_indexes (create missing _v1+alias, upsert-populate). rebuild_program_indexes stays destructive but is reserved for the explicit admin Rebuild Index button, documented as such.
…new no-cover pragmas The es8->es9 cluster migration is done and nothing imports those files; they were also the source of the lint/mypy failures. The pragma gate forbids newly added 'pragma: no cover' lines, so the ensure_program_indexes failure branches are now covered by tests instead.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## ephemeral-1 #6255 +/- ##
============================================
Coverage 96.92% 96.92%
============================================
Files 490 490
Lines 33304 33319 +15
Branches 3940 3942 +2
============================================
+ Hits 32279 32295 +16
Misses 565 565
+ Partials 460 459 -1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…-created indexes A doc write racing index creation auto-creates a dynamic-mapping index; ensure-style create-if-missing then skips the proper mapping and search silently returns nothing (caught by test_search filter tests). rebuild_search_index is an explicit console/dev recovery tool, so delete->create->populate is the right semantics there. The automatic path (program-activation signal) stays on ensure_program_indexes.
johniak
marked this pull request as draft
July 28, 2026 10:45
… _v1 A rebuild during a blue-green sanity window (alias on _v2, unaliased _v1 still lingering) would delete _v2 and then collide creating _v1, stranding the program with no index and no alias. max(existing)+1 sidesteps the collision.
…elta always runs - es_bootstrap_aliases: open the dark clone (unblock + replicas) BEFORE the atomic takeover so a crash after the swap leaves nothing blocked; any failure before the takeover unblocks the still-live source; the alias branch self-heals a lingering write block from a crashed older run (S1/G1) - missing-index branch full-populates instead of leaving an empty index behind a successful message (S2) - the final es_populate_delta pass runs even when some indexes failed, so healthy programs don't keep freeze-window drift (S3) - hard-delete-during-freeze documented as an accepted M1 risk (S4) - ambiguous --program <code> (unique only per BA) is rejected (S5) - es_populate_delta propagates --using through create/populate/verify/reconcile and bulk writes/deletes (S6) - es_mutate_stream: deletion opt-in (default 0) and refuses to run outside DEBUG without --i-am-sure (S7) - README rewritten for the M1 alias bootstrap tooling (S8); working-note markers removed (S9) - failure-injection tests: clone/health/alias-API failures unblock the source, target opened before takeover, heal path, ambiguous code, DEBUG guard (G2)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Milestone 1 of the ES blue-green plan - pure infra, no changes from the blocked PRs (#5912/#5913).
es_populate_delta+es_mutate_streamfrom feat(household): es shadow-cluster delta sync + change-stream tools (master) #6176, with imports fixed for elasticsearch 9 (master droppedelasticsearch_dsl)es_bootstrap_aliasescommand: per program x {individuals, households} - write-block ->_cloneto<name>_v1(replicas=0) -> wait active primary -> count sanity -> atomicremove_index+addalias -> unblock + restore replicas; resumable from ES state, ES-index lock,--status/--dry-run, final delta pass sweeps freeze-window writescreate_program_indexesnow creates_v1+ alias, so new programs are born on the alias schemerebuild_search_indexroute through the new non-destructiveensure_program_indexes;rebuild_program_indexesstays destructive only behind the explicit admin buttonRehearsal runbook for eph-1: canary -> verify ->
--allwithes_mutate_streamrunning -> chaos checks (kill mid-run, parallel run, program activation mid-test) -> clean--status.