ES blue-green M2: es_reindex + drop + delta --target-suffix (eph-1 rehearsal) - #6286
Merged
Conversation
…exes Admin "Rebuild Index" button now goes through a confirm_action page (DESTRUCTIVE warning), and the administration ES panel requires a "confirm_live_index_write" checkbox for rebuild_search_index / populate_all_indexes; choice labels mark both as live-index writes. Read-only actions (info, test_connection) are unchanged.
…lta --target-suffix es_reindex (requires bootstrap done): per program, one lockstep _vN+1 for the individuals+households pair, dark create from the code mapping (no alias), full populate, one delta --target-suffix catch-up pass, count verify vs Postgres with a single retry delta (abort before swap on mismatch), ONE atomic 4-action alias swap (must_exist + postcondition), two post-swap delta passes. Old version stays unaliased for rollback; crashed runs are abandoned, not resumed. es_populate_delta --target-suffix wraps both doc classes with versioned_doc() so upserts/deletes hit the dark _vN pair; a missing target is an error, never a create. es_drop_old_index_versions sweeps unaliased _vN leftovers (prints-only without --confirm, never touches anything aliased). index_management gains versioned_doc, existing_version_numbers, next_version_suffix, and create_versioned_index(suffix, attach_alias) returning the created name.
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## ephemeral-1 #6286 +/- ##
===============================================
- Coverage 96.92% 96.92% -0.01%
===============================================
Files 490 490
Lines 33319 33342 +23
Branches 3942 3944 +2
===============================================
+ Hits 32295 32317 +22
Misses 565 565
- Partials 459 460 +1
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
…heir number A leftover dark _vN from a crashed or verify-failed run is never trusted (may be half-populated or carry an older deployment's mapping), and abandoning it to a higher version number accumulated junk. Now the run deletes unaliased versions NEWER than the current alias target before computing the next suffix, so the rebuild lands under the same number. Versions below the alias target (sanity-window rollback net) stay untouched.
…ean rebuild The working version is now deterministic - highest alias-target version of the pair + 1 - so a re-run lands on the dark pair a previous run left behind and RESUMES it, skipping the expensive full populate. Three guards make that safe: - create_versioned_index stamps mappings._meta.hope_mapping_hash at creation; a leftover whose stamp does not match the current code mapping (older deployment) is deleted and rebuilt under the same number automatically, so a stale mapping can never be swapped live, - the catch-up delta for a resumed pair anchors at the leftover's own index creation_date, covering everything the dead run may have missed, - a half-populated leftover cannot be topped up by a delta: the count verify gate aborts before the swap and the error hints at --sweep-wrecks, which deletes dark versions newer than the alias target upfront for an explicit clean rebuild. next_version_suffix (physical max + 1) is gone - it would skip past the resumable leftover instead of landing on it.
…r gap in the stamp - _report_plan skips (with a warning) a program whose alias target sits outside the _vN scheme instead of aborting the whole --dry-run, - mapping_content_hash drops default=str: a non-JSON mapping value now raises loudly instead of producing a repr-based hash that would silently never match and kill resume forever, - documented accepted gap: the stamp covers mappings only, so an analyzer/similarity-only change is invisible to it - reindex with --sweep-wrecks after such a deploy (noted in the flag's help too).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second slice of the blue-green ES work (follow-up to #6255, M1 bootstrap — already executed on eph-1, 292/292 programs).
What's here
es_reindex— the recurring blue-green reindex (requires bootstrap done; refuses non-alias indexes). Per program, the individuals+households pair is one unit:_vN+1for the pair (max over both indexes' existing versions + 1),_vN+1indexes from the code mapping — no alias, app unaffected,es_populate_delta --target-suffixcatch-up pass,must_exist+ postcondition read-back),Crashed runs are abandoned, not resumed (next run takes a higher version; leftovers swept by the drop command).
es_populate_delta --target-suffix vN— writes the delta into the dark_vNpair via aversioned_doc()wrapper; missing target = error, never a create. Rejected together with--reconcile/--verify(those compare through the alias).es_drop_old_index_versions— sweeps unaliased_vNleftovers after the sanity window; prints-only without--confirm, never touches the alias target or anything aliased.Confirmation gates for live-index writers (separate commit): admin "Rebuild Index" button now goes through a
confirm_actionpage; the administration ES panel requires an explicit checkbox forrebuild_search_index/populate_all_indexes.Tests
104 ES-related tests green (
test_es_reindex,test_es_drop_old_index_versions,test_index_versioning_helpers, extended delta/bootstrap/panel suites); mypy and ruff clean.Next
M2 rehearsal on eph-1:
es_reindexcanary →--allwithes_mutate_streamnoise, chaos drills, rollback drill, then drop.