fix: strict upsert accounting and opt-in end-of-index flush - #638
Open
j-sperling wants to merge 3 commits into
Open
fix: strict upsert accounting and opt-in end-of-index flush#638j-sperling wants to merge 3 commits into
j-sperling wants to merge 3 commits into
Conversation
Milvus can acknowledge an upsert while reporting upsert_count 0, and no flush happens anywhere in the indexing path, so a reader in another process can see an empty or stale collection right after `memsearch index` reports success (zilliztech#534). - MilvusStore.upsert() trusts the server-reported upsert_count and raises on an explicit zero-write response instead of silently falling back to len(chunks). - New milvus.flush_on_index config key (default false, preserving current behavior) seals pending writes once at the end of an indexing run; per-upsert flushing is deliberately avoided because it creates many small sealed segments. - Document read-after-write visibility in the troubleshooting guide and the config key table.
…ptin-flush # Conflicts: # src/memsearch/config.py # src/memsearch/core.py
The end-of-index flush was gated on the upserted-chunk count, so a run whose only writes were deletions (a removed section's stale chunks or deleted-file cleanup) never flushed — leaving other readers with stale results in exactly the cross-process scenario flush_on_index exists for. _index_file now reports stale-chunk deletions and the gate treats any write as flush-worthy. Partial server counts (0 < n < batch) are logged instead of passing silently.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MilvusStore.upsert()returnslen(chunks)as the success count even when the server explicitly reportsupsert_count: 0, somemsearch indexcan report success while nothing became durable (MilvusStore.upsert() reports success but writes are not durable on remote Milvus 2.5+ (missing flush) #534). It now trusts the server-reported count and raises on an explicit zero-write response; responses without a count keep the historical fallback.milvus.flush_on_index(bool, defaultfalse— current behavior is unchanged): oneflush()at the end of an indexing run, for pipelines where a reader in another process searches immediately aftermemsearch indexreturns. Deliberately not per-upsert flushing, which would create many small sealed segments.docs/troubleshooting.mdand the config key table.This is the fix shape proposed by the maintainer in #534 (strict count + opt-in end-of-index flush + documented consistency behavior). Datapoint from a different environment than that issue's repro: we run
memsearch indexfrom an agent Stop/SessionEnd hook on macOS with Milvus Lite, and a separate short-lived process that searches immediately afterwards intermittently missed the fresh chunks until we added an external flush after each run — so the cross-process visibility gap is not exclusive to remote Milvus, even though Lite auto-flushes on close.Test plan
tests/test_store.py: server-reported count trusted overlen(chunks); explicit zero-write raisesRuntimeError; missing count falls back;flush()targets the configured collectiontests/test_flush_on_index.py: end-of-index flush fires exactly once when enabled, never by default, and not on runs that index nothing_cfg_to_memsearch_kwargstranslation test covers the new key;config set milvus.flush_on_index truecoerces via_BOOL_FIELDSruff check/ruff format --checkclean