Describe the bug
With the default V2 RDB writer (rdb-write-v2 yes), the RDB save path CHECK-crashes (SIGABRT) in the index-extension serializer when a key sits in the lazily-drained multi-mutation queue whose tracked record has already been consumed by an ordinary (non-MULTI) mutation of the same key:
CHECK failure: tracked_mutated_records_.find(key) != tracked_mutated_records_.end()
On BGSAVE/full-sync the fork child aborts and the save fails (rdb_last_bgsave_status:err) — and keeps failing on every retry until something drains the stranded queue, so persistence and replica full-syncs stay broken. Anything that runs the save in the parent process would abort the server itself. Reproduced on the exact 1.2.0 and 1.2.1 release digests.
To Reproduce (deterministic, uses the official debug pausepoint)
Config:
loadmodule /usr/lib/valkey/libjson.so
loadmodule /usr/lib/valkey/libsearch.so --debug-mode yes --rdb-write-v2 yes
search.writer-threads 2
appendonly no
save ""
Steps:
FT.CREATE idx ON HASH PREFIX 1 doc: SCHEMA content TEXT
HSET doc:1 content initial
FT._DEBUG PAUSEPOINT SET mutation_processing
# async, from a second connection — the scheduled worker parks at the pausepoint:
HSET doc:1 content ordinary
# wait until FT._DEBUG PAUSEPOINT TEST mutation_processing > 0, then:
MULTI
HSET doc:1 content transactional
EXEC
# ^ reuses the already-tracked record but also leaves doc:1 in the multi queue
FT._DEBUG PAUSEPOINT RESET mutation_processing
# (wait for the async HSET to return)
BGSAVE # -> child hits the CHECK above and aborts
With --rdb-write-v2 no the identical sequence completes (rdb_last_bgsave_status:ok): the V1 writer skips that extension and index contents are rebuilt by backfill after restore.
Two additional observations:
Expected behavior
Saving the pending-mutation state must tolerate a multi-queue entry whose tracked record was consumed by an interleaved ordinary mutation (or the interleaving must not produce that state). A save must not abort.
Environment
valkey/valkey-bundle:9.1.1-trixie@sha256:477e4579ddcdf15c9eedaae4c7c385774961da61227cd14697aba5ecbedc3aa2 (valkey core 9.1.0, search 1.2.1 / module ver 66049); also reproduces on the 1.2.0-era bundle digest- Linux amd64, Docker (
--cpus 1), standalone
- Reproduction requires
--debug-mode yes only for determinism (the pausepoint); the underlying interleaving is reachable in normal operation
Happy to provide the full scripted reproduction (bash) on request.
Describe the bug
With the default V2 RDB writer (
rdb-write-v2 yes), the RDB save path CHECK-crashes (SIGABRT) in the index-extension serializer when a key sits in the lazily-drained multi-mutation queue whose tracked record has already been consumed by an ordinary (non-MULTI) mutation of the same key:On
BGSAVE/full-sync the fork child aborts and the save fails (rdb_last_bgsave_status:err) — and keeps failing on every retry until something drains the stranded queue, so persistence and replica full-syncs stay broken. Anything that runs the save in the parent process would abort the server itself. Reproduced on the exact 1.2.0 and 1.2.1 release digests.To Reproduce (deterministic, uses the official debug pausepoint)
Config:
Steps:
With
--rdb-write-v2 nothe identical sequence completes (rdb_last_bgsave_status:ok): the V1 writer skips that extension and index contents are rebuilt by backfill after restore.Two additional observations:
FT.SEARCHbefore theBGSAVEdrains the stranded multi queue and avoids the crash — consistent with the queue being drained lazily on the query path (ProcessMultiQueue()). That same lazy-drain-on-query design also enables a separate main-thread deadlock during fork windows, reported in the companion issue [BUG] Permanent main-thread deadlock: FT.SEARCH during a fork window blocks in ProcessMultiQueue() on the suspended writer pool #1260.Expected behavior
Saving the pending-mutation state must tolerate a multi-queue entry whose tracked record was consumed by an interleaved ordinary mutation (or the interleaving must not produce that state). A save must not abort.
Environment
valkey/valkey-bundle:9.1.1-trixie@sha256:477e4579ddcdf15c9eedaae4c7c385774961da61227cd14697aba5ecbedc3aa2(valkey core 9.1.0, search 1.2.1 / module ver 66049); also reproduces on the 1.2.0-era bundle digest--cpus 1), standalone--debug-mode yesonly for determinism (the pausepoint); the underlying interleaving is reachable in normal operationHappy to provide the full scripted reproduction (bash) on request.