Follow up items from Atomic slot migraiton

[madolson]

So I am getting the following AIs (which I am hoping we can address after RC1):

Required for 9.0 GA:

• : Prevent exposure of importing keys on replicas during atomic slot migration #2635
• : Finalize on keyspace notifications/valkey-search integration
• : Slot migration start/end module notifications
  • Introduce mechanism to block atomic slot migration when a module does not support it #2579
  • Make modules opt-in to atomic slot migration and add server events #2593
• Introduce mechanism to block atomic slot migration when a module does not support it #2579
• : More stats for slot migration (e.g. output buffer size)
• : 12. Update docs with information about slot migration overhead
• Add slot migration client flags and module context flags #2639
• Fix atomic slot migration snapshot never proceeding with hz 1 #2636
• Address leakage of importing keys in stats and slot-stats (either let it leak, or be consistent about containing it throughout)

Not Required for 9.0 GA:

• : Add tests for integration with pubsub, blocking, and slotstats

  • Add atomic slot migration test to validate DB and slot stats #2638
  • Add atomic slot migration test for unblock on migration complete #2637

• : Simplify atomic slot migration internal communication #2591

• : 9. Design for how to address WAIT propagation

• : 10. Design for how to ensure replica doesn't lose slot ownership if failover happens during takeover

• : 14. Move testing from COUNTKEYSINSLOT to progress metrics and hide importing slots from GETKEYSINSLOT and COUNTKEYSINSLOT (kvstore should not support access into hashtable that is importing, even by direct reference)

• : 15. Somewhat orthogonal to ASM, but some refactoring of RDB/networking code to make replication and slot migration decoupled

• : Configurable auth options for CLUSTER MIGRATE #2392

• Defer slot migration failover until async modules are consistent #2609

• [NEW] Use atomic slot migration in valkey-cli #2504

• Split SLOT_EXPORT_AUTHENTICATING into SEND and READ to avoid synchronous reading of auth response #2494

[dvkashapov]

Hello! I hope this is the right place to discuss this: I was testing out #1949 on unstable and migrations seem to stop working when you set hz 1, every 5 sec logs show waiting before snapshotting due to pending writes in output buffer Is it expected behaviour? Tried on x86 and Arm, same behaviour.
Added some prints for pending_write flag on client and it seems that for hz 1 state machine always checks client when it has pending_write flag set to 1
Tried some other values other than 5000 ms for run_with_period() and it did not help, maybe this will help (or not, but worth trying)

Debug server logs + pending_write print

66205:M 01 Sep 2025 12:23:56.507 * Slot migration {name: 0ea7075b13bbf2cc9727e09daaf1c001d5f122e0, operation: export, target_node_id: 61fd5b0d371a22cea8537a08502a5ad0a323aa15, slots: 7777-7777} waiting before snapshotting due to pending writes in output buffer.
!!! handleClientsWithPendingWrites pending write 1, setting 0
66205:M 01 Sep 2025 12:23:57.333 . --- Processing packet of type ping, 2304 bytes
66205:M 01 Sep 2025 12:23:57.333 . Sending pong packet to node 61fd5b0d371a22cea8537a08502a5ad0a323aa15 () on inbound link
66205:M 01 Sep 2025 12:23:57.333 . ping packet received: 61fd5b0d371a22cea8537a08502a5ad0a323aa15
!!! network.c pending write 0, setting 1
!!! handleClientsWithPendingWrites pending write 1, setting 0
!!! network.c pending write 0, setting 1
66205:M 01 Sep 2025 12:23:58.509 . Sending ping packet to node 61fd5b0d371a22cea8537a08502a5ad0a323aa15 () on outbound link
!!! handleClientsWithPendingWrites pending write 1, setting 0
66205:M 01 Sep 2025 12:23:58.510 . --- Processing packet of type pong, 2304 bytes
66205:M 01 Sep 2025 12:23:58.510 . pong packet received: 61fd5b0d371a22cea8537a08502a5ad0a323aa15
!!! network.c pending write 0, setting 1
!!! handleClientsWithPendingWrites pending write 1, setting 0
66205:M 01 Sep 2025 12:24:00.336 . --- Processing packet of type ping, 2304 bytes
66205:M 01 Sep 2025 12:24:00.337 . Sending pong packet to node 61fd5b0d371a22cea8537a08502a5ad0a323aa15 () on inbound link
66205:M 01 Sep 2025 12:24:00.337 . ping packet received: 61fd5b0d371a22cea8537a08502a5ad0a323aa15
!!! network.c pending write 0, setting 1
!!! handleClientsWithPendingWrites pending write 1, setting 0
66205:M 01 Sep 2025 12:24:01.511 . Total: 2 clients connected (0 replicas), 2310032 (2.20M) bytes in use
!!! network.c pending write 0, setting 1
66205:M 01 Sep 2025 12:24:01.511 * Slot migration {name: 0ea7075b13bbf2cc9727e09daaf1c001d5f122e0, operation: export, target_node_id: 61fd5b0d371a22cea8537a08502a5ad0a323aa15, slots: 7777-7777} waiting before snapshotting due to pending writes in output buffer.

[madolson]

: Containment of keys on replicas

We'll focus on this for GA. We'll punt the remaining items out to 9.1. We'll discuss next week about the state of search with respect to consistency.

@murphyjacob4 Please add the relevant PRs and issues so we can keep track. Viktor wants this, and you should make him happy.

[murphyjacob4]

Hey @dvkashapov - just posted a PR to fix the hz 1 issue: #2636

[murphyjacob4]

Address leakage of importing keys in stats and slot-stats (either let it leak, or be consistent about containing it throughout)

I just checked - in the current implementation it is not leaked in these:

1. For stats, we don't include it in the KEYSPACE section
2. For slot-stats, we already don't return unassigned slots, so this is solved

I have filed a PR to add tests for the above: #2638. Keeping as a draft until the base PR (replica key containment) is merged