DOC-13173 Enhance documentation for Index Rebalance and Upgrade #3805
Conversation
|
Swap rebalance is not an index rebalance method like DCP/ file based, can be excluded from here: https://preview.docs-test.couchbase.com/docs-server-DOC_13173_Rebalance_doc_enhancements/server/current/learn/clusters-and-availability/rebalance-and-index-service.html#index-rebalance-methods |
can we rephrase this as "index-shard affinity is introduced"?
this is only true for on-prem deployments. For Capella, the default is FBR. I'm not sure if we are using the same for Server and Capella.
index partitions may not share same shard ID. Partition and its replicas share same slot ID. in mixed versions, we run FBR for provisioned clusters so that all 7.6 clusters have shard affinity by default.
like mentioned above, mixed mode functionality is supported for provisioned clusters |
|
@Nischal1729, @NightWing1998 |
|
@shivanshrustagi |
|
All discussed contents, then the ones shared by Nischal and Shivansh, including the responses to my questions (excluding 2 to 3) and review comments, all are documented. |
| NOTE: If at least one node in the cluster is running Server 7.1 or a later version, most smart batching features apply across the cluster, | ||
| even when some nodes are running earlier versions. | ||
|
|
||
| ==== Empty Node Batching |
There was a problem hiding this comment.
We should add explanation in case rebalance is cancelled / failed and then retried. In such cases, it won't restart from previously known state, and index rebalance may choose to move indexes across all nodes.
CC @Nischal1729
| === Swap Rebalance for Index Service | ||
|
|
||
| A swap rebalance removes one or more source nodes and adds the same number of destination nodes. | ||
| During the swap rebalance operation, indexes move only between those source and destination nodes. |
There was a problem hiding this comment.
As mentioned in the other review comment, it is better to mention what happens on cancelled / failed rebalance + retry.
|
Hi @rao-shwe, For example, index rebalance failure and restart, empty-node batching is not being used, optimize index placement is enabled etc. Let's get these topics documented and highlighted properly (), if already not. Thanks. |
|
@shivanshrustagi and @Nischal1729,
Please review the technical correctness. Cc: @amithk Preview doc site login credentials. |
|
|
||
| The Index Service maintains cluster-wide index definitions and metadata to <<#index-redistribution,redistribute indexes and replicas>> during rebalance operations. | ||
|
|
||
| The rebalance operation evaluates each node's CPU, RAM, and disk bandwidth to minimize effects on database performance. |
There was a problem hiding this comment.
The rebalance operation evaluates each node's CPU, RAM, and disk bandwidth to minimize effects on database performance.
Maybe we can mention Availability of indexes as well.
The rebalance operation evaluates each node's CPU, RAM, disk bandwidth and user defined Server Groups(Availability Zones) to minimize effects on database performance and maximise availability of indexes
| As a result, this design provides the following benefits: | ||
|
|
||
| * Provides deterministic collocation, where all indexes with the same shard ID share the same shard files. | ||
| * Prevents replica-on-same-node conflicts. |
There was a problem hiding this comment.
While this is true, shard affinity alone did not enable this. We had support for this when replicated indexes were introduced.
@Nischal1729 @rao-shwe maybe we can add replica repair benefits.
There was a problem hiding this comment.
While this is true, shard affinity alone did not enable this. We had support for this when replicated indexes were introduced.
I think context for adding these points was to emphasise why GSI has to maintain alternate shard ID etc, and plasma itself is not directly handling this assignment, so it makes sense in that context- maybe we can exclude adding these to docs or give additional context.
|
|
||
| A rebalance automatically redistributes indexes in the following situations: | ||
|
|
||
| * *Rebalance when you add an index node*: |
There was a problem hiding this comment.
Only enabled by default in Capella.
On-prem customers have to enable redistribute_indexes flag
| Rebalance always moves indexes off of nodes that you're removing from the cluster to the remaining nodes. | ||
| A rebalance does not affect indexes that reside on nodes that you're not removing. | ||
|
|
||
| * *Rebalance when you add and remove index nodes*: |
There was a problem hiding this comment.
This is technically incorrect, reditribute_indexes flag makes every index in the cluster as eligible for planning/movement.
Whenever we have any node getting removed, only the Indexes on the to-be-removed nodes are considered for planning. So for Add + Remove Index, the indexes on the Removed node gets moved to the new Node irrespective of the redistribute_indexes flag
| A retry triggers a full re-planning cycle and not a continuation. | ||
| The new plan may choose different nodes, causing more indexes to move than in the previous attempt. | ||
|
|
||
| * Optimize Index Placement is enabled: |
There was a problem hiding this comment.
Maybe add Caution/Important note, where we can see the rebalance time increasing due to tons of index movement, when flag is enabled when it need not be used
|
This point can be removed as it is expected that all indexes on the removed node will be moved during swap rebalance:
|
amithk
left a comment
amithk
left a comment
There was a problem hiding this comment.
I have reviewed until before the examples section. Will review examples and rest of the sections later.
|
|
||
| * xref:learn:services-and-indexes/indexes/storage-modes.adoc#memory-optimized-index-storage[Memory Optimized GSI Storage], which stores most index structures in-memory and does not support file-based operations. | ||
|
|
||
| Plasma, the storage engine for GSI, stores index data in shards. |
There was a problem hiding this comment.
Plasma, the storage engine for standard GSI
|
|
||
| == Index Service | ||
|
|
||
| The Index Service maintains cluster-wide index definitions and metadata to <<#index-redistribution,redistribute indexes and replicas>> during rebalance operations. |
There was a problem hiding this comment.
Minor: Is it too early to refer to the index-redistribution section here?
| In Couchbase Server versions earlier than 7.6, Plasma automatically chose which shard an index partition belonged to. | ||
| Couchbase Server 7.6 introduces index-shard affinity and continues to support it in later versions. | ||
|
|
||
| With shard affinity, the GSI layer assigns a shard slot (an alternate shard ID) to each index partition and provides this information to Plasma. |
There was a problem hiding this comment.
This paragraph and a bullet point below talk about separate "layering" of GSI and storage. I think this needs to be reworded as such explanation is internal to the product. We should focus on explaining product behaviour without exposing internal details like, which layer implements what functionality.
| * Prevents replica-on-same-node conflicts. | ||
| * Enables efficient data movement through File-Based Rebalance. | ||
|
|
||
| Shard affinity keeps all index partitions with the same shard ID on the same node and moves them as a single unit during index rebalancing. |
There was a problem hiding this comment.
@Nischal1729 is this always true? Across all permutations of partition placements and further index movements?
There was a problem hiding this comment.
Wording can be clearer here that it is referring to the shard UUID by plasma and not alternate shard ID. But given that shard affinity is there and other prerequisites for shard based rebalance are there this should always be true.
| ==== | ||
| * File-Based Rebalance is supported only if you have enabled Standard Global Secondary xref:manage:manage-settings/general-settings.adoc#index-storage-mode[Index Storage Mode] on your cluster. | ||
|
|
||
| * You cannot use File-Based Rebalance if you have enabled xref:learn:services-and-indexes/indexes/storage-modes.adoc#memory-optimized-index-storage[Memory Optimized Index Storage] in the xref:manage:manage-settings/general-settings.adoc#index-storage-mode[Index Storage Mode] settings on your cluster because it does not store index metadata in files. |
There was a problem hiding this comment.
because it does not store index metadata in files.
This needs to be reworded. In general, I don't see a need to provide reason here. We can just say: with memory optimised indexes, the file based rebalance is not supported.
| .. <<#enabling-fbr,Enable File-Based Rebalance>> (or shard affinity). | ||
|
|
||
| .. When the Rebalance operation is triggered, Couchbase Server does not use File-Based Rebalance method for those indexes right away, | ||
| because those indexes do not have <<#shard-affinity,Shard Affinity>>. |
There was a problem hiding this comment.
because those indexes do not have "shard affinity metadata".
|
|
||
| If a File-Based Rebalance fails, you can start a new rebalance. | ||
| The subsequent rebalance does not repeat any shard transfers that completed successfully during the earlier attempt. | ||
| The subsequent rebalance only transfers the indexes that were incomplete or still in progress when the failure occurred. |
There was a problem hiding this comment.
This doesn't seem to be correct. After restart of the rebalance, entirely new plan is generated and a new set of index movements will be attempted.
Let's double check this @Nischal1729.
It is already explained in detail below.
There was a problem hiding this comment.
Yes this statement is incorrect:
The subsequent rebalance only transfers the indexes that were incomplete or still in progress when the failure occurred.
After failure/cancel the cluster reverts to normal operation with whatever placements were fully committed and unfinished moves are rolled back. When new rebalance is started, it will use a fresh plan and corresponding index movements, which are not related to the older incomplete index movements.
|
|
||
| The default batch size is `3`, which means that a rebalance rebuilds up to 3 indexes at the same time. | ||
|
|
||
| NOTE: Smart batching does not work if you have enabled File-Based Rebalance. |
|
|
||
| Users with Full Admin or Cluster Admin roles can xref:rest-api:rest-modify-index-batch-size.adoc[Modify Index Batch Size] using the REST API. | ||
|
|
||
| You can use smart batching for one or more of the following purposes: |
There was a problem hiding this comment.
I am not sure if the information added by the bullet points here is consumable by the customers.
We can just say that "smart batching" optimises the batching of the indexes during index transfer done during rebalance. This speeds up the index rebalance process. That should be enough. The bullet points don't have any extra value.
|
|
||
| Empty node batching is a behavior in rebalance planner that occurs when a newly added Index Service node has no indexes on it. | ||
| Because the node is empty, there are no placement conflicts, no replica-placement constraints, and no query workload running on the node. | ||
| This allows the planner to group multiple index movements together and schedule them as a single batch, rather than planning each index transfer individually. |
There was a problem hiding this comment.
We should refer to empty node batch size setting , and mention how it is different than regular batch size setting. We should also mention the defaults for both of these batch sized. May be add a note.
|
@amithk, @Nischal1729, and @shivanshrustagi, You complete the task of reviewing and adding your comments. I'll begin implementing your comments on Friday morning. I started working on a few comments last evening, but it can create a conflict with the version where you are adding comments. So, I'll continue implementing your comments after a round of your reviews are complete. |
DOC-13173
Preview doc login details.
New page: Index Rebalance.
(Currently, I've set ToC/Headings on the RHS to level-3 for your convenience. It looks crowded, so I'll reduce it to 2 before publishing.)
Please ignore the preview yml file.