diff --git a/src/current/_includes/releases/v25.3/v25.3.0-alpha.2.md b/src/current/_includes/releases/v25.3/v25.3.0-alpha.2.md index 1ea7394e3cd..dd67b55ca0c 100644 --- a/src/current/_includes/releases/v25.3/v25.3.0-alpha.2.md +++ b/src/current/_includes/releases/v25.3/v25.3.0-alpha.2.md @@ -30,7 +30,7 @@ Release Date: June 16, 2025 - Added an `alter_changefeed` structured log event to provide more visibility into when an `ALTER CHANGEFEED` event occurred and what changed. [#147679][#147679] -- Added new timeseries metrics to the `storage.value_separation.*` namespace for observing the behavior of storage engine value separation. +- Added new [timeseries metrics to the `storage.value_separation.*` namespace]({% link v25.3/essential-metrics-self-hosted.md %}#storage-value-separation) for observing the behavior of [storage engine value separation]({% link v25.3/architecture/storage-layer.md %}#value-separation). [#147728][#147728]

DB Console changes

diff --git a/src/current/_includes/releases/v25.3/v25.3.0-beta.1.md b/src/current/_includes/releases/v25.3/v25.3.0-beta.1.md index e55916fdfdc..03c512d7c1b 100644 --- a/src/current/_includes/releases/v25.3/v25.3.0-beta.1.md +++ b/src/current/_includes/releases/v25.3/v25.3.0-beta.1.md @@ -15,7 +15,7 @@ Release Date: July 2, 2025

Operational changes

-- Introduced the following cluster settings for enabling and configuring value separation in the storage engine: `storage.value_separation.enabled`, `storage.value_separation.minimum_size`, and `storage.value_separation.max_reference_depth`. +- Introduced the following cluster settings for enabling and configuring [value separation in the storage engine]({% link v25.3/architecture/storage-layer.md %}#value-separation): `storage.value_separation.enabled`, `storage.value_separation.minimum_size`, and `storage.value_separation.max_reference_depth`. [#148535][#148535] - Non-admin users no longer have access to changefeed jobs they do not own and which are not owned by a role of which they are a member, regardless of whether they have the `CHANGEFEED` privilege on the table or tables those jobs may be watching. Admin users, or those with global `SHOWJOB` / `CONTROLJOB` privileges, can still interact with all jobs, regardless of ownership. [#148537][#148537] diff --git a/src/current/_includes/v25.3/essential-metrics.md b/src/current/_includes/v25.3/essential-metrics.md index 7c958db3f50..372aa259f33 100644 --- a/src/current/_includes/v25.3/essential-metrics.md +++ b/src/current/_includes/v25.3/essential-metrics.md @@ -40,6 +40,10 @@ The **Usage** column explains why each metric is important to visualize in a cus | rocksdb.compactions | rocksdb.compactions.total | Number of SST compactions | This metric reports the number of a node's [LSM compactions]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#lsm-health). If the number of compactions remains elevated while the LSM health does not improve, compactions are not keeping up with the workload. If the condition persists for an extended period, the cluster will initially exhibit performance issues that will eventually escalate into stability issues. | | rocksdb.block.cache.hits | rocksdb.block.cache.hits | Count of block cache hits | This metric gives hits to block cache which is reserved memory. It is allocated upon the start of a node process by the [`--cache` flag]({% link {{ page.version.version }}/cockroach-start.md %}#general) and never shrinks. By observing block cache hits and misses, you can fine-tune memory allocations in the node process for the demands of the workload. | | rocksdb.block.cache.misses | rocksdb.block.cache.misses | Count of block cache misses | This metric gives misses to block cache which is reserved memory. It is allocated upon the start of a node process by the [`--cache` flag]({% link {{ page.version.version }}/cockroach-start.md %}#general) and never shrinks. By observing block cache hits and misses, you can fine-tune memory allocations in the node process for the demands of the workload. | +| storage.value_separation.blob_files.count | storage.value_separation.blob_files.count | The number of blob files that are used to store [separated values]({% link {{ page.version.version }}/architecture/storage-layer.md %}#value-separation) within the storage engine. | Use this metric to track how many values (of key-value pairs) are being stored outside of the [LSM]({% link {{ page.version.version }}/architecture/storage-layer.md %}#log-structured-merge-trees) by the storage engine due to their large size. | +| storage.value_separation.blob_files.size | storage.value_separation.blob_files.size | The size of the physical blob files that are used to store [separated values]({% link {{ page.version.version }}/architecture/storage-layer.md %}#value-separation) within the storage engine. This sum is the physical post-compression sum of the `storage.value_separation.value_bytes.referenced` and `storage.value_separation.value_bytes.unreferenced` metrics. | Use this metric to see how much of your physical storage capacity is being used by separated values in blob files. | +| storage.value_separation.value_bytes.referenced | storage.value_separation.value_bytes.referenced | The size of storage engine value bytes (pre-compression) that are [stored separately in blob files]({% link {{ page.version.version }}/architecture/storage-layer.md %}#value-separation) and referenced by a live [SSTable]({% link {{ page.version.version }}/architecture/storage-layer.md %}#ssts). | Use this metric to see how much live (a.k.a., not yet eligible for compaction) blob storage is in use by separated values. | +| storage.value_separation.value_bytes.unreferenced | storage.value_separation.value_bytes.unreferenced | The size of storage engine value bytes (pre-compression) that are [stored separately in blob files]({% link {{ page.version.version }}/architecture/storage-layer.md %}#value-separation) and not referenced by any live [SSTable]({% link {{ page.version.version }}/architecture/storage-layer.md %}#ssts). These bytes are garbage that could be reclaimed by a [compaction]({% link {{ page.version.version }}/architecture/storage-layer.md %}#compaction). | Use this metric to see how much blob storage is dead and waiting to be compacted. | ## Health diff --git a/src/current/v25.3/architecture/storage-layer.md b/src/current/v25.3/architecture/storage-layer.md index 9fc28659e67..bbbb64039c5 100644 --- a/src/current/v25.3/architecture/storage-layer.md +++ b/src/current/v25.3/architecture/storage-layer.md @@ -80,6 +80,34 @@ The SSTs within each level are guaranteed to be non-overlapping: for example, if LSM tree with SST files +##### Write amplification + +_Write amplification_ measures the volume of data written to disk relative to the volume of data logically committed to the storage engine. When values are committed, CockroachDB writes them to the [write-ahead log (WAL)](#memtable-and-write-ahead-log) and then to [SSTables](#ssts) during flushes. [Compactions](#compaction) rewrite those SSTables multiple times over the value's lifetime. Most write amplification, and write bandwidth more broadly, originates from compactions. + +This is a necessary tradeoff, because if the storage engine performs too few compactions, the size of [L0](#lsm-levels) will get too large and an inverted LSM will result, which also has ill effects. In contrast, writes to the WAL are a small fraction of a [store]({% link {{ page.version.version }}/cockroach-start.md %}#store)'s overall write bandwidth and IOPs. + +Reducing write amplification helps lower disk bandwidth usage, CPU consumption for compactions, and storage costs. + +##### Read amplification + +_Read amplification_ measures the number of SSTable files consulted to satisfy a logical read. High read amplification occurs when value lookups must search multiple LSM levels or SST files, such as in an inverted LSM state. Keeping read and [write amplification](#write-amplification) in balance is critical for optimal storage engine performance. + +Read amplification is high [when the LSM is inverted](#inverted-lsms). In the inverted LSM state, reads need to start in higher levels and "look down" through a lot of SSTs to read a key's correct (freshest) value. + +Read amplification can be especially bad if a large [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) is overloading the cluster (due to insufficient CPU and/or IOPS) and the storage engine has to consult many small SSTs in L0 to determine the most up-to-date value of the keys being read (e.g., using a [`SELECT`]({% link {{ page.version.version }}/select-clause.md %})). + +A certain amount of read amplification is expected in a normally functioning CockroachDB cluster. For example, a read amplification factor less than 10 as shown in the [**Read Amplification** graph on the **Storage** dashboard]({% link {{ page.version.version }}/ui-storage-dashboard.md %}#other-graphs) is considered healthy. + +##### Value separation + +{% include_cached new-in.html version="v25.3" %} + +The storage engine implements a performance optimization called _value separation_. When the engine encounters a key-value pair with a sufficiently large value component, it stores the key in the [LSM](#log-structured-merge-trees) alongside a pointer to the value's location in a _blob file_ which is located outside the LSM. This indirection allows [compactions](#compaction) of the LSM to skip rewriting large values over and over; instead, compactions can copy a pointer to the large value's location. + +Value separation is especially beneficial for workloads with large values relative to key size (for example, [Raft log]({% link {{ page.version.version }}/architecture/replication-layer.md %}#raft) entries). It reduces [write amplification](#write-amplification) by about 50%, at the cost of about 20% in [space amplification]({% link {{ page.version.version }}/operational-faqs.md %}#space-amplification). What this means in practice is that the storage engine uses far less [IOPS]({% link {{ page.version.version }}/common-issues-to-monitor.md %}#disk-iops) and storage bandwidth overall, which are expensive, at the cost of an increase in storage capacity, which is much cheaper. + +To monitor this feature, see [the documentation for the metrics in the `storage.value_separation.*` namespace]({% link {{ page.version.version }}/essential-metrics-self-hosted.md %}#storage-value-separation). + ##### Compaction The process of merging SSTs and moving them from L0 down to L6 in the LSM is called _compaction_. The storage engine works to compact data as quickly as possible. As a result of this process, lower levels of the LSM should contain larger SSTs that contain less recently updated keys, while higher levels of the LSM should contain smaller SSTs that contain more recently updated keys. @@ -102,19 +130,6 @@ During normal operation, the LSM should look like this: ◣. An inverted LSM loo An inverted LSM will have degraded read performance. - - -Read amplification is high when the LSM is inverted. In the inverted LSM state, reads need to start in higher levels and "look down" through a lot of SSTs to read a key's correct (freshest) value. When the storage engine needs to read from multiple SST files in order to service a single logical read, this state is known as _read amplification_. - -Read amplification can be especially bad if a large [`IMPORT INTO`]({% link {{ page.version.version }}/import-into.md %}) is overloading the cluster (due to insufficient CPU and/or IOPS) and the storage engine has to consult many small SSTs in L0 to determine the most up-to-date value of the keys being read (e.g., using a [`SELECT`]({% link {{ page.version.version }}/select-clause.md %})). - -A certain amount of read amplification is expected in a normally functioning CockroachDB cluster. For example, a read amplification factor less than 10 as shown in the [**Read Amplification** graph on the **Storage** dashboard]({% link {{ page.version.version }}/ui-storage-dashboard.md %}#other-graphs) is considered healthy. - - - -_Write amplification_ measures the volume of data written to disk, relative to the volume of data logically committed to the storage engine. When a value is committed to the storage engine, CockroachDB writes it once to the [write-ahead log (WAL)](#memtable-and-write-ahead-log). CockroachDB writes the value again when flushing it to an [SSTable](#ssts). CockroachDB subsequently writes the value multiple times as part of [compactions](#compaction) over the lifetime of the value. Most write amplification, and write bandwidth more broadly, originates from compactions. This is a necessary tradeoff, because if the storage engine performs too few compactions, the size of [L0](#lsm-levels) will get too large and an inverted LSM will result, which also has ill effects. In contrast, writes to the WAL are a small fraction of a [store]({% link {{ page.version.version }}/cockroach-start.md %}#store)'s overall write bandwidth and IOPs. - -Read amplification and write amplification are key metrics for LSM performance. Neither is inherently "good" or "bad", but they must not occur in excess, and for optimum performance they must be kept in balance. That balance involves tradeoffs. Inverted LSMs also have excessive compaction debt. In this state, the storage engine has a large backlog of [compactions](#compaction) to do to return the inverted LSM to a normal, non-inverted state. diff --git a/src/current/v25.3/cockroachdb-feature-availability.md b/src/current/v25.3/cockroachdb-feature-availability.md index 69521140f78..92762a671be 100644 --- a/src/current/v25.3/cockroachdb-feature-availability.md +++ b/src/current/v25.3/cockroachdb-feature-availability.md @@ -47,6 +47,10 @@ Any feature made available in a phase prior to GA is provided without any warran **The following features are in preview** and are subject to change. To share feedback and/or issues, contact [Support](https://support.cockroachlabs.com/hc). {{site.data.alerts.end}} +### Value separation + +[Value separation]({% link {{ page.version.version }}/architecture/storage-layer.md %}#value-separation) reduces write amplification by storing large values separately from the LSM in blob files. Value separation is available as a Public Preview feature. This feature can reduce write amplification by up to 50% for large-value workloads, while introducing minor read overhead and a slight increase in disk space usage. + ### `database` and `application_name` labels for certain metrics diff --git a/src/current/v25.3/operational-faqs.md b/src/current/v25.3/operational-faqs.md index 7a073b9c0dc..658a1430fe6 100644 --- a/src/current/v25.3/operational-faqs.md +++ b/src/current/v25.3/operational-faqs.md @@ -77,7 +77,9 @@ For more information about how MVCC works, see [MVCC]({% link {{ page.version.ve ### The data could be in the process of being compacted -When MVCC garbage is deleted by garbage collection, the data is still not yet physically removed from the filesystem by the [Storage Layer]({% link {{ page.version.version }}/architecture/storage-layer.md %}). Removing data from the filesystem requires rewriting the files containing the data using a process also known as [compaction]({% link {{ page.version.version }}/architecture/storage-layer.md %}#compaction), which can be expensive. The storage engine has heuristics to compact data and remove deleted rows when enough garbage has accumulated to warrant a compaction. It strives to always restrict the overhead of obsolete data (called the space amplification) to at most 10%. If a lot of data was just deleted, it may take the storage engine some time to compact the files and restore this property. + + +When MVCC garbage is deleted by garbage collection, the data is still not yet physically removed from the filesystem by the [Storage Layer]({% link {{ page.version.version }}/architecture/storage-layer.md %}). Removing data from the filesystem requires rewriting the files containing the data using a process also known as [compaction]({% link {{ page.version.version }}/architecture/storage-layer.md %}#compaction), which can be expensive. The storage engine has heuristics to compact data and remove deleted rows when enough garbage has accumulated to warrant a compaction. It strives to limit the overhead of this obsolete data (called the _space amplification_) to a small fixed percentage. If a lot of data was just deleted, it may take the storage engine some time to compact the files and restore this property. {% include {{page.version.version}}/storage/free-up-disk-space.md %}