Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion modules/develop/pages/transactions.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -320,11 +320,13 @@ Different transactions require different approaches to handling failures within

== Transactions with compacted segments

Transactions are supported on topics with compaction configured. The compaction process removes the aborted transaction's data and all transactional control markers from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches).
Transactions are supported on topics with compaction configured. The compaction process removes aborted transaction data from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches).

ifndef::env-cloud[]
At a cluster-level, compaction is set when xref:reference:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] or xref:reference:topic-properties.adoc#cleanuppolicy[`cleanup.policy`] are set to either `compact` or `compact,delete`.

Optionally, you can enable removal of transactional control batches (commit and abort markers) during compaction by setting xref:reference:properties/cluster-properties.adoc#log_compaction_tx_batch_removal_enabled[`log_compaction_tx_batch_removal_enabled`] to `true`. When enabled, the topic's xref:reference:properties/topic-properties.adoc#delete-retention-ms[`delete.retention.ms`] setting is applied to these markers. For topics with a `compact` only cleanup policy, you must explicitly set `delete.retention.ms` at the topic level. This feature is not applied when Tiered Storage is enabled. See xref:manage:cluster-maintenance/compaction-settings.adoc#transactional-control-batch-removal[Transactional control batch removal].

endif::[]

include::shared:partial$suggested-reading.adoc[]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,25 @@ If obtaining a complete snapshot of the log, including tombstone records, is imp

On the other hand, if more frequent cleanup of tombstones is important for optimizing workloads and space management, consider setting a shorter tombstone retention, for example the typical default of 24 hours (86400000 ms).

Compaction and tombstone removal are coordinated across replicas, preventing inconsistencies and ensuring that deleted records are properly recognized by all readers. As a result, tombstone removal on one replica may be delayed if another replica is stopped or lagging.
Compaction and tombstone removal are coordinated across replicas, preventing inconsistencies and ensuring that deleted records are properly recognized by all readers. As a result, tombstone removal on one replica may be delayed if another replica is stopped or lagging.

== Transactional control batch removal

Transactional workloads write control batches (commit and abort markers) to the log. By default, these markers are retained indefinitely. You can enable their removal during compaction by setting xref:reference:properties/cluster-properties.adoc#log_compaction_tx_batch_removal_enabled[`log_compaction_tx_batch_removal_enabled`] to `true`:

[,bash]
----
rpk cluster config set log_compaction_tx_batch_removal_enabled=true
----

When enabled, the xref:reference:properties/topic-properties.adoc#delete-retention-ms[`delete.retention.ms`] setting is applied to transactional control batches, removing them after the retention period. This uses the same coordinated compaction mechanism as tombstone removal.

[NOTE]
====
For topics with a `compact` only cleanup policy, you must explicitly set `delete.retention.ms` at the topic level to enable removal of transactional control batches.
====

Consider enabling this feature if you have compacted topics with heavy transactional workloads and observe disk usage from accumulated transaction markers.

== Compaction policy settings

Expand Down