diff --git a/modules/develop/pages/transactions.adoc b/modules/develop/pages/transactions.adoc index 6dde9265f2..afa7fffe58 100644 --- a/modules/develop/pages/transactions.adoc +++ b/modules/develop/pages/transactions.adoc @@ -320,11 +320,13 @@ Different transactions require different approaches to handling failures within == Transactions with compacted segments -Transactions are supported on topics with compaction configured. The compaction process removes the aborted transaction's data and all transactional control markers from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches). +Transactions are supported on topics with compaction configured. The compaction process removes aborted transaction data from the log. The resulting compacted segment contains only committed data batches (and potentially harmless gaps in the offsets due to skipped batches). ifndef::env-cloud[] At a cluster-level, compaction is set when xref:reference:cluster-properties.adoc#log_cleanup_policy[`log_cleanup_policy`] or xref:reference:topic-properties.adoc#cleanuppolicy[`cleanup.policy`] are set to either `compact` or `compact,delete`. +Optionally, you can enable removal of transactional control batches (commit and abort markers) during compaction by setting xref:reference:properties/cluster-properties.adoc#log_compaction_tx_batch_removal_enabled[`log_compaction_tx_batch_removal_enabled`] to `true`. When enabled, the topic's xref:reference:properties/topic-properties.adoc#delete-retention-ms[`delete.retention.ms`] setting is applied to these markers. For topics with a `compact` only cleanup policy, you must explicitly set `delete.retention.ms` at the topic level. This feature is not applied when Tiered Storage is enabled. See xref:manage:cluster-maintenance/compaction-settings.adoc#transactional-control-batch-removal[Transactional control batch removal]. + endif::[] include::shared:partial$suggested-reading.adoc[] diff --git a/modules/manage/pages/cluster-maintenance/compaction-settings.adoc b/modules/manage/pages/cluster-maintenance/compaction-settings.adoc index 17a8475445..a4c55a2f79 100644 --- a/modules/manage/pages/cluster-maintenance/compaction-settings.adoc +++ b/modules/manage/pages/cluster-maintenance/compaction-settings.adoc @@ -154,7 +154,25 @@ If obtaining a complete snapshot of the log, including tombstone records, is imp On the other hand, if more frequent cleanup of tombstones is important for optimizing workloads and space management, consider setting a shorter tombstone retention, for example the typical default of 24 hours (86400000 ms). -Compaction and tombstone removal are coordinated across replicas, preventing inconsistencies and ensuring that deleted records are properly recognized by all readers. As a result, tombstone removal on one replica may be delayed if another replica is stopped or lagging. +Compaction and tombstone removal are coordinated across replicas, preventing inconsistencies and ensuring that deleted records are properly recognized by all readers. As a result, tombstone removal on one replica may be delayed if another replica is stopped or lagging. + +== Transactional control batch removal + +Transactional workloads write control batches (commit and abort markers) to the log. By default, these markers are retained indefinitely. You can enable their removal during compaction by setting xref:reference:properties/cluster-properties.adoc#log_compaction_tx_batch_removal_enabled[`log_compaction_tx_batch_removal_enabled`] to `true`: + +[,bash] +---- +rpk cluster config set log_compaction_tx_batch_removal_enabled=true +---- + +When enabled, the xref:reference:properties/topic-properties.adoc#delete-retention-ms[`delete.retention.ms`] setting is applied to transactional control batches, removing them after the retention period. This uses the same coordinated compaction mechanism as tombstone removal. + +[NOTE] +==== +For topics with a `compact` only cleanup policy, you must explicitly set `delete.retention.ms` at the topic level to enable removal of transactional control batches. +==== + +Consider enabling this feature if you have compacted topics with heavy transactional workloads and observe disk usage from accumulated transaction markers. == Compaction policy settings