KAFKA-20664: Clarify docs on max compaction lag, segment.ms, and segment.bytes for active segment rolling#22489
Merged
Merged
Conversation
…and max.compaction.lag.ms
alanlau28
changed the title
lucliu1108
reviewed
Jun 5, 2026
| This can be used to prevent log with low produce rate from remaining ineligible for compaction for an unbounded duration. If not set, logs that do not exceed min.cleanable.dirty.ratio are not compacted. Note that this compaction deadline is not a hard guarantee since it is still subjected to the availability of log cleaner threads and the actual compaction time. You will want to monitor the uncleanable-partitions-count, max-clean-time-secs and max-compaction-delay-secs metrics. | ||
| This can be used to prevent log with low produce rate from remaining ineligible for compaction for an unbounded duration. If not set, logs that do not exceed min.cleanable.dirty.ratio are not compacted. | ||
|
|
||
| Because the active segment is never compacted (as noted above), records become eligible for compaction only through active segment rolling. For a compacted topic the active segment is rolled when the first of these is reached: it grows to segment.bytes, or its age reaches the smaller of segment.ms and max.compaction.lag.ms. So max.compaction.lag.ms governs two distinct things. First, it triggers active segment rolling by lowering the effective time-based roll deadline to the smaller of segment.ms and max.compaction.lag.ms, moving older records out of the active segment. This active segment rolling is evaluated when records are appended, so a partition that has stopped receiving writes will not roll its active segment until the next append. Second, max.compaction.lag.ms then makes the rolled records eligible for compaction even when the log does not exceed min.cleanable.dirty.ratio. |
Contributor
There was a problem hiding this comment.
This active segment rolling is evaluated when records are appended
The size/time roll check in maybeRoll is indeed only invoked from the append path, but the active segment can still be rolled from the retention path without an append. Illustrating all these situations might be too much for the doc
We could consider dropping the sentence, or narrowing it to the operator-facing consequence, e.g. "lowering max.compaction.lag.ms won't force-roll an idle partition; a new produce is needed before the dirty records become eligible for compaction."
lucliu1108
reviewed
Jun 5, 2026
| "ineligible for compaction in the log. Only applicable for logs that are being compacted."; | ||
| "ineligible for compaction in the log. Only applicable for logs that are being compacted. " + | ||
| "Because the active segment is never compacted, for compacted topics this value also drives " + | ||
| "active segment rolling: the effective time-based roll deadline is the smaller of segment.ms " + |
Contributor
There was a problem hiding this comment.
the effective time-based roll deadline is
"threshold" might be more suitable than "deadline" here
Also, could you set each config name wrapped in tags, like <code>segment.ms</code>
lucliu1108
reviewed
Jun 5, 2026
lucliu1108
left a comment
lucliu1108
left a comment
Contributor
There was a problem hiding this comment.
@alanlau28 Thanks for the PR! overall LGTM
lucliu1108
reviewed
Jun 9, 2026
| This can be used to prevent log with low produce rate from remaining ineligible for compaction for an unbounded duration. If not set, logs that do not exceed min.cleanable.dirty.ratio are not compacted. Note that this compaction deadline is not a hard guarantee since it is still subjected to the availability of log cleaner threads and the actual compaction time. You will want to monitor the uncleanable-partitions-count, max-clean-time-secs and max-compaction-delay-secs metrics. | ||
| This can be used to prevent log with low produce rate from remaining ineligible for compaction for an unbounded duration. If not set, logs that do not exceed min.cleanable.dirty.ratio are not compacted. | ||
|
|
||
| Because the active segment is never compacted (as noted above), records become eligible for compaction only through active segment rolling. For a compacted topic the active segment is rolled when the first of these is reached: it grows to segment.bytes, or its age reaches the smaller of segment.ms and max.compaction.lag.ms. So max.compaction.lag.ms governs two distinct things. First, it triggers active segment rolling by lowering the effective time-based roll threshold to the smaller of segment.ms and max.compaction.lag.ms, moving older records out of the active segment. Second, max.compaction.lag.ms then makes the rolled records eligible for compaction even when the log does not exceed min.cleanable.dirty.ratio. |
Contributor
There was a problem hiding this comment.
Nit: For the last sentence, could change to:
even when the log's dirty ratio is below min.cleanable.dirty.ratio
mjsax
approved these changes
Jun 9, 2026
alanlau28
changed the title
Member
|
Thanks for the PR. Merged to |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Improves the documentation for
segment.bytes,segment.ms, andmax.compaction.lag.mswith respect to active segment rolling.Reviewers: Lucy Liu lucliu@confluent.io, Matthias J. Sax
matthias@confluent.io