Skip to content

[perf] split: reuse the per-line buffer in round-robin mode - #14790

Merged
sylvestre merged 1 commit into
uutils:mainfrom
haydonryan:split-round-robin-vec-outside-loop
Sep 22, 2026
Merged

sylvestre merged 1 commit into
uutils:mainfrom
haydonryan:split-round-robin-vec-outside-loop

Conversation

@haydonryan

Copy link
Copy Markdown
Contributor

Small improvement today.

Moving the allocation out of the loop, and reuse the vec instead of allocating and freeing it every line. Because it's allocated on every line, it will stretch (reallocate) if the line length exceeds the allocated buffer.

Clanker generated:

Vec Mechanics recap:

  • First byte pushed → allocates a capacity-8 buffer.
  • Every time len hits capacity, it reallocs to 2× the current capacity (amortized O(1) appends).

So for read_until reusing the buffer: it grows through the doubling sequence only while a single line is being read and only until it fits that line; thereafter clear() keeps whatever
capacity that line reached, and shorter subsequent lines reuse it with zero realloc. The 64-byte figure in my earlier answer was just an illustration of "a capacity value in the growth
series," not a fixed allocation size.

Verification (A/B on this branch, main vs split-round-robin-vec-outside-loop)

Release fat-LTO coreutils binaries, split --number=r/4 on a 500k-line file:

Metric main branch Δ
Binary size 13,163,224 B 13,163,416 B +192 B
Runtime (mean ± σ) 448.1 ± 21.9 ms 396.0 ± 37.1 ms 1.13x faster (±0.12)
User time 73.1 ms 62.6 ms −10.5 ms
Output byte-identical

cargo test -p uu_split passes.

Comment thread src/uu/split/src/split.rs
let mut closed_writers = 0;

let mut i = 0;
let mut line = Vec::new();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you please add a bench for the r/N mode in benches/split_bench.rs? otherwise codspeed won't track this gain

@sylvestre
sylvestre merged commit 68ea3f6 into uutils:main Sep 22, 2026
101 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants