[ThomasWaldmann]

Description

Implements the remaining item 1 of #10058 (the "pieces" refactor): the last avoidable full-size copies on the create side.

Before, each new chunk was copied up to three times before reaching the pack:

1. key.encrypt for authenticated-* keys: b"".join([header, tag, data]) copied the whole payload to prepend 34 bytes,
2. RepoObj.format: hdr + meta_encrypted + data_encrypted copied the encrypted data again,
3. PackWriter: b"".join(...) of all objects of a pack (needed anyway).

Now:

• KeyBase.encrypt_parts() returns the envelope as a list of parts; for authenticated-* keys that is [header + tag, data] (no payload copy), AEAD keys return [encrypt(...)]. encrypt() is unchanged in what it returns.
• RepoObj.format_parts() returns [header + meta_encrypted, *data_parts]; format() joins it, so its other callers are unchanged.
• Repository.put() / PackWriter.add() accept bytes or a list of parts (MAX_DATA_SIZE applies to the total); the PackWriter joins all parts of all objects once when it builds the pack.
• cache.add_chunk() (create / transfer / recreate hot path) uses format_parts().

A part may reference the caller's buffer: on create, the chunk data is a memoryview into the reused chunker buffer, and with an authenticated-* key and data that stays uncompressed, it ends up as a part as is. As the pack is only built later, PackWriter.add() copies every part that is not bytes (bytes parts are buffered by reference).

Full-size payload copies per new chunk, from encryption to the finished pack:

key	payload after compression	master	PR	saved
authenticated-*	stored as is (incompressible, or none): borrowed chunker buffer	3 (key join, format concat, pack join)	2 (PackWriter.add copy, pack join)	1
authenticated-*	compressed (fresh bytes)	3 (key join, format concat, pack join)	1 (pack join)	2
AEAD (encrypted)	any	2 (format concat, pack join)	1 (pack join)	1

The copies saved are of the payload after compression, so the case that saves 2 copies is also the one where the copied payload is smaller than the chunk.

Tests:

• test_format_parts: authenticated / chacha20-poly1305 / aes-ocb × none / lz4: joined parts parse, equal format() output (deterministic key), uncompressed authenticated data is not copied.
• test_put_parts: the caller overwrites its buffer after put() and before the flush; stored objects, offsets and sizes are still correct.
• test_max_data_size_parts: the size limit covers all parts together.

Benchmarks

Unmodified master vs. this branch (same compiled extensions), each tree's own path: format() / format_parts() → PackWriter.add → pack join + store hash, null store. 2 MiB chunks as a memoryview into one reused buffer (like the chunker), 200 chunks (~8 packs) per run, each run in a fresh process; old/new interleaved with alternating order, 2 batches × 7 rounds, medians, macOS / Apple Silicon.

key	compression	data	master ms/chunk	PR ms/chunk	batch 1	batch 2
authenticated (sha256)	none	random	0.838	0.80	−3.6%	−4.2%
authenticated (sha256)	lz4	random	0.88	0.84	−4.9%	−4.5%
authenticated-blake3	none	random	0.73	0.70	−4.5%	−4.3%
authenticated-blake3	lz4	random	0.79	0.75	−4.0%	−6.2%
chacha20-poly1305	none	random	1.05	1.02	−3.1%	−3.1%
chacha20-poly1305	lz4	random	1.10	1.07	−3.2%	−2.9%
aes-ocb	none	random	0.72	0.68	−5.0%	−5.3%
aes-ocb	lz4	random	0.76	0.72	−5.5%	−4.5%
all keys	lz4 / zstd,3	text	0.27–0.44	same	−3.8% … +2.4%	−2.1% … +1.7%

Incompressible data saves 1 copy of the full 2 MiB with every key (see the table above): ~25–50 µs per chunk. For comparison, an isolated 2 MiB copy costs ~50–55 µs here into an existing buffer and ~85 µs into a freshly allocated one (macOS returns large blocks to the OS, so a fresh one page-faults), so the saving is somewhat less than one isolated copy.

To see the 2-copy case, a third batch used data that lz4 compresses to ~50% (2 MiB of alternating 4 KiB random / 4 KiB zero blocks), so ~1 MiB payloads:

key	data	master ms/chunk	PR ms/chunk	change	saved
authenticated (sha256)	half	0.828	0.797	−3.8%	31 µs (2 × ~1 MiB)
authenticated-blake3	half	0.969	0.936	−3.4%	33 µs (2 × ~1 MiB)
chacha20-poly1305	half	0.938	0.923	−1.7%	15 µs (1 × ~1 MiB)
aes-ocb	half	0.769	0.764	−0.7%	5 µs (1 × ~1 MiB)

The authenticated keys save clearly more there than the AEAD keys (2 vs. 1 copies; an isolated 1 MiB copy costs ~11 µs here). The AEAD rows are close to the noise level, so the ratio between the two is not precise.

Compressible text (lz4: 7.8%, zstd,3: 1.3% of the input) leaves only small payloads to copy, so the difference is within noise there. On a whole borg create (reading, chunking, id hashing on top) the relative gain is smaller, in line with the 2–3% estimate in #10058.

Checklist

• PR is against master
• New code has tests
• Tests pass (full suite locally: 3673 passed, 1302 skipped)
• Commits are signed

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 89.34%. Comparing base (5a78cf4) to head (cd127b6).
⚠️ Report is 11 commits behind head on master.
✅ All tests successful. No failed tests found.

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #10547      +/-   ##
==========================================
- Coverage   89.36%   89.34%   -0.02%     
==========================================
  Files         104      104              
  Lines       20082    20092      +10     
  Branches     3152     3153       +1     
==========================================
+ Hits        17947    17952       +5     
- Misses       1478     1482       +4     
- Partials      657      658       +1     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.