RFC: bypass defrag prevention on mostly-matchable content

[XciD]

Stacked on #886 (base branch set accordingly); only the last commit is new. Opening as an RFC: this changes upload dedup behavior and the threshold deserves discussion.

Problem

Defrag prevention re-stores dedup ranges shorter than its rolling chunks-per-range average, so the re-stored chunks merge with neighboring new data into long contiguous fresh runs. That reasoning only holds when the surrounding content is mostly new.

On mostly-matchable content, i.e. a re-upload of a file whose bytes already sit in CAS with a fragmented layout (interrupted-and-retried imports, mounts re-writing files, copy fallbacks), the throttle degenerates into a worst-of-both-worlds fixed point: it alternates accepted and rejected ranges around the hysteresis band, so the file keeps referencing the fragmented old ranges AND re-stores roughly half the bytes as fresh xorbs.

Observed on production Storage Bucket files (multi-GB media re-imported over an interleaved first upload):

• ~2.1x stored bytes (18.6 GB of xorb extents referenced by an 8.76 GB file)
• +90-160% xorb count vs ideal, reconstructions of 15-17k segments
• terms alternating gap-2 between old xorbs and re-stored ones
• CPR pinned at exactly the min_n_chunks_per_range hysteresis equilibrium: 8.07 and 7.8 measured on two files, which is the fingerprint of the throttle oscillating

Reproduced end to end with the two ignored exploration harnesses included in this PR (local test server, no network). Key findings from the harness:

• the pathology needs variance in match-run lengths: with uniform run lengths the throttle never rejects (rejection compares against the rolling average);
• a single re-upload reaches the degenerate fixed point (it does not decay further across generations);
• with the throttle disabled, the re-upload dedups 100% and simply inherits the reference layout with zero storage waste.

Change

Track how many processed chunks had a dedup match on offer, whether the throttle accepted or rejected them. Once that matchable density exceeds deduplication.defrag_prevention_matchable_density_bypass (new config, default 0.7, env HF_XET_DEDUPLICATION_DEFRAG_PREVENTION_MATCHABLE_DENSITY_BYPASS, set > 1.0 for the previous behavior), the throttle is bypassed: skipping dedup on content that is almost entirely matchable buys no contiguity — the file still references the fragmented ranges in between — and only duplicates storage. A 512-chunk warm-up prevents a tiny early sample from flipping the gate.

Validation with the harnesses:

scenario	before	after
re-upload over fragmented reference (variable pieces)	defrag_prevented 5.9%, re-stored 30.9%	defrag_prevented 0%, re-stored 25.0% (= only genuinely new CDC-boundary chunks)
same, uniform pieces	100% dedup (throttle idle)	unchanged
sparse short matches in mostly-new content (~26% dedup)	throttle idle, 172 terms	byte-identical
existing regression test (#886, defrag_prevented > 0)	passes	passes (warm-up window still exercises the throttle)

Full xet-data suite passes (284 lib tests + integration suites).

Discussion points

1. Threshold value: 0.7 catches the re-upload pathology with margin. A conservative 0.9+ would still catch full re-uploads (density ~1.0) while leaving incremental-update-heavy workloads (checkpoints sharing most content with short scattered matches) under the current policy. Genuinely unclear which default is right — input welcome.
2. Cumulative vs rolling density: the prototype uses cumulative-since-start, so a file that is new for its first half and duplicate for its second may not cross the gate. A rolling window (like the CPR estimator's) would adapt; happy to implement if the mechanism is agreed on.
3. The known mitigation for affected deployments today is HF_XET_DEDUPLICATION_MIN_N_CHUNKS_PER_RANGE=0, but that gives up the throttle entirely; this PR keeps it for the case it was designed for.

Related: the test-only random_data() helper in range_upload.rs has a 16 MB period (bits 16..24 of i*K mod 2^24), so >16 MB test buffers dedup against themselves and distort layout measurements — worth a separate small fix; the harnesses here use the full-period DeterministicRng instead.