Add snapshot-ready MinerU-HTML extraction with managed Dynamo

[VibhuJawa]

Summary

Adds MinerU-HTML main-content extraction and a Curator-native path for processing a complete Common Crawl snapshot through benchmarking/run.py on Slurm.

This consolidates the useful MinerU and managed-Dynamo work from the earlier PRs into one branch built on the upstream Ray 2.57 / scheduler-observability changes from NVIDIA-NeMo#2304. The implementation intentionally does not use custom async scheduling.

WARC-to-output architecture

• CommonCrawlWARCManifestSourceStage reads a frozen official warc.paths manifest and emits one deterministic Curator source task per complete WARC. It performs no snapshot discovery or URL generation.
• Curator's native Slurm-array source filtering assigns every WARC to exactly one logical shard. There is no second application-level work-unit manifest or double sharding.
• CommonCrawlWARCDownloadAndReadStage downloads and reads a WARC on the same Ray worker, supports configurable S3-compatible whole-object transport, and cleans up its temporary object after iteration.
• Response bodies are independently Zstandard-compressed and emitted in bounded document batches. This keeps HTML memory bounded while allowing simplify, inference-client, and extraction stages to scale independently.
• Compressed HTML is dropped before the final Parquet write unless an explicitly requested audit field requires it.
• Output chunks have deterministic filenames derived from their source WARC and Curator task identity.

Download, simplify, inference-client, extraction, Ray object-store, Slurm-array, and GPU-node concurrency remain independent tuning controls.

MinerU serving baseline

• Curator-managed Dynamo 1.4 / vLLM 0.26 with one replica per GPU.
• opendatalab/MinerU-HTML-v1.1-hunyuan0.5B-compact.
• FP8 KV cache, prefix caching, ArcticInference suffix speculation with 16 draft tokens, and FULL_AND_PIECEWISE CUDA graphs.
• The real per-document structured-output regex is passed through Dynamo's supported guided_regex request extension.
• Native/custom async scheduling is disabled because this suffix path does not support it.
• Default model context length is 32,768 and the default per-element DOM cutoff is 500.
• Independently compressed HTML cells are expanded one at a time; no uncompressed-size column is required.

Recovery and verification

Pipeline.run(checkpoint_path=...) tracks completion at the source-WARC boundary. A retry preserves the logical shard geometry and replays only unfinished source parents. Existing chunks are overwritten at deterministic paths instead of being appended as duplicate files.

A dependent CPU run.py entry verifies:

• all logical shard completion manifests are present;
• every output Parquet footer is readable and includes url, text, and _mineru_status;
• the output contains at least one deterministic chunk per input WARC;
• sampled outputs satisfy status, non-empty-text, and conversion-error quality gates.

Only then does it atomically write SNAPSHOT_SUCCESS.json. This is an operational completeness and gross-quality check; model, prompt, or parser changes still require a labelled quality canary.

Validation

• 216 passed across MinerU extraction, Common Crawl download/read, snapshot verification, benchmark integration, Ray resource handling, and Dynamo configuration tests.
• Ruff lint and format checks pass on all affected Python files.
• uv lock --check, git diff --check, and shell syntax checks pass.

Checklist

• PR 22 functionality consolidated here.
• Built on the merged upstream Ray 2.57 changes from PR 2304.
• No custom async-scheduling implementation.
• WARC → MinerU extraction → deterministic Parquet; no intermediate HTML dataset.
• Native Curator Slurm sharding, checkpoint recovery, and snapshot-level verification.

[VibhuJawa]

Use VLLM utils to load the model

[VibhuJawa]

Benchmarking session: results, and what I removed after measuring it

Benchmarked this on 8× H100 (10k Common Crawl documents, the strided sample described below). Headline: 28.5 → 62.3 docs/s (2.2×), quality unchanged. But most of what I tried made things worse, and I've deleted all of it rather than leave knobs nobody should turn. Recording the negative results here so nobody re-runs these.

What shipped

Config	docs/s	extraction_rate
in-process baseline (8×1)	28.5	0.810
in-process + fp8 KV + pinned CPU workers	33.6	0.809
server backend (vllm serve --data-parallel-size 8)	62.3	0.810

status_counts are identical across all three (ok 9777 / too_long 150 / empty_input 40 / convert_error 33), so this is the same work done faster, not a quality trade.

What I added and then removed (all measured)

Removed	Result	Why it failed
gpu_fraction (2 engines/GPU)	−6%	Two engines contend for the same SMs on a compute-bound workload
... same, with explicit kv_cache_memory_bytes	−4%	Made packing start (a fraction can't: the 2nd engine sees the 1st's memory as used and gets ~0 KV), but not help
... same, with CUDA MPS spatial sharing	−4%	Published reports of >100% for prefill-heavy ≤3B models did not replicate — those are short-context; our token-weighted mean is ~22k
--data-parallel-size 16	won't start	AssertionError: DP adjusted local rank 9 is out of bounds — vLLM maps one replica per physical GPU
sort_by_budget (longest-answer-first)	−5%	Front-loading the largest documents raises peak KV pressure more than it recovers at the tail
--max-num-batched-tokens 8k→32k	flat
--max-num-seqs 256→512	flat
coarse partitions (8 instead of 32)	−23%	One partition per engine removes load balancing; engines that finish sit idle
structured_outputs="none"	+0.6%	Guided decoding is essentially free here — worth knowing, since it's the usual suspect

The pattern: every knob that reshuffles work inside one engine moved throughput by ≤18%. The only thing that mattered was removing the per-partition drain, which is what the server backend does.

Two things worth a second opinion

1. BENCHMARKS.md attributes the low FLOP utilisation to K=1024 GEMM shape. Its own table says 72.4% of prefill FLOPs are attention (which runs at ~75% of peak on FlashAttention-3), so the GEMM story caps out around 1.31× — it can't explain the observed gap. The evidence points at host-side/pipeline overhead instead: fp8 KV gave a GPU-stage speedup that didn't appear end-to-end, which is the signature of a pipeline bubble rather than slow kernels.

2. benchmarking/mineru-html-benchmark.yaml has absolute paths under my scratch dir. benchmarking/README.md documents the opposite convention (machine-specific paths: in a separate YAML merged with --config a --config b). Should be genericized before merge — happy to split it.

Separately: four bugs found in Curator itself

Not fixed here, but reproduced and worth filing:

1. InferenceServer (Ray Serve backend) advertises an endpoint that refuses connections. _host is hardcoded to "localhost" (core/serve/server.py:53) while serve.start() passes no host (ray_serve/backend.py:93), so Ray binds 127.0.0.1. Ray logged ready at http://127.0.0.1:8000/ and _wait_for_healthy() passed, yet requests failed with Connection refused from the driver and from actors one second later. Not IPv6 (getaddrinfo("localhost") returns only 127.0.0.1). Root cause still open, but a health check that passes against a dead endpoint is its own bug — it's what let a 100%-failed run look like a 2.2× speedup for an hour. Also: binding loopback breaks the multi-node deployments the docs describe.
2. Ray Serve 2.55.1 rejects vLLM ≥ 0.22 — ImportError: Neither vLLM nor SGLang is installed with 0.22.1 installed. Confirms chore(deps): Update to Ray 2.56.1 + Dynamo 1.3.0 + vLLM 0.22 (cu129) NVIDIA-NeMo/Curator#2064 must pair vLLM 0.22 with Ray 3.0 nightly; it's a coupled upgrade.
3. ray_data never sets max_concurrent_batches / max_tasks_in_flight_per_actor (Ray's defaults are 8/16). Unset means a GPU actor can't overlap work across partitions — the drain this PR works around. Observed, not measured — I did not verify that setting them helps.
4. Object store is sized ~0.9× node memory, landing exactly at the /dev/shm ceiling, so a stale byte (a loky semaphore leaked by a previous entry) makes cluster startup fail unrecoverably — retries can't clear a file. Worked around with object_store_size per entry.

🤖 Generated with Claude Code

[VibhuJawa]

Correction to the previous comment: the server backend is at parity, not 2.2×

My earlier comment reported 28.5 → 62.3 docs/s (2.2×). That comparison was not like-for-like and I'm withdrawing the claim. Pushed as ab0cce0b.

What the per-stage timings actually show

Normalising the inference stage by worker count (from tasks.pkl):

	inference stage total	workers	wall contribution
in-process (fp8 KV + pinned CPU)	624.7 s	8	78.1 s
server backend (DP=8)	1207.6 s	16	75.5 s

The inference work costs the same either way. The 137 s end-to-end difference is almost entirely vLLM engine startup, which the in-process path pays inside its measured window:

16:41:41  Starting MinerU-HTML
16:42:59  Setup on node complete      <- 78s of engine startup, inside the timer
16:46:38  execution finished in 219.08s
          e2e 297.6s  ->  33.6 docs/s

A persistent server pays that once, outside. Per document the server stage is in fact ~2× slower (120.8 ms vs 62.5 ms) from HTTP + serialization — it only keeps up because the latency is spread over twice as many workers.

This also fails to confirm the motivating hypothesis

I argued the per-partition drain (LLM.generate() blocking on each partition's slowest document) was costing real throughput, and that continuous submission would recover it. If that were the dominant cost, the server's inference wall would be shorter, not equal. It's equal. The drain is real and visible in the power trace, but it is not worth 2×.

Why the backend is still worth having

Operational, not throughput:

• engine startup is amortised across runs instead of repaid every run (~78 s/run here)
• engines can be scaled, restarted, and shared independently of the pipeline
• the pipeline itself needs no GPU allocation

For a single run over a large corpus, the in-process backend is simpler and no slower. That's now what the module docstring says.

What still stands from the previous comment

The list of measured-and-removed dead ends is unaffected — packing (−6%/−4%/−4% across fractional GPU, explicit KV budget, and CUDA MPS), DP=16 (local rank 9 out of bounds), sort_by_budget (−5%), coarse partitions (−23%), and the batching knobs (flat). So is the observation that guided decoding is nearly free (+0.6%), and the four Curator bugs.

The BENCHMARKS.md FLOP-attribution question also stands, and this correction sharpens it: if the pipeline and the engine agree on inference cost, the remaining gap to vllm bench serve's 26.1 docs/s per H100 is in the CPU stages (simplify 360.9 s, extract 513.5 s across 32 tasks) — not the GPU, and not the GEMM shape.

🤖 Generated with Claude Code

[VibhuJawa]

Superseding the two comments above: the PR is now server-only

The description has been rewritten to match the code. Both earlier comments are now partly stale — this note says what changed so nobody works from them.

What changed since those comments

a1e36e01 removes the in-process MinerUHtmlInferenceStage entirely. All three stages are CPU-only; the model is hosted in a vllm serve the pipeline never manages.

Validated by measurement, not assumption:

	docs/s	extraction
before (both backends present)	62.3, 60.2	0.8095–0.8096
after (server-only, −280 lines)	60.1, 60.1	0.8090–0.8096

Two reps of the new code agree to 0.0% and sit inside the pre-refactor band. Deleting the GPU stage changed nothing measurable.

What is now stale in the earlier comments

• The "2.2×" in the first comment — already retracted in the second, restated here for anyone reading top to bottom. The backends were at parity; the apparent gap was engine startup counted inside the in-process measurement window.
• The backend="server" / backend="in_process" selector — gone. There is one path, and base_url is required.
• In-process engine knobs (gpu_memory_utilization, kv_cache_dtype, quantization, vllm_init_kwargs on the stage) — gone. These are now vllm serve flags; BENCHMARKS.md carries the knob-to-flag mapping.

Still accurate from those comments

The dead-end table (packing −6%/−4%/−4%, DP=16 unstartable, ordering −5%, coarse partitions −23%, batching knobs flat, guided decoding +0.6%) and the four Curator bugs. Both are reproduced in the updated description.

One correction to the second comment, though: I attributed the residual gap to the CPU stages. The per-stage timings say otherwise — inference is still the largest single stage (75.5s wall vs 11.3s simplify and 16.0s extract), so that claim was wrong.

One further correction, on a test that was never valid

I reported that the ray distributed-executor backend silently disables async scheduling and that --distributed-executor-backend mp would recover it. Checking the actual server log: asynchronous scheduling is enabled: 17, disabled warnings: 0 — it was already enabled under ray. The warning I built that on came from the earlier Ray Serve run, which uses a different executor path. The mp-vs-ray comparison had no premise and its results should be ignored.

🤖 Generated with Claude Code

[VibhuJawa]

Please fix this