Benchmark Curator-managed Dynamo against a standalone vllm serve - #22
Closed
VibhuJawa wants to merge 18 commits into
Closed
Benchmark Curator-managed Dynamo against a standalone vllm serve#22VibhuJawa wants to merge 18 commits into
VibhuJawa wants to merge 18 commits into
Conversation
MinerU-HTML extracts the main content of a web page by asking a small language model to label every element of a simplified DOM as main or other. Its reference implementation runs all six of its steps in one process, so the GPU idles through the DOM work. This adds three Curator stages split along the CPU/GPU boundary -- simplify (CPU), vLLM inference (GPU), prune and render (CPU) -- plus a MinerUHtmlExtractor composite and a runnable tutorial pipeline. Measured on Common Crawl CC-MAIN-2025-26 (100k pages) on an L4; see tutorials/text/mineru-html-extraction/BENCHMARKS.md for method and full results. The split itself is the main structural win, and it grows with GPU speed: 1.17x on an L4, ~2.3x at 8x the GPU speed, because the ~80 ms of per-document CPU stops adding to GPU time and starts overlapping it. Also included, each measured: * extract_main_html rewritten to index _item_id in one pass instead of one XPath scan per label -- 4.5x faster, byte-identical on 282 real documents and on all 64 label assignments of a fixture document. * Per-document max_tokens instead of a flat 16k, which a 32k-context engine otherwise rejects every prompt over 16k against. Fallbacks drop from 17/300 to 7/300. * Pre-tokenization on the CPU workers, keeping ~8 ms/document out of the single vLLM process. * Documents whose simplified DOM has no _item_id skip the GPU (6.4% of requests). * The chat template is applied once. The reference implementation applies it twice, leaving a stray BOS/user pair inside the user turn; that changes the answer on ~20% of documents against a ~1% noise floor. chat_template_mode="upstream_double" reproduces the original. The largest engine-side knob is kv_cache_dtype="fp8" (1.57x, and it perturbs labels less than quantizing weights). It is not the default because Ada needs flashinfer-jit-cache installed first and a default that fails to start is worse than one that is slower. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
MinerUHtmlExtractor exposed simplify_workers and extract_workers but no way to size the GPU stage. Left unset, backends autoscale the actor pool from a single worker, and because each new worker cold-starts a vLLM engine the pool can fail to reach the available GPU count before a short run drains. Measured on 8x H100 with the ray_data executor, 10k Common Crawl documents over 32 partitions: the pool reached only ~4 actors in 509s, and GPUs 5-7 recorded 0.0% utilisation for the entire run. Allocating GPUs to Ray does not pin the pool; only num_workers does. Add inference_workers, forwarded to the inference stage in decompose() the same way the existing two knobs are. Also add benchmarking/scripts/mineru_html_benchmark.py, following the nightly driver contract. It counts documents from the written output rather than from task.num_items on the returned tasks -- the final stage is a ParquetWriter whose FileGroupTasks carry num_items == 1 per output file, so summing those counts files and under-reports throughput by the partition size. Extraction rate is gated at 200 characters and reported alongside the raw non-empty rate and the pipeline's own _mineru_status distribution, because Common Crawl carries sub-200-byte pages that yield a character or two of Markdown and would otherwise count as successful extractions. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
In-process `LLM.generate()` is called once per partition and blocks until that
partition's slowest document finishes. Answer lengths span ~37x (mean 149, max
5517 tokens), so the engine drains to a batch of ~1 at every partition boundary
-- visible as GPU power oscillating 700W -> 150W.
Add `MinerUHtmlExtractor(backend="server", base_url=...)`, which swaps the GPU
stage for a CPU-only stage submitting to an OpenAI-compatible endpoint. A
persistent endpoint has no partition boundaries: every worker submits into one
continuously-batched queue, so a straggler overlaps with fresh work.
Measured on 8x H100, 10k Common Crawl documents, via benchmarking/run.py:
in-process baseline 28.5 docs/s extraction 0.810
in-process + fp8 KV + pinned CPU 33.6 docs/s extraction 0.809
server backend (vllm serve DP=8) 62.3 docs/s extraction 0.810
Quality is unchanged -- status counts are identical across all three
(ok 9777 / too_long 150 / empty_input 40 / convert_error 33).
For reference, `vllm bench serve` on real MinerU prompts sustains 26.1 docs/s on
a single H100, so even at 62.3 the pipeline extracts well under half of what 8
GPUs can do. The remaining gap is the CPU stages, not the engine.
Notes on the implementation:
* The stage uses the `openai` SDK, which is a core dependency, and which owns
retry/backoff. Token-id prompts go through `/v1/completions` unchanged, so
CPU-side pre-tokenization is preserved and the server does not re-tokenize.
* Sampling is pinned per request. The checkpoint ships a generation_config.json
(temperature 0.7, top_k 20, repetition_penalty 1.05) that an OpenAI server
applies as request defaults, while the in-process path never sees it because
it builds SamplingParams directly. repetition_penalty is actively harmful
here: the answer is deliberately repetitive ("1main2other3main..."), and
serving with the checkpoint defaults measured extraction_rate 0.015 vs 0.809.
* A batch where every request fails now raises instead of returning throughput.
Before that guard, a run where all 9583 requests failed reported a 2.2x
"speedup" -- it had silently stopped doing inference and fallen back.
* `compact_answer_regex()` is shared by both backends so they cannot drift into
constraining output differently.
Also fixes a latent bug in the in-process path: `dtype`, `trust_remote_code`,
`enforce_eager` and `limit_mm_per_prompt` were passed to `create_vllm_llm` both
explicitly and via `**engine_kwargs`, so setting any of them in
`vllm_init_kwargs` raised "got multiple values for keyword argument".
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
The previous commit reported 62.3 vs 33.6 docs/s as a throughput win. That
comparison was not like-for-like and the claim is withdrawn.
Per-stage timings from tasks.pkl show the inference work costs the same either
way once normalised by worker count:
in-process 624.7s across 8 workers -> 78.1s wall
server 1207.6s across 16 workers -> 75.5s wall
The 137s end-to-end difference is almost entirely vLLM engine startup, which the
in-process path pays inside its measured window (16:41:41 start -> 16:42:59
"Setup on node complete" = 78s, plus its ramp) and a persistent server pays once,
outside. Per document the server stage is actually ~2x slower -- 120.8ms vs
62.5ms -- from HTTP and serialization; it keeps up only because that latency is
spread over twice as many workers.
This also fails to confirm the motivating hypothesis. The claim was that
in-process LLM.generate() drains the engine at every partition boundary and that
continuous submission recovers the loss. If that were the dominant cost the
server's inference wall would be shorter, not equal. The drain is real but it is
not worth 2x.
The backend is still worth having, for reasons that are operational rather than
throughput: engine startup is amortised across runs instead of repaid each run,
engines scale/restart/shard independently of the pipeline, and the pipeline needs
no GPU. Docstring rewritten to say exactly that, with the numbers.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
Every stage is now CPU-only. The model is hosted in a `vllm serve` the pipeline
never starts or stops, and the stages talk to its OpenAI-compatible endpoint:
simplify (CPU) -> HTTP -> extract (CPU)
Removes `MinerUHtmlInferenceStage` and everything that existed only for it:
`create_vllm_llm`, `gpu_memory_utilization`, `kv_cache_dtype`, `quantization`,
`vllm_init_kwargs`, and the `backend` selector (with one backend there is nothing
to select, so `base_url` is simply required).
Measured on 8x H100 over the same 10k Common Crawl sample, via benchmarking/run.py:
before (both backends present) 62.3, 60.2 docs/s extraction 0.8095-0.8096
after (server-only) 60.1, 60.1 docs/s extraction 0.8090-0.8096
Two reps of the new code agree to 0.0%, and both sit inside the band of the old.
Removing 280 lines and the entire GPU stage changed nothing measurable.
Why the in-process path went rather than staying as an option: it was never
faster. The earlier 33.6 vs 62.3 comparison was not like-for-like -- normalised
by worker count the inference work costs the same either way (78.1s vs 75.5s of
wall time), and the end-to-end gap was vLLM engine startup, which the in-process
path pays inside its measured window and a persistent server pays once, outside.
Keeping two code paths at identical throughput was not worth the surface area.
Consequences worth noting:
* The pipeline no longer imports vLLM at all -- verified by loading the composite,
running setup() on all three stages and constructing the client, then checking
sys.modules. `mineru_html` does not pull it transitively either (vLLM is only in
its [vllm]/[all] extras). So the install line drops from
`pip install "nemo_curator[vllm]" mineru_html` to `pip install nemo_curator
mineru_html`; vLLM is now needed only to host the server.
* A Curator job needs no GPU allocation: `ray: num_gpus: 0` in every entry.
* BENCHMARKS.md keeps every engine measurement, reframed as `vllm serve` flags
with a knob-to-flag mapping table, plus a section recording why the in-process
stage was removed.
`create_vllm_llm` stays in `nemo_curator/utils/vllm_utils.py` -- Nemotron-Parse
still uses it.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Signed-off-by: Vibhu Jawa <vjawa@nvidia.com>
A 29-run sweep on 8x H100 against a persistent DP=8 vLLM server took the 40k
benchmark from 98.5 to 114.4 docs/s at unchanged extraction quality (0.803).
None of the three changes that mattered is a `vllm serve` flag.
1. ~156 rows per input shard (+8.5%, replicated at 20k as +6.7%).
FilePartitioningStage groups whole files and never splits one, so shard
count sets partition count, and the drain tail is bounded by the LARGEST
partition. It is an optimum, not a trend: halving again to ~79 rows gives
3.5% back, because dividing 4 source shards 507 ways leaves a 3.04x row
skew and the largest partition grows even as the mean falls.
2. 32 total CPU actors rather than 64 (+2.6%). Every Ray Data actor adds
GPU-idle startup and holds its CPU slot for the whole run, and both CPU
stages are ~5x over-provisioned by default -- simplify alone measures 578
docs/s against a pipeline running at ~114.
3. USE_TORCH=0 HF_HUB_OFFLINE=1 TRANSFORMERS_OFFLINE=1 (+3.5%, no code
change). `import mineru_html` eagerly pulls transformers -> torch +
sklearn, ~26s per actor, for a pipeline whose inference is a remote HTTP
call and which runs no local torch model.
The unifying result is that the engine was never the constraint. GPU-busy time
is 290-297s in EVERY 40k run regardless of configuration; all the variation is
in the 60-85s of GPU-idle ramp and tail around it, and the ramp is mostly Python
imports. Five `vllm serve` flags, queue depth, and inference workers past 32 all
measured flat or negative.
That fixed overhead also makes throughput strongly scale-dependent -- the same
config reports ~60 docs/s at 10k, ~87 at 20k and ~114 at 40k -- so the benchmark
entry now runs 40k documents and BENCHMARKS.md warns against comparing docs/s
across corpus sizes. Repeat runs of one config differ by ~1.3% at 20k.
Also corrected here: "--data-parallel-size cannot exceed the GPU count" is true
only per node; going wider is supported via --data-parallel-size-local plus a
--headless second node, though for a dense model two independent -dp 8 servers
are throughput-equivalent and simpler. And --inference-workers is silently
capped at (cores - actors) / 2 because the stage declares cpus=2.0, so values
above that were never actually tested.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…coding
Three strands: correctness fixes from a review pass, a simplification pass, and
the largest throughput win found in this study.
CORRECTNESS. The worst of these is that a PARTIAL inference failure was
invisible. The existing guard only fired when every request in a batch failed;
when some failed, the rows kept status "ok", the extract stage parsed the empty
response into an empty label map, pruned the whole document and emitted "\n" --
blank output that never reached the fallback and still scored as a success in
both status_ok_rate and nonempty_rate. One server replica restarting mid-run was
enough to lose thousands of documents silently. Failed rows are now marked
"inference_error" so they fall back. This is the same shape as the bug that once
made a 100% HTTP failure read as a 2.2x speedup, one notch down.
Also fixed:
- resp.choices[0] moved inside the try: a malformed response raised IndexError
out of gather() and killed the whole partition, where every other failure only
degrades one row.
- _as_text now uses pd.isna: bool(pd.NA) raises "boolean value of NA is
ambiguous" and escaped process(), taking every good row in the batch with it.
Both entry points read with dtype_backend="numpy_nullable", which produces
pd.NA for a null in a string column.
- The simplify stage no longer calls the tokenizer with an empty batch, which
raises IndexError. pretokenize=True is the default, so an empty partition
killed a real pipeline while the tests, mostly pretokenize=False, stayed green.
- drop_html_field is now `fallback == "empty"`. It was `!= "trafilatura"`, so
"bypass" lost the raw HTML it returns and was silently identical to "empty".
- URLs are normalised to None. np.nan and pd.NA both raise inside the converter,
and _render has no fallback path, so one null URL turned a good document into
an empty convert_error.
- The pretokenize=False length estimate used len//2 and claimed to be
"conservative so nothing that fits is dropped". Measured 3.8 chars/token, so it
overestimated ~1.9x and did the opposite -- documents well inside max_model_len
were diverted to the fallback.
SIMPLIFICATION. The extract stage no longer imports mineru_html at all. Its
package __init__ pulls the transformers and vLLM inference backends -- ~25s of
startup per actor, all GPU-idle ramp -- for code a CPU-only stage never runs. It
needs one thin trafilatura wrapper, and trafilatura is already a declared
dependency while mineru_html is not. Guarded by a parity test against the
upstream handlers and a subprocess test asserting mineru_html stays out of
sys.modules. The AsyncOpenAI client and its event loop are now owned per worker
rather than rebuilt per batch: asyncio.run closes its loop, which invalidated the
client, so every batch was discarding the keep-alive pool and paying up to
max_concurrency TCP handshakes before its first request. STATUS_FIELD is now
exported -- the benchmark had re-declared it as a literal, so renaming the column
would have silently reported status_ok_rate 0.0 forever. And an inert skipif in
test_mineru_utils.py was skipping 15 lxml-only tests whenever mineru_html was
absent.
SPECULATIVE DECODING, the largest single win measured here: 125.6 -> 160.5 docs/s
over 100k documents (+27.8%) at identical extraction. It is purely a `vllm serve`
flag; the pipeline is untouched. Decode on this model is memory-bound by ~30x --
it keeps 48 KiB of KV per token, ~75% of Llama-3.1-8B's footprint at 1/16 the
FLOPs -- so a drafted token costs ~2.4us against the ~147us KV re-read it avoids,
and the answer format ("1main2other3main...") is predictable enough that suffix
decoding accepts most of what it drafts. Measured 9.19 tokens per forward pass at
16 draft tokens. Output is unchanged: every drafted token is verified against the
target model, and the status histogram over 100k documents matches the control to
within one document.
Worth recording why five earlier `vllm serve` flags all measured flat: they were
aimed at prefill. By GPU time this workload is ~43% prefill and ~57% decode
despite being 26.9:1 prefill by token count.
125 tests (110 before), five of them covering failure modes that had no coverage
at all -- nothing previously exercised a client that returned or raised.
Re-verified end to end after the changes: 160.2 docs/s at extraction 0.8075
against 160.5 / 0.8076 before, inside the noise floor.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Four review comments on PR #21, all on mineru_html.py. "We should always use the real path, remove this fallback" and "I dont think we need to support both paths" -- `pretokenize` is gone and prompts are always tokenized on the simplify workers. That removes the whole second path: the PROMPT_FIELD column, the TOKENS_FIELD-or-PROMPT_FIELD sniff in the inference stage, the `--no-pretokenize` tutorial flag, and the character-count length estimate the over-long pre-filter used when not pre-tokenizing. The filter now counts real token ids, which is also strictly more accurate -- the estimate it replaces overshot ~1.9x and diverted documents that would have fit. "Can we come up with a better `_as_text`" -- now `decode_html_cell` in mineru_utils, with a two-line docstring. The pd.NA rationale is one line at the branch rather than a five-line essay above the function. "Should this be in utils" -- the three fallback handlers are now `FallbackExtractor` in mineru_utils, alongside `decode_html_cell` which it uses. The extract stage's setup() is a single line again. Tests updated rather than deleted: the chat-template assertions used the text prompt column, so they now exercise `_chat_wrap` directly, which is the thing they were actually testing. A new test asserts the emitted token ids are exactly the tokenization of the chat-wrapped prompt, which is the property the old text-vs-tokens comparison was checking across the two paths. 125 tests pass, ruff and format clean, both CLIs still parse every flag the benchmark config passes. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Review: "Can we then organize our files better to prevent circular imports".
The cycle was mineru_server importing the shared column names and prompt helpers
from mineru_html, while mineru_html's decompose() needs the server stage -- which
forced a function-scope import with a comment explaining why it had to be there.
Those shared names were never really stage code. They now live in mineru_utils,
which imports neither stage module, so the package is a DAG:
mineru_utils <- column names, compact_answer_regex/response_budget,
decode_html_cell, FallbackExtractor, the DOM helpers
mineru_server <- imports mineru_utils
mineru_html <- imports mineru_utils and mineru_server, normally, at
module scope
Moved: DEFAULT_MODEL, TOKENS/MAP_HTML/N_ITEMS/RESPONSE/STATUS_FIELD,
INTERNAL_FIELDS (was _INTERNAL_FIELDS -- no longer private to one module),
compact_answer_regex, compact_response_budget. mineru_utils' docstring now says
what it is: the foundation the stage modules are built on, not a grab bag.
Verified the cycle is actually gone by importing each stage module first in a
fresh interpreter -- a real cycle breaks one of the two orders. 125 tests pass,
ruff and format clean, both CLIs still parse.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…n CI The feature's two third-party packages appeared nowhere in pyproject.toml or uv.lock; the requirement was communicated only by an ImportError string. Two consequences: `pip install nemo-curator[all]` could not produce a working pipeline, and every CI job skipped the tests that cover the stages. ONE EXTRA, NOT TWO. `webpage_converter` is not its own distribution -- it is the import name of mineru-webkit, which mineru-html already depends on. So the two imports cannot be separated by packaging: installing mineru-html necessarily installs webpage_converter, and a second extra could only serve a webkit-without-mineru-html install that no stage combination asks for. Both are still declared explicitly, along with lxml and trafilatura, because Curator imports all four directly rather than reaching through mineru-html. trafilatura is restated at ==2.0.0. mineru-html declares it unpinned, so without the restatement `pip install nemo-curator[mineru_html]` could drift off the version text_cpu pins -- verified it holds under pip's own resolver, which does not read uv's constraint-dependencies. KEPT OFF text_cpu. On its first conversion webpage_converter imports cairosvg, which dlopens libcairo.so.2. Folding these into text_cpu would attach a system-library requirement to every `pip install nemo-curator[text_cpu]`. For the same reason the stages-text CI job now installs libcairo2 explicitly rather than betting on it arriving transitively through the runner image, where it is not a top-level apt package. CI. install-test.yml only validates install + `import nemo_curator`; it never runs pytest. The job that actually executes tests/stages/text/html_extraction is cicd-cpu-tests -> tests/L0_Unit_Test_CPU.sh, so the extra is wired into both: install-test.yml for install validation across the pip and uv legs, and L0_Unit_Test_CPU.sh so the tests stop skipping. 15 of 125 tests ran before, all of them in test_mineru_utils.py; 125 run now. Measured by hiding both packages in a real venv: 15 passed / 65 skipped becomes 125 passed. All 46 tests in test_mineru_html.py were among the zero. The lockfile refresh is purely additive -- 11 packages added, no existing version perturbed. Verified `uv sync --locked` for the stages-text extra combination, for `--extra mineru_html` alone, and for `--extra all`, plus the pip leg resolving independently at 167 packages. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…retarget the tutorial
Four review agents (reuse, simplification, efficiency, altitude) over the feature
diff. The two findings worth the whole exercise were both measured, and neither
was visible in any metric this PR reports.
**The OpenAI SDK was type-walking every token id.** `completions.create(prompt=
[~5000 ints])` runs the body through the SDK's param transform, which recurses
per list element -- profiled at 2.1M calls for 188 requests. Measured on 200 real
documents: 48.7 ms of CPU per request against 2.0 ms for `client.post(
"/completions", cast_to=Completion, body={...})`, the SDK's documented raw
route, which keeps retries, timeout and the keep-alive pool. That was ~37% of the
pipeline's entire CPU budget. Worse, the transform never awaits, so it ran on the
inference worker's event loop and capped one worker near 20 requests/s no matter
what max_concurrency said (measured cpu/wall 0.96 -- saturated, not waiting).
That is very likely why 16-32 inference workers were needed to feed the server.
**Non-UTF-8 pages were being destroyed.** decode_html_cell did
`decode("utf-8", errors="replace")`, so every windows-1252, shift_jis or gb2312
page became replacement characters before the simplifier saw it -- the model then
labelled garbage while the row kept status "ok" and still cleared the 200-char
extraction_rate gate. Curator already has `decode_html`, which falls back to
charset detection. Verified at realistic page size: 3 of 4 encodings now recover
exactly and the fourth lands on a neighbouring codepage (readable, wrong
accents), against 4 of 4 destroyed before.
Also fixed:
- `cpus` 2.0 -> 1.0 on the inference stage. Ray Data charges the reservation
against the cluster total for the pool's whole life, so it silently capped
concurrent workers at (cores - actors)/2; asking for 80 ran 48.
- `openai` (~2.7s to import) hoisted into a real `setup()`; the stage had none,
so every worker paid it inside its first batch while the server sat idle.
- inputs()/outputs() now declare what the stages actually read and write.
validate_input IS enforced (backends/base.py:104 -> process_batch), so the
extract stage declaring html_field turns a silent degradation to
fallback="empty" into a loud error.
- Status values are now a `Status` literal in mineru_utils next to STATUS_FIELD,
which is exported "so callers can read it back" but never enumerated what.
- mineru_utils' docstring had the dependency arrow backwards.
- Dead `_trafilatura` attrs, the `PromptT` str arm, and an orphaned comment --
all leftovers from removing the in-process backend and `pretokenize`.
- README claimed `bypass` drops the raw HTML column; it needs it.
Tutorial retargeted at the user's ask: it is now about the model, not the
benchmarks. It opens with a worked example -- a real page, its `_item_id`
annotation, the `<answer>1main2main3main4other</answer>` the model returns, and
the Markdown that falls out -- all generated from the actual pipeline rather than
invented. Server command ships the best configuration including speculative
decoding. Worker defaults are the measured-best ratios sized from
`len(os.sched_getaffinity(0))`, not `os.cpu_count()`, which reports the machine
rather than the cgroup and would size a SLURM job straight into the actor-pool
hang. BENCHMARKS.md moved to benchmarking/ where the config that produced it
lives.
Skipped, with reasons: deduplicating num_documents_processed (30+ recorded runs
and the results viewer read that key); wiring the fallback through Curator's
HTMLExtractorAlgorithm (changes extraction output); a shared AsyncProcessingStage
(right call, but it means migrating three other stages).
127 tests, ruff and format clean.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Five references, each with a line on why it matters for this pipeline rather than a bare citation list. arXiv ids and the ACL entry were verified against the listings rather than written from memory, and every link was checked to resolve. - MinerU-HTML / Dripper (arXiv 2511.23119), the extractor itself. Its WebMainBench result -- ROUGE-N F1 0.84 against Trafilatura's 0.64 over 7,887 annotated pages -- is the number that justifies spending a GPU on extraction, and equally why Trafilatura is a sensible fallback rather than the primary path. - Trafilatura (Barbaresi, ACL 2021), which is what every unlabelled document falls back to. - RefinedWeb (arXiv 2306.01116), for why extraction quality is worth the trouble: boilerplate that survives extraction is boilerplate you train on. - PagedAttention / vLLM (arXiv 2309.06180), for why the KV cache dominates here. - Speculative decoding (arXiv 2211.17192) for the draft-then-verify idea and its proof that the output distribution is unchanged -- which is why --speculative-config costs nothing in quality -- and SuffixDecoding (arXiv 2411.04975) for the model-free variant this pipeline actually uses, whose suffix tree suits answers as repetitive as "1main2other3main...". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…swer -> Markdown The "How it works" section started at the simplified DOM, which skipped the part a reader most needs: what the model is actually asked, and how its answer turns into output. It now walks all five stages with the genuine artifact at each step. It also fixes an error. The previous example showed `_mineru_map_html` -- the annotated ORIGINAL that the extract stage prunes -- and labelled it "what the model sees". The model is shown a far more aggressively simplified page: in this example the <nav> block is gone entirely, while the <footer> survives to be judged. That contrast is now the point of step 2, because it is the division of labour between the two halves of the system: the simplifier removes only what is unambiguous and leaves every judgement call to the model. Added: the verbatim instruction prompt (it was nowhere in the docs, so the "Guidelines" defining main vs other were invisible), the token count for the toy page against a real one, and a table reading the answer back element by element so `1main2main3main4other` means something on first encounter. Every artifact is generated by running the pipeline, and a check confirms the README's token count and Markdown block still match what the code produces. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The tutorial's only worked example was a prose recipe, which never exercises the maths path -- so the README asserted that `mm_md` preserves formulae without ever showing it. This adds a second example: section 3.4 of arXiv's KAN paper (a Poisson problem and a PINN loss), left inside the real arXiv page so the input carries genuine banner/header/licence/footer chrome. 43 KB, 36 numbered elements, a 4,752-token prompt, 23 <math> elements. Both outputs are from a live run against the model, not written by hand. The result is stark -- every formula survives as LaTeX through MinerU-HTML, and trafilatura emits none of the 23, leaving "We consider the data for which is the true solution", a sentence that parses and says nothing. The README says explicitly what that does and does not show. Both extractors selected the same article and dropped the same boilerplate, so this is NOT a labelling win: it is architectural. MinerU-HTML deletes nodes from the original DOM, so MathML and its alttext reach the renderer intact, while trafilatura rebuilds a clean tree that has nowhere to put a formula -- its own Markdown mode drops them too, so it is not an artefact of how the pipeline calls it. It would have been easy to present this as the paper's equation-category result; it isn't that, and claiming so would have been wrong. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
It was a 77-line section inside a tutorial README that had just been rewritten to be quick to read. The example earns its place in the repo but not in the main flow: a reader working out how to run the pipeline does not need two full before/after Markdown dumps in the middle of it. README keeps a four-line pointer with the number that matters (23 equations in, 23 out against 0) and a link. The standalone doc gains the framing it needs to stand alone -- what page, what the model did, what the fallback did, and a section on what the comparison does and does not demonstrate. README is back to 273 lines from 343. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The document showed the pipeline's Markdown as source, which is what it is, but
meant a reader saw `\pi^{2}` rather than the formula. GitHub renders LaTeX via
MathJax, so the same content is now also shown rendered, as a blockquote after
the raw output. The raw block is untouched -- it is the artifact.
Two differences between the two, both GitHub's constraints rather than the
pipeline's, and both stated in the text rather than quietly applied:
- The PDE is set as an `aligned` block. The pipeline emits it as an HTML <table>
because arXiv marks equation groups up that way, and GitHub does not process
`$...$` inside HTML tables.
- `\coloneqq` is written `:=`. That macro lives in MathJax's mathtools
extension, which GitHub does not load, so the raw output pasted into a GitHub
file unchanged renders everything except that one symbol. That is worth
knowing for anyone publishing extracted maths to GitHub, so it is called out.
Verified `\coloneqq` now appears only inside the raw ```markdown fence and in
prose backticks, never in a live math block.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…d/__init__ The charset fix in the /simplify pass reached for Curator's `decode_html` in `stages/text/download/utils`. Importing that executes the download package's `__init__`, which imports the whole subpackage -- and that needs `pycld2`, which is declared in the `text_cpu` extra, not `mineru_html`. So `pip install nemo_curator[mineru_html]` raised ModuleNotFoundError on the first bytes-valued page, which is the Common Crawl path. I traded a silent quality bug for a hard crash in a narrower install and did not notice, because the venv I tested in happens to have text_cpu too. It surfaced only when the pipeline ran in a different environment. Now calls `charset_normalizer` directly -- which is what `decode_html` does underneath -- so the behaviour is unchanged and the package `__init__` is never executed. That also drops the ~1.2s per-worker import the reuse review measured. `charset-normalizer` is declared in the extra rather than leaned on transitively. Verified in the environment that exposed the bug: windows-1252 recovers, UTF-8 is untouched, undecodable bytes and pd.NA both give "", and `download` never appears in sys.modules. 127 tests, ruff clean. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Adds a benchmark entry and script that let Curator own the engines -- an InferenceServer with a DynamoVLLMModelConfig brings up num_replicas vLLM engines as Ray actors in the same cluster the pipeline runs on -- instead of requiring an endpoint you start and tear down yourself. Measured on 100k Common Crawl documents, both paths with fp8 KV and suffix speculative decoding at 16 draft tokens, both through benchmarking/run.py: Dynamo, M=32 x B=256 169.5, 167.4 docs/s extraction 0.8089 standalone, M=32 x B=48 160.5, 160.2 docs/s extraction 0.8076 +4.9% for Dynamo. The two ran at different queue depths, so the obvious objection is that standalone was under-fed; it was not. Re-running standalone at Dynamo's depth on one node against one server gives 158.1 at B=256 against 158.2 at B=48 -- a 0.06% difference. Standalone does not benefit from a deeper client queue, Dynamo needs one, and node-to-node variation (~1.5%) is well below the 4.3% gap between Dynamo's worst run and standalone's best. Request-level timing locates the difference: the CPU stages are identical across all four runs, and standalone's inference actors are idle for 0s in both runs while Dynamo's idle for 29-59 minutes of actor time. Standalone is serving-limited; Dynamo has headroom. Depends on NVIDIA-NeMo#2064. The actor venv does not resolve on main without it -- that PR adds the CUDA 12.9 wheel index, the ai-dynamo version pin, and the nixl-cu13 exclusion. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
VibhuJawa
force-pushed
the
mineru-dynamo-benchmark
branch
from
August 4, 2026 21:00
e01ce28 to
63badce
Compare
Owner
Author
|
Superseded by PR #21. The managed Dynamo benchmark/serving work has been consolidated into PR #21 as one snapshot-ready commit, rebased on NVIDIA-NeMo#2304 and validated end to end on Slurm. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Stacked on #21 — it needs the MinerU-HTML stages that PR introduces.
Also depends on NVIDIA-NeMo/Curator#2064. The Dynamo actor venv does not resolve on
mainwithout it: that PR adds the CUDA 12.9 wheel index, theai-dynamoversion pin, and thenixl-cu13exclusion. Everything 2064 provides is deliberately left out of this branch rather than vendored — I verified my working copies ofcore/serve/dynamo/vllm.pyandcore/serve/ray_serve/backend.pywere byte-identical to 2064's and dropped them.What this adds
Curator can own the engines instead of requiring an endpoint you start yourself: an
InferenceServerwith aDynamoVLLMModelConfigbrings upnum_replicasvLLM engines as Ray actors in the same cluster the pipeline runs on. One new benchmark script and one entry.Result
100k Common Crawl documents, both paths with fp8 KV and suffix speculative decoding at 16 draft tokens, both through
benchmarking/run.py:M=32 × B=256M=32 × B=48+4.9% for Dynamo, replicated on both sides, non-overlapping ranges.
The obvious objection, tested
The two ran at different queue depths, so standalone may just have been under-fed. It wasn't — re-running standalone at Dynamo's depth, on one node against one server:
B=48B=256A 0.06% difference. Standalone doesn't benefit from a deeper client queue; Dynamo needs one (~1024 requests/replica against 192) because its frontend adds per-request latency. Node-to-node variation is ~1.5%, below the 4.3% gap between Dynamo's worst run and standalone's best.
Where the difference actually is
Request-level timing from
tasks.pkl. The CPU stages are identical across all four runs (simplify 37.9–40.2 ms/doc, extract 65.7–72.5), so it's all inference:The idle column is the load-bearing one. Standalone's inference actors were never idle — the server was the constraint. Dynamo's idled for 29–59 minutes of actor time, i.e. its serving layer outran the CPU stages feeding it. Don't read the ms/doc gap as pure serving efficiency:
process_timeis actor wall-clock with B requests in flight, so a larger B mechanically compresses actor-seconds per document.What it costs
structured_outputsunavailable — Dynamo's frontend rejects the extra bodyvllm serveaccepts (400 Validation: Unsupported parameter(s)). Measured at 40k this costs nothing (extraction 0.8043–0.8047 unconstrained vs 0.8026–0.8032 constrained), but it's a real capability gap.etcdandnats-servermust be on PATH — static Go binaries the control plane shells out to, not Python packages.arctic-inferencein the actor venv, unpinned. vLLM's error message recommends==0.1.1, and following that fails: every release is sdist-only so it always builds from source, and 0.1.1's build needstorch==2.7.0, unsatisfiable against the torch vLLM pulls in — which fails the entireruntime_envsetup, not just that package.Not included
This does not replace the standalone tutorial or benchmark. Both paths remain; the entries sit side by side so the comparison stays reproducible. Replacing the standalone path is a separate decision that should wait until 2064 lands, since until then a Dynamo-only tutorial wouldn't work for anyone on
main.🤖 Generated with Claude Code