perf: use PyArrow fast path for JSONL reads

[praateekmahajan]

What changed

This PR makes direct PyArrow parsing the default for JSONL files and keeps the result as a pa.Table until a stage actually needs pandas.

Reader-side _curator_dedup_id generation and assignment now work with both Arrow tables and pandas DataFrames. As a result, LanceReader now supports _generate_ids and _assign_ids without converting its Arrow output to pandas first.

When pandas is needed, DocumentBatch.to_pandas() handles the conversion and preserves Arrow-backed string columns.

Choosing an engine

JSONL reads use direct PyArrow parsing by default. Callers can select engine="pandas" when they need pandas-specific parsing or type inference.

The two engines do not always infer identical types. For example, pandas may interpret an ISO created_at value as a timezone-aware datetime while PyArrow retains it as a string. Mixed-type columns are another situation where the pandas engine may be the better fit.

The PyArrow path continues to support remote files through fsspec. It normally reads with an 8 MiB block. If PyArrow reports that a single JSONL record straddles the block boundary, the reader retries with progressively larger blocks so that it can read the complete row. The 256 MiB maximum is a safety ceiling for unusually large individual records, such as rows containing a base64-encoded image or PDF; it is not the default amount read for every row.

Lance and reader IDs

Because ID handling lives in BaseReader, Arrow-backed Lance reads now support _generate_ids and _assign_ids too.

Lance tasks use a stable identity based on the dataset path, version, and fragment IDs. File readers retain their existing path-based registry keys for backward compatibility; NMCUR-315 tracks moving them to deterministic task IDs as well.

Compatibility

This work is based directly on main and does not depend on the RAPIDS/Pandas 3 upgrade in #2303. It is tested with the current main versions: pandas 2.3.3 and PyArrow 19.0.1.

Testing

The tests cover:

• Direct PyArrow and explicit pandas JSONL reads
• ID generation and assignment for Arrow and pandas
• Large JSONL records and remote fsspec inputs
• Expected pandas/PyArrow dtype differences
• DocumentBatch Arrow-string conversion
• Lance ID generation and assignment
• Related reader, writer, and dedup-removal workflows

Benchmarks

The table compares the latest completed main nightly (nightly-2026_08_20__06_39_17_UTC) with this PR (pr-2325-2026_08_21__00_25_05_UTC-519ad6a9). Reader time is the mean jsonl_reader_process_time per task. Lower is better.

Benchmark	Mean JSONL reader time, nightly → PR	E2E runtime change
domain_classification_raydata	1.244s → 0.324s (74.0% faster)	0.8% slower
domain_classification_xenna	1.357s → 0.416s (69.4% faster)	7.5% faster
embedding_generation_raydata	0.183s → 0.048s (73.7% faster)	0.7% faster
embedding_generation_xenna	0.212s → 0.054s (74.4% faster)	0.2% faster
dedup_removal_raydata	49.02s → 9.05s (81.5% faster)	46.9% faster
dedup_removal_xenna	33.62s → 9.83s (70.8% faster)	68.0% faster
score_filter_raydata	0.333s → 0.079s (76.3% faster)	3.8% faster
score_filter_xenna	0.303s → 0.069s (77.3% faster)	2.0% slower
fasttext_filter_raydata	0.308s → 0.073s (76.4% faster)	0.8% faster
fasttext_filter_xenna	0.372s → 0.084s (77.3% faster)	5.8% faster
modifier_raydata	0.358s → 0.085s (76.1% faster)	3.2% faster
modifier_xenna	0.341s → 0.082s (75.9% faster)	4.7% faster
ndd_dynamo_dp8	0.175s → 0.046s (73.8% faster)	7.0% slower
ndd_ray_serve_dp8	0.168s → 0.046s (72.8% faster)	<0.1% faster

Across all 14 entries, the median JSONL reader task is 75.2% faster and median E2E runtime improves by 2.0%. Looking only at the 12 entries where JSONL reading is not the dominant cost, median E2E runtime improves by 0.8%.

[copy-pr-bot]

Auto-sync is disabled for draft pull requests in this repository. Workflows must be run manually.

Contributors can view more details about this message here.

[github-actions]

🌿 Preview your docs: https://nvidia-preview-codex-pr2303-jsonl-arrow.docs.buildwithfern.com/nemo/curator

Here are the markdown pages you've updated:

• nemo/curator/curate-text/process-data/deduplication/exact
• nemo/curator/curate-text/process-data/deduplication/fuzzy

[praateekmahajan]

/ok to test

[praateekmahajan]

@claude review