[codex] Add high-throughput Lance image reader

[VibhuJawa]

Summary

• add a public-API Lance 7 image reader that pins the dataset version and scans contiguous fragment-local slices
• interleave 100-row slices across 40 fragments, project only image by default, and execute 32 scans concurrently in one long-lived Ray actor
• retain a Lance Session with a 4 GiB index-cache limit and expose version-local fragment/row-offset work keys without reading extra columns
• set the measured 32-CPU defaults (LANCE_CPU_THREADS=16, LANCE_IO_THREADS=64) while keeping each setting configurable
• document the production pattern and require pylance>=7.0.0

Benchmark basis

On the production object-store benchmark, 40 fragments x 100 contiguous rows returned 4,000 encoded images at 225.7 images/s and 52.3 MiB/s with the public fragment scanner. The comparable public fragment.take path reached 229.5 images/s. The scanner path is used here because it is public, supported, and avoids URL B-tree fan-out for bulk reads.

The reader deliberately does not compact or redistribute data. Coordinates are explicitly pinned to a dataset version.

Test plan

• uv run ruff check nemo_curator/stages/image/io/lance_reader.py nemo_curator/stages/image/io/__init__.py tests/stages/image/io/test_lance_reader.py
• uv run --extra image_lance pytest -q -m "not gpu" tests/stages/image/io tests/stages/text/io/reader/test_lance.py
  • 32 passed

Stack

This PR is intentionally based on feat/mint-image-lance-workflow (draft PR #7) so its diff contains only the optimized read path.