Skip to content

[codex] Add indexed Lance column fetch for interleaved batches - #8

Draft
VibhuJawa wants to merge 2 commits into
feat/lance-annotation-writerfrom
feat/interleaved-lance-materialization
Draft

[codex] Add indexed Lance column fetch for interleaved batches#8
VibhuJawa wants to merge 2 commits into
feat/lance-annotation-writerfrom
feat/interleaved-lance-materialization

Conversation

@VibhuJawa

@VibhuJawa VibhuJawa commented Jul 3, 2026

Copy link
Copy Markdown
Owner

What changed

  • add LanceDatasetConfig, LanceIndexCacheConfig, and the public LanceColumnFetchStage
  • support presence-only exact-key lookup and arbitrary typed Lance column projections
  • preserve input ordering while handling duplicate input keys, missing keys, and explicit destination-column policies
  • keep a persistent Lance session, B-tree cache, and _take_rows adapter in an actor stage
  • add InterleavedLanceReader and allow LanceWriter to accept Arrow-backed InterleavedBatch tasks
  • document the API and add focused reader/writer/fetch tests

Why

Interleaved workflows need one reusable keyed-fetch primitive for image presence, binary payloads, metadata, hashes, and future annotation columns. Keeping the dataset identity separate from the projected payload removes per-row Lance locators and lets downstream workflows fetch only the columns they need.

Validation

  • 24 passed across the new stage, inherited Lance reader/writer tests, and the unchanged Ray adapter regression suite
  • 310 passed, 6 skipped in the broader interleaved suite; four unrelated existing log-capture assertions fail because Loguru output is not surfaced through caplog
  • real version-pinned S3 smoke test against the stable-row-ID MINT image table and its 16-segment URL B-tree
  • focused Ruff checks passed for all touched files

This PR intentionally excludes the MINT converter, Slurm workflows, pilot data-building scripts, and benchmark results.

@VibhuJawa
VibhuJawa force-pushed the feat/interleaved-lance-materialization branch from 6384b8f to 7beaac8 Compare July 3, 2026 03:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant