Skip to content

feat(experimental): add stable luDF sorting and top-K - #2959

Open
ibgreen-openai wants to merge 2 commits into
codex/ludf-reductions-histogramsfrom
codex/ludf-sorting-topk
Open

feat(experimental): add stable luDF sorting and top-K#2959
ibgreen-openai wants to merge 2 commits into
codex/ludf-reductions-histogramsfrom
codex/ludf-sorting-topk

Conversation

@ibgreen-openai

Copy link
Copy Markdown
Collaborator

Goals

Expose stable, GPU-resident numeric sorting and top-K selection through the optional luDF dataframe API while preserving source record batches, stable row identity, explicit null behavior, and caller-owned command submission.

Changes

  • Add immutable LuDataFrame.sortBy, LuDataFrame.topK, and matching query methods for float32, sint32, and uint32 columns, with typed direction, null placement, NaN placement, and existing bitonic/radix algorithm controls.
  • Compile two stable per-batch GPU sorting passes using full-width monotonic numeric keys followed by explicit selected/null/NaN classes; preserve duplicate stability, signed zeros, infinities, the complete unsigned domain, and encoder-ordered filter updates.
  • Publish sorted stable source-row identifiers, coherent top-K selection masks, and per-batch GPU counts without packing source chunks, submitting commands, or reading rows back to the CPU.
  • Reject scalar constants, unavailable nullable sidecars, invalid limits/source offsets, and padded or interleaved scalar buffers before GPU allocation.
  • Cover nullable floating-point and signed/unsigned extrema, empty batches, derived columns, retained owned-source leases, explicit null/NaN ordering, bounded top-K, and repeated graph encoding in focused Node and real-WebGPU tests.

Verification

  • nvm use: Node v22.22.1.
  • yarn install: not rerun; reused the already verified local dependency tree because the existing enterprise alpha registry dependency returns HTTP 403.
  • yarn lint fix: passed after final source changes.
  • yarn build: passed after final formatting.
  • yarn test: Node 606 passed/1 skipped; real headless Chromium/WebGPU 1,581 passed/25 skipped.
  • Focused luDF Node suite: 7 files/63 tests; focused real-WebGPU sorting suite: 5 tests.
  • yarn website:build: passed and validated 461 documentation pages.
  • (cd website && yarn build): passed and validated 461 documentation pages.
  • yarn examples:typecheck: passed across all 46 example workspaces.
  • yarn bundle-size: passed all seven existing budgets.
  • Built optional ESM and CommonJS luDF sorting exports were loaded successfully.

Risks and follow-up

  • Sorting is explicitly per existing source batch; global materialization, nonnumeric ordering, and sorting constant columns are intentionally unsupported.
  • Nullable rows and NaNs are kept as explicit independently ordered classes; deselected rows always remain outside the published count.
  • Independent physical-alias and scaled-dispatch prerequisites are tracked separately.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants