Skip to content

feat(arrow) ArrowGPUVector improvements - #2610

Merged
ibgreen-openai merged 2 commits into
masterfrom
ib/arrow-improvements
May 12, 2026
Merged

feat(arrow) ArrowGPUVector improvements#2610
ibgreen-openai merged 2 commits into
masterfrom
ib/arrow-improvements

Conversation

@ibgreen-openai

Copy link
Copy Markdown
Collaborator

Goals

  • Improve Arrow GPU buffer ownership semantics.
  • Support constructing ArrowGPUVector and ArrowGPUTable from existing GPU buffers/vectors.
  • Document GPU memory layouts: packed, interleaved, and segmented.
  • Prepare Arrow GPU objects for future compute/GPGPU workflows without changing GPGPU core classes.

Changes

  • Added discriminated ArrowGPUVector construction:
    • type: 'arrow' uploads Arrow vector data.
    • type: 'buffer' wraps an existing typed GPU buffer.
    • type: 'interleaved' wraps an interleaved GPU buffer as Arrow Binary.
  • Added name, byteOffset, byteStride, optional bufferLayout, and ownership tracking to ArrowGPUVector.
  • Added ArrowGPUVector.transferBufferOwnership() for in-place/reinterpreted buffer workflows.
  • Updated ArrowGPUVector.destroy() to destroy only owned buffers.
  • Added new ArrowGPUTable({vectors}) construction from named GPU vectors.
  • Derived ArrowGPUTable schema, rows, columns, buffer layouts, and attributes from supplied vectors.
  • Added support for interleaved vector metadata in ArrowGPUTable.
  • Added @luma.gl/arrow as a @luma.gl/gpgpu dependency and TypeScript project reference.
  • Added GPU memory layout documentation for packed, interleaved, and segmented buffers.
  • Updated Arrow docs/README with ownership semantics.

Tests

  • Added tests for ArrowGPUVector constructor variants and ownership transfer.
  • Added tests for ArrowGPUTable({vectors}), including interleaved layouts.
  • Ran corepack yarn build.
  • Ran targeted headless tests:
    • modules/arrow/test/arrow/arrow-fixed-size-list.spec.ts
    • modules/arrow/test/arrow/plain-gpu-table.spec.ts

@ibgreen-openai ibgreen-openai changed the title Ib/arrow improvements feat(arrow) ArrowGPUVector improvements May 12, 2026
@ibgreen-openai
ibgreen-openai merged commit ac35150 into master May 12, 2026
4 checks passed
@ibgreen-openai
ibgreen-openai deleted the ib/arrow-improvements branch May 12, 2026 13:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants