|
1 | 1 | # @luma.gl/splats |
2 | 2 |
|
3 | | -Experimental Gaussian splat rendering utilities for luma.gl. This package owns prepared GPU splat |
4 | | -data, anisotropic Gaussian rendering, and backend-specific render models without depending on |
5 | | -Apache Arrow, loaders.gl, or deck.gl. |
| 3 | +Experimental Gaussian splat rendering utilities for luma.gl, without dependencies on Apache Arrow, |
| 4 | +loaders.gl, or deck.gl. |
6 | 5 |
|
7 | | -The package is a private luma.gl workspace and is not published to npm. Install dependencies from |
8 | | -the repository root and reference it from another workspace with `"@luma.gl/splats": "workspace:*"`. |
| 6 | +`makeGPUSplatData(...)` prepares caller-owned GPU data. `SplatRenderer` supports WebGPU and WebGL2; |
| 7 | +`GPUSplatGraphRenderer` progressively streams preserved batches through reusable WebGPU command |
| 8 | +graphs, global GPU sorting, and one indirect draw. |
9 | 9 |
|
10 | | -## Usage |
| 10 | +Use optional `expectedSplatCount` and `expectedBatchCount` hints to reserve graph capacity. |
| 11 | +Renderers borrow source batches, preserve HDR colors, and must be destroyed before their data. |
11 | 12 |
|
12 | | -```ts |
13 | | -import {makeGPUSplatData, SplatRenderer} from '@luma.gl/splats'; |
| 13 | +This private package is not published to npm. Reference it from another luma.gl workspace with |
| 14 | +`"@luma.gl/splats": "workspace:*"`. |
14 | 15 |
|
15 | | -const splatData = makeGPUSplatData(device, { |
16 | | - positions: new Float32Array([0, 0, -2]), |
17 | | - scales: new Float32Array([0.25, 0.12, 0.08]), |
18 | | - rotations: new Float32Array([1, 0, 0, 0]), |
19 | | - colors: new Uint8Array([235, 150, 80, 255]), |
20 | | - opacities: new Float32Array([0.8]) |
21 | | -}); |
22 | | - |
23 | | -const renderer = new SplatRenderer(device, {data: splatData}); |
24 | | - |
25 | | -renderer.draw(renderPass); |
26 | | -renderer.appendData(nextSplatBatch); |
27 | | - |
28 | | -// Destroy borrowing renderers before destroying caller-owned prepared data. |
29 | | -renderer.destroy(); |
30 | | -splatData.destroy(); |
31 | | -nextSplatBatch.destroy(); |
32 | | -``` |
33 | | - |
34 | | -`GPUSplatData` is caller-owned prepared data. Renderers borrow source buffers and release only their |
35 | | -own models and temporary resources. Streaming retains source batch boundaries instead of |
36 | | -concatenating or reuploading previously prepared batches. |
37 | | - |
38 | | -Source colors can use normalized `Uint8Array` RGBA or linear `Float32Array` RGBA. Floating-point |
39 | | -colors preserve high-dynamic-range radiance without premature clamping or quantization. The |
40 | | -renderer automatically applies Reinhard highlight compression on standard dynamic range targets, |
41 | | -preserves extended WebGPU presentation output, and supports explicit `exposure` and `toneMapping` |
42 | | -controls. |
43 | | - |
44 | | -## WebGPU command-graph rendering |
45 | | - |
46 | | -`GPUSplatGraphRenderer` is a WebGPU-only alternative for large captured scenes. Its compiled GPU |
47 | | -command graph projects and culls each borrowed source batch, globally sorts camera-dependent depth |
48 | | -keys, and renders every visible Gaussian with one GPU-driven indirect draw: |
49 | | - |
50 | | -```ts |
51 | | -import {GPUSplatGraphRenderer, type GPUSplatData} from '@luma.gl/splats'; |
52 | | - |
53 | | -const renderer = new GPUSplatGraphRenderer(webgpuDevice, { |
54 | | - viewportSize: [width, height], |
55 | | - expectedSplatCount: 741_883, |
56 | | - expectedBatchCount: 12 |
57 | | -}); |
58 | | - |
59 | | -const preparedBatches: GPUSplatData[] = []; |
60 | | -for await (const batch of preparedBatchStream) { |
61 | | - preparedBatches.push(batch); |
62 | | - renderer.appendData(batch); |
63 | | - |
64 | | - const commandEncoder = webgpuDevice.createCommandEncoder(); |
65 | | - renderer.encode(commandEncoder); |
66 | | - webgpuDevice.submit(commandEncoder.finish()); |
67 | | -} |
68 | | - |
69 | | -// Source batches belong to their caller and must outlive the borrowing renderer. |
70 | | -renderer.destroy(); |
71 | | -for (const batch of preparedBatches) { |
72 | | - batch.destroy(); |
73 | | -} |
74 | | -``` |
75 | | - |
76 | | -The example reserves room for the complete 741,883-splat Train scene and its twelve Arrow batches. |
77 | | -The first nonempty batch compiles the graph and renders immediately; later batches replace their |
78 | | -reserved source bindings without replacing the graph. The resulting pipeline is: |
79 | | - |
80 | | -```text |
81 | | -Caller-owned, independently allocated batches One compiled GPU command graph |
82 | | -
|
83 | | -batch 0: five original GPU columns --------+ |
84 | | -batch 1: five original GPU columns --------+--> reusable imported source slots |
85 | | -batch N: five original GPU columns --------+ | |
86 | | - v |
87 | | -initialize every reserved row ---------------> project and cull populated slots |
88 | | -identity indices + invalid depth keys per-slot uniforms + 48-byte records |
89 | | -reset indirect visible count | |
90 | | - +--> global 16-bit radix sort --+ |
91 | | - | | |
92 | | - +--> atomic visible count -----+ |
93 | | - | |
94 | | - v |
95 | | - one indirect draw |
96 | | -``` |
97 | | - |
98 | | -### Source batches and reusable graph slots |
99 | | - |
100 | | -Parsing remains outside this package: `@loaders.gl/splats` recognizes file formats, `@luma.gl/arrow` |
101 | | -converts individual Apache Arrow record batches to `GPUSplatData`, and this renderer consumes the |
102 | | -resulting framework-neutral GPU objects. Source batches stay separate and retain their original |
103 | | -buffer allocations throughout loading, rendering, capacity growth, and destruction. |
104 | | - |
105 | | -Each reserved graph slot represents exactly one source batch and imports five columns: positions, |
106 | | -scales, rotations, colors, and opacities. Graph topology is immutable after compilation, so every |
107 | | -slot initially has five distinct, renderer-owned placeholder buffers. Their minimum sizes are 12, |
108 | | -12, 16, 4, and 4 bytes respectively. At encoding time, populated slots override those imported |
109 | | -buffers with the original caller-owned allocations; unpopulated slots keep their harmless |
110 | | -placeholders and do not dispatch projection work. This changes bindings, not graph topology, and |
111 | | -never concatenates, repacks, copies, or reuploads existing source data. |
112 | | - |
113 | | -A separate 128-byte uniform buffer per slot carries the camera transform, viewport, Gaussian |
114 | | -styling, batch offset, active row count, color format, exposure, and tone-mapping mode. Updating |
115 | | -these uniforms costs work proportional to the reserved batch-slot count, not the splat-row count. |
116 | | - |
117 | | -### GPU execution phases |
118 | | - |
119 | | -Each necessary `encode(commandEncoder)` records the following dependent graph operations into the |
120 | | -caller's command encoder. The caller still owns submission; no intermediate submissions or |
121 | | -synchronous GPU readbacks are required. |
122 | | - |
123 | | -1. **Initialize reserved rows.** A compute pass writes stable identity indices across the entire |
124 | | - reserved row capacity, initializes every depth key to the invalid sentinel `65,535`, and resets |
125 | | - the indirect draw's visible-instance count to zero. Unfilled capacity therefore cannot be drawn. |
126 | | - |
127 | | -2. **Project and cull active batches.** One compute node per reserved slot processes its source |
128 | | - only when that slot contains a nonempty batch. Visible anisotropic Gaussians produce 48-byte |
129 | | - camera-dependent records containing clip position, screen-space axes, and linear RGBA radiance. |
130 | | - Invalid, clipped, transparent, or otherwise culled rows keep their sentinel keys. Valid rows |
131 | | - receive far-to-near keys in the range `0` through `65,534`, and each increments the indirect |
132 | | - visible-instance count atomically. Packed `Uint8Array` colors and floating-point HDR colors can |
133 | | - coexist in different batches. |
134 | | - |
135 | | -3. **Globally sort depth keys.** `GPUSort` performs a stable ascending radix sort using only the 16 |
136 | | - meaningful depth-key bits. Its values are global projected-record indices, so transparent |
137 | | - Gaussians from different source batches share one correct back-to-front order. The maximum |
138 | | - sentinel sorts culled and inactive rows after every visible record. |
139 | | - |
140 | | -4. **Render once.** One graph-native indirect render pass consumes sorted projected-record indices |
141 | | - and the GPU-written visible count. The fragment shader expands and blends anisotropic Gaussian |
142 | | - quads, preserving source HDR radiance until exposure and display-appropriate tone mapping are |
143 | | - applied. No source-batch rebinding or one-draw-per-batch ordering is required in this pass. |
144 | | - |
145 | | -### Progressive capacity and graph lifetime |
146 | | - |
147 | | -`expectedSplatCount` and `expectedBatchCount` are optional, positive-integer capacity hints. When |
148 | | -the complete source metadata is known, supplying both reserves the final row and batch-slot |
149 | | -capacities up front. The first nonempty `encode()` compiles one graph; appending any batch that |
150 | | -fits both capacities reuses the same `renderer.compiledGraph` object and only updates its imported |
151 | | -source bindings and uniforms. |
152 | | - |
153 | | -Unknown or underestimated dimensions remain valid: |
154 | | - |
155 | | -- Without a row hint, the initial capacity is at least four times the first nonempty batch size; |
156 | | - without a batch hint, at least four source slots are reserved. |
157 | | -- If incoming rows or batch slots exceed their current capacity, the corresponding capacity |
158 | | - doubles until it can accommodate the accumulated stream. A new graph is compiled only at that |
159 | | - growth boundary; subsequent fitting batches reuse it. |
160 | | -- `renderer.capacity` exposes the current allocated row and slot counts, while `renderer.stats` |
161 | | - reports actual loaded rows, actual loaded batches, and source-versus-renderer GPU memory. |
162 | | -- Appending a batch or changing camera/style properties marks the graph dirty. Once the stream and |
163 | | - camera are stationary, another `encode()` returns `undefined` instead of repeating projection, |
164 | | - sorting, and drawing. |
165 | | - |
166 | | -The lifecycle is therefore: no source rows means no graph; the first nonempty batch compiles and |
167 | | -renders immediately; fitting batches stream through the same graph; capacity overflow occasionally |
168 | | -recompiles; and a stationary completed scene performs no repeated GPU work. Replacing the entire |
169 | | -source list with `setProps({data})` invalidates the old graph without destroying either the old or |
170 | | -new caller-owned batches. |
171 | | - |
172 | | -### Ownership, precision, and performance tradeoffs |
173 | | - |
174 | | -`GPUSplatData` and all original source-column allocations remain caller-owned and must outlive the |
175 | | -renderer. The renderer owns only derived projected records, depth keys, source indices, sorted |
176 | | -keys/indices, radix-sort scratch, uniform buffers, slot placeholders, the indirect command, and its |
177 | | -graph/model resources. Call `renderer.destroy()` before destroying the prepared source batches. |
178 | | - |
179 | | -The baseline persistent working buffers require 64 bytes per reserved row: 48 bytes for each |
180 | | -projected record and four 4-byte key/index arrays. Reserving the complete 741,883-splat Train scene |
181 | | -therefore uses approximately 45.28 MiB before graph scratch, per-slot uniforms/placeholders, and |
182 | | -the separately owned source columns. `renderer.stats` and `renderer.graphStats` expose the actual |
183 | | -allocation and scheduling diagnostics without reading back source rows or visible counts. |
184 | | - |
185 | | -The fixed-topology radix sort currently processes the **entire reserved row capacity on every dirty |
186 | | -frame**, even while only the first streamed batch is populated. Exact final-size hints eliminate |
187 | | -recompilation and improve CPU responsiveness, but front-load GPU memory and full-scene sorting |
188 | | -work; unknown-size geometric growth trades occasional recompilation for less initially reserved |
189 | | -capacity. Sixteen-bit keys reduce radix work but provide finite depth-ordering precision; the sort |
190 | | -is stable when rows quantize to the same key. |
191 | | - |
192 | | -Projected records also must fit the adapter's `maxStorageBufferBindingSize`. For the common 128 MiB |
193 | | -binding limit, the 48-byte stride permits at most 2,796,202 projected rows in one graph; actual |
194 | | -available GPU memory or other implementation limits may lower that practical ceiling. Applications |
195 | | -should retain their existing fallback policy when an adapter cannot allocate the requested scene. |
196 | | - |
197 | | -`GPUSplatGraphRenderer` requires WebGPU. Continue to use `SplatRenderer` for WebGL2 devices or for |
198 | | -explicit CPU-ordered comparison; both renderers preserve caller ownership and original source-batch |
199 | | -boundaries. Keep deck.gl-specific layers in downstream adapters instead of introducing framework |
200 | | -or file-format dependencies into `@luma.gl/splats`. |
| 16 | +See [luma.gl](https://luma.gl/docs/api-reference/splats) for documentation. |
0 commit comments