GARfVDB

[swahtz]

Summary

• Adds GARfVDB scale-conditioned instance segmentation as a first-class fvdb-reality-capture method (Python API plus frgs segment-instances), where an existing Gaussian reconstruction is consumed as input and the learned scale-conditioned feature field is the product.
• Introduces a portable, pickle-free .garfvdb bundle containing NanoVDBs (encoder grids and features), safetensors (dense network weights and scale-quantile buffers), and a PLY Gaussian model, with explicit schema versioning, checksums, and validation on load. This is the trained end-product of a GARfVDB model (i.e. the equivalent of a .ply file for the optimized Gaussian splat reconstruction scene), not a resumable checkpoint.
• Adds shared, method-neutral infrastructure so GARfVDB and any future, non-Gaussian products fit cleanly:
  • versioned training-checkpoint container with a reader registry
  • a CLI resume-handler registry
  • a reusable scene-transform configuration
  • a namespaced scene attribute for per-image mask supervision

> frgs segment-instances DATASET --reconstruction-path scene.ply --out-path scene.garfvdb trains from an existing reconstruction and writes a portable bundle

Motivation

GARfVDB previously lived as a standalone project in fvdb-examples that was already coupled to fvdb-reality-capture for SfM loading and Gaussian reconstruction, but its deliverable is a second learned product (based on a filtered Gaussian scene) rather than more Gaussians. This is the first fvdb-reality-capture method whose product is not a Gaussian splat, so the integration deliberately separates shared workflow infrastructure from product-specific behavior instead of bolting a training script onto the Gaussian commands. The shared pieces are designed against a concrete second method (for example integrating the LangSplatV2 method that also uses SAM2) so they generalize rather than hard-coding GARfVDB.

Artifact contract

The .garfvdb/ bundle is the trained end-product of the instance segmentation method (equivalent to what the .ply file is to the reconstruct process).

• A *.garfvdb/ bundle contains no PyTorch pickle files: manifest.json, encoder.nvdb, network.safetensors, and gaussians.ply.
• encoder.nvdb stores all encoder grid topologies and their learned per-voxel features through fVDB NanoVDB I/O with compression; grids are named deterministically (encoder_00 ... encoder_NN) and that order is validated on load.
• network.safetensors stores the MLP weights, optional sparse-convolution weights, maximum scale, and scale-quantile lookup buffers as named dense tensors.
• gaussians.ply stores the exact filtered GaussianSplat3d model plus reconstruction and camera metadata.

Note: The manifest carries an explicit integer schema_version and per-payload SHA-256 checksums. Loading dispatches through a version-specific reader that validates canonical grid names and order, grid/voxel counts, voxel sizes and origins, feature shape and dtype, Gaussian count, and every payload checksum. Missing, renamed, reordered, corrupted, or incompatible payloads fail with clear errors, and newer-than-supported bundle versions fail before any payload is read.

Checkpoint and resume architecture

To accommodate checkpoints beyond the reconstruction method checkpoints being resumed (and without specific methods needed to be invoked by users resume-instance-segmentation, etc.), this method formalizes the checkpoint container concept.

• New training checkpoints use a generic container: schema, schema_version, method, method_version, and method-owned state. Parsing first validates the container, selects an exact schema-version reader, and can consult registered legacy adapters for pre-container formats.
• The container carries two independent versions on orthogonal axes.
  • schema/schema_version version the container format itself — the fixed top-level keys and how they are parsed; schema is a constant shared by every method and schema_version is owned centrally, bumped only when the wrapper structure changes (which affects all methods at once).
  • method/method_version version the method-owned payload in state: method is the stable id of the producing method and method_version, owned by the method's package as a single source of truth, versions that method's state layout and is bumped only when that one method's state changes. Everything method-specific — model weights, optimizer/scheduler state, config, global step, and method metadata — lives opaquely inside state; the container layer never interprets it.
• The generic module contains no method names, CLI options, or product extensions. Method identity and legacy recognition are owned by their packages:
  • radiance_fields.checkpoint defines the Gaussian method id (radiance_fields.gaussian_splat) and a legacy adapter for released flat Gaussian checkpoints;
  • instance_segmentation.checkpoint defines the GARfVDB method id (instance_segmentation.garfvdb).
• Each method exposes a single version constant that both its state_dict/from_state_dict and its disk writer container derive from, so the recorded container version and the method state version cannot drift.
• frgs resume loads one validated container, resolves the stable method id, and invokes a CLI-owned resume handler that also owns the default output name. Unknown methods fail explicitly.
• The Gaussian reconstruction writer and reader now use the same container, and previously released flat Gaussian checkpoints remain loadable through the legacy adapter.

Transform and scene-attribute integration

So that a second, non-Gaussian method can reuse the reconstruction's scene preprocessing without duplicating it or overwriting shared scene state, the standard transform stages are factored into a reusable SceneTransformConfig, and each method's per-image supervision is carried as a namespaced scene attribute rather than by replacing the scene cache.

• A reusable SceneTransformConfig builds the standard alignment, point filtering, image downsampling, low-point image filtering, and cropping stages; both reconstruction and GARfVDB build on it. Reconstruction supplies normalization while GARfVDB injects the reconstruction's saved alignment transform.
• SAM2 mask and scene-unit scale supervision is attached as a registered, namespaced GARfVDBMaskAttribute on SfmScene rather than replacing the scene cache, so existing caches and attributes are preserved and the dataset consumes only its named attribute.
• Mask generation runs last and rejects later downsample, crop, or spatial transforms, because resizing or rescaling after generation would invalidate mask pixels or scene-unit scales. Other segmentation methods can attach their own attribute types and generators through the same generic mechanism without adopting the GARfVDB supervision format.

Interactive viewer

Updated the viewer to be able to visualize the segmentation mask output (using PCA to project the mask to a 3-channel RGB visualization) as an overlay with interactive parameters to control the visualization.

• Adds GARfVDBOverlayViewer, which renders the GARfVDB feature field as a live overlay on the Gaussians in the fvdb viewer with interactive "Scene Params" widgets:
  • a normalized grouping-scale slider (mapped to the model's maximum grouping scale)
  • an overlay-opacity slider, a show/hide checkbox for the overlay
  • a "Lock PCA colors" checkbox that freezes the PCA-to-RGB transform so the feature coloring does not flicker during camera movement
• A single shared core drives both the offline frgs show viewer and a new live training viewer enabled with frgs segment-instances --viewer, which pumps overlay renders from the training loop between steps so the in-progress feature field can be inspected while training runs.
• The offline viewer renders continuously while the camera or any widget is changing

SAM2 mask-generation performance

In addition to porting the SAM2 mask-generation transforms from fvdb-examples, this PR also makes some performance improvements during the transition:

• Vectorizes the two hottest per-image post-processing steps in mask generation: pixel_to_mask_id construction becomes a single cumulative-sum plus scatter instead of an O(masks * max_overlap) Python loop that forced a device synchronization on every inner iteration, and per-mask scale computation deduplicates on integer gaussian ids rather than sorting float3 world points. Both are equivalent to the originals.
• Stops storing the per-pixel mask-selection CDF on disk (the largest field in the mask cache, a full [H, W, max_overlap] float32 tensor) and recomputes it from pixel_to_mask_id at load time via instance_segmentation.util.compute_mask_cdf. This roughly quarters both the on-disk artifact size and the disk-bound write time.
• Overlaps cache writes with computation by handing each image's disk write to a background writer thread while the next image's SAM2 pass runs on the GPU. SfmCache's FileLock now only arms its SIGALRM-based timeout on the main thread and falls back to a plain blocking flock on worker threads, since signal-based timeouts cannot be used off the main thread.
• Exposes and raises SAM2 points_per_batch so more point prompts are processed per mask-decoder forward pass.

Device defaults and import updates

• Defaults the compute device to cuda:0 across the CLI, foundation models, config, and docs, because the current fvdb build requires an explicit device index and cuda alone raises "Device must specify an index".

Dependencies

• Requires fvdb-core>=0.6.0 and adds safetensors.
• The new fvdb.viz features that bring the API changes to add interactive, definable parameters require this PR to merge before this branch will work: Add viewer widgets fvdb-core#649
• cuML clustering, NVOS evaluation, and discrete-instance export remain out of the core dependency set and stay in fvdb-examples for the moment.

Documentation

• Adds an instance-segmentation tutorial and API pages, a checkpoints API page, and resume/show/transforms updates describing the bundle layout, grid naming, coordinate transforms, versioning, and how future readers or migrations are introduced.

Testing

• Ported training math is at parity with the prototype for loss (including per-view pair-count normalization), mask-CDF sampling, scale interpolation, pixel sampling, encoder feature extraction, MLP forward paths, and SAM2 mask/scale preprocessing. The port additionally fixes the prototype's scheduler-resume gap by checkpointing and restoring scheduler state.
• The interactive viewer is covered by unit/test_garfvdb_viewer.py (widget registration, normalized-to-raw scale mapping, camera/widget change detection and the render/idle contract, show/hide, and PCA lock), and the vectorized mask post-processing and recomputed mask CDF were checked for equivalence against the original implementations, including against a real on-disk cache file.

Backwards compatibility and follow-ups

• Released flat Gaussian reconstruction checkpoints remain loadable and resumable through the registered legacy adapter; a Gaussian .ply remains an export product and is not resumable.
• Standalone fvdb-examples GARfVDB checkpoints are intentionally not migrated; there is no importer for the prototype format, and old SAM2 mask caches must be regenerated because supervision is now a versioned scene attribute.
• use_grid_conv=True remains experimental and is not exercised by the first-class path; the per-Gaussian affinity path references a grid attribute that does not exist and would fail if enabled. First-class training and products require use_grid=True, so this does not affect the supported configuration, but the option should be fixed or gated before it is advertised as supported.
• The overlay-opacity slider depends on an upstream nanovdb-editor fix (its image2d.slang composited the image view opaquely and ignored per-pixel alpha) Fix compositing image2d over the scene using per-pixel alpha nanovdb-editor#217

[harrism]

Submitting a partial review. Haven't reviewed the big files yet.

[Copilot]

Pull request overview

This PR integrates GARfVDB scale-conditioned instance segmentation into fvdb-reality-capture as a first-class method, including a portable .garfvdb inference bundle format, shared checkpoint/resume infrastructure, and viewer/transform updates to support non-Gaussian derived products alongside Gaussian reconstruction.

Changes:

• Added GARfVDB training + inference product support (artifact I/O, CLI command, scene attribute/transforms, viewer overlay, and unit tests).
• Introduced a method-neutral, versioned training checkpoint container plus a CLI resume-handler registry; migrated Gaussian checkpoints to the container while keeping legacy flat Gaussian checkpoints loadable.
• Refactored common scene preprocessing into a reusable SceneTransformConfig and updated CLI/device defaults and docs accordingly.

Reviewed changes

Copilot reviewed 68 out of 69 changed files in this pull request and generated 5 comments.

Show a summary per file

File	Description
tests/unit/test_training_checkpoints.py	Tests for checkpoint container parsing/loading and resume dispatch registry behavior.
tests/unit/test_garfvdb_viewer.py	Unit tests for GARfVDB overlay viewer core behavior and widget-driven rerendering.
tests/unit/test_garfvdb_scene_transforms.py	Tests for GARfVDB mask attribute + transform ordering and cache behaviors.
tests/unit/test_garfvdb_cli.py	CLI contract tests for frgs instance-segmentation, frgs show, and frgs resume.
tests/unit/test_garfvdb_artifact.py	Tests for .garfvdb bundle format, schema versioning, validation, and round-trips.
tests/benchmarks/test_3dgs.py	Updated benchmark to load Gaussian checkpoints via the container loader.
tests/benchmarks/contract.py	Contract bump + validation through container parsing and method ID checking.
README.md	README updated to mention instance feature fields alongside radiance fields.
pyproject.toml	Adds safetensors dependency.
fvdb_reality_capture/transforms/scene_transform_config.py	New reusable scene transform configuration/pipeline builder.
fvdb_reality_capture/transforms/init.py	Exposes SceneTransformConfig in transforms public API.
fvdb_reality_capture/sfm_scene/sfm_cache.py	File lock updated to avoid SIGALRM timeout logic on worker threads.
fvdb_reality_capture/radiance_fields/io.py	New helper to load splats from PLY or checkpoint container.
fvdb_reality_capture/radiance_fields/gaussian_splat_reconstruction.py	Uses method version constant; default device updated to cuda:0.
fvdb_reality_capture/radiance_fields/gaussian_splat_reconstruction_writer.py	Writes checkpoints as versioned containers.
fvdb_reality_capture/radiance_fields/checkpoint.py	New Gaussian method ID/version constants + legacy adapter registration.
fvdb_reality_capture/radiance_fields/init.py	Re-exports method ID/version constants and load_splats_from_file.
fvdb_reality_capture/instance_segmentation/viewer.py	Interactive GARfVDB overlay viewer + show_garfvdb_bundle.
fvdb_reality_capture/instance_segmentation/util.py	New utilities incl. PCA projection helpers and mask CDF computation.
fvdb_reality_capture/instance_segmentation/training/segmentation_writer.py	New GARfVDB training writer (images/metrics/checkpoints).
fvdb_reality_capture/instance_segmentation/training/dataset.py	Segmentation dataset that loads mask attribute and recomputes mask CDF.
fvdb_reality_capture/instance_segmentation/training/dataset_transforms.py	Dataset transforms for mask selection, pixel sampling, resizing, etc.
fvdb_reality_capture/instance_segmentation/training/init.py	Training package initializer.
fvdb_reality_capture/instance_segmentation/scene_transforms/init.py	Exposes GenerateGARfVDBMasks.
fvdb_reality_capture/instance_segmentation/scene_attribute.py	New namespaced scene attribute for per-image GARfVDB mask supervision.
fvdb_reality_capture/instance_segmentation/optim.py	Adds Exponential LR scheduler with ramp-up for segmentation training.
fvdb_reality_capture/instance_segmentation/loss.py	Adds GARfVDB contrastive loss implementation and per-view normalization.
fvdb_reality_capture/instance_segmentation/garfvdb.py	Public GARfVDB inference product API (render, affinities, save/load).
fvdb_reality_capture/instance_segmentation/config.py	GARfVDB model/training/transform configs leveraging shared transform config.
fvdb_reality_capture/instance_segmentation/checkpoint.py	GARfVDB checkpoint method ID/version constants.
fvdb_reality_capture/instance_segmentation/artifact.py	Portable .garfvdb bundle save/load with schema versioning + checksums.
fvdb_reality_capture/instance_segmentation/init.py	Public instance-segmentation module exports.
fvdb_reality_capture/foundation_models/sam2.py	Default device cuda:0 + points_per_batch applied in flat mode.
fvdb_reality_capture/foundation_models/sam1.py	Default device updated to cuda:0.
fvdb_reality_capture/foundation_models/openclip.py	Default device updated to cuda:0.
fvdb_reality_capture/foundation_models/dlnr.py	Default device updated to cuda:0 and docstring updated.
fvdb_reality_capture/cli/frgs/_show.py	frgs show now supports .garfvdb bundles and GARfVDB overlay controls.
fvdb_reality_capture/cli/frgs/_resume.py	frgs resume now dispatches via checkpoint container + resume registry.
fvdb_reality_capture/cli/frgs/_resume_registry.py	New CLI-owned resume-handler registry + method lookup.
fvdb_reality_capture/cli/frgs/_reconstruct.py	CLI transform config now subclasses shared SceneTransformConfig; device default cuda:0.
fvdb_reality_capture/cli/frgs/_points.py	Device default updated to cuda:0.
fvdb_reality_capture/cli/frgs/_mesh_dlnr.py	Device default updated to cuda:0.
fvdb_reality_capture/cli/frgs/_mesh_basic.py	Device default updated to cuda:0.
fvdb_reality_capture/cli/frgs/_instance_segmentation.py	New frgs instance-segmentation command.
fvdb_reality_capture/cli/frgs/_evaluate.py	Loads Gaussian checkpoints via container loader; device default cuda:0.
fvdb_reality_capture/cli/frgs/_convert.py	Loads Gaussian checkpoints via container loader for conversion paths.
fvdb_reality_capture/cli/frgs/_common.py	Switches to library-layer load_splats_from_file and re-exports it.
fvdb_reality_capture/cli/frgs/init.py	Registers InstanceSegmentation command with CLI entrypoint.
fvdb_reality_capture/checkpoints.py	New method-neutral, versioned training checkpoint container + legacy adapters.
fvdb_reality_capture/init.py	Exposes new submodules (including checkpoints and instance_segmentation).
docs/tutorials/instance_segmentation.rst	New tutorial covering training, artifact layout, viewing, and resume.
docs/index.rst	Adds instance segmentation tutorial/API docs and checkpoints API page.
docs/api/transforms.rst	Documents SceneTransformConfig in transforms API listing.
docs/api/instance_segmentation.rst	New instance segmentation API page.
docs/api/frgs/show.rst	Updates frgs show docs for .garfvdb support and device default.
docs/api/frgs/resume.rst	Updates frgs resume docs for container dispatch and new options.
docs/api/frgs/reconstruct.rst	Updates reconstruct docs for device default cuda:0.
docs/api/frgs/points.rst	Updates points docs for device default cuda:0.
docs/api/frgs/mesh_dlnr.rst	Updates mesh-dlnr docs for device default cuda:0.
docs/api/frgs/mesh_basic.rst	Updates mesh-basic docs for device default cuda:0.
docs/api/frgs/instance_segmentation.rst	New CLI doc snippet for frgs instance-segmentation.
docs/api/frgs/evaluate.rst	Updates evaluate docs for device default cuda:0.
docs/api/frgs.rst	Updates CLI overview and includes instance-segmentation docs.
docs/api/checkpoints.rst	New documentation page for the training checkpoint container design.
CHANGES.md	Adds an unreleased 0.6.0 changelog section describing GARfVDB + infra.
.gitignore	Ignores .garfvdb bundles.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[Copilot]

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

fvdb_reality_capture/instance_segmentation/training/dataset_transforms.py:82

• RandomSelectMaskIDAndScale builds random tensors via torch.full(..., torch.rand(...).item()) without specifying a device. If this transform is ever applied to CUDA tensors (e.g. after moving a batch to GPU), it will create CPU random tensors and fail with a device-mismatch error in comparisons / indexing. It’s easy to keep the intended “one random value per image” behavior while allocating on per_pixel_index.device.

[Copilot]

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 2 comments.

[Copilot]

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 3 comments.

[Copilot]

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 1 comment.

[Copilot]

Pull request overview

Copilot reviewed 72 out of 73 changed files in this pull request and generated 1 comment.

Comments suppressed due to low confidence (1)

docs/api/frgs/show.rst:28

• The frgs show help snippet appears incomplete: the CLI now exposes GARfVDB-specific flags (e.g. --scale-fraction, --mask-blend, --lock-pca-colors, overlay sizing), but they are not reflected here. This documentation will be out of sync with the actual CLI interface.

    │ -v, --verbose, --no-verbose                                                                  │
    │                         If True, then the viewer will log verbosely. (default: False)        │
    │ --device STR|DEVICE     Device to use for computation (default is "cuda:0"). (default: cuda:0) │
    ╰──────────────────────────────────────────────────────────────────────────────────────────────╯