GARfVDB

.gitignore

@@ -78,3 +78,4 @@ tests/data/
 *.h264
 *.mp4
 *.ipynb_checkpoints
+*.garfvdb

CHANGES.md

@@ -1,7 +1,19 @@
 fVDB-Reality-Capture Version History
 ====================================
 
-## Unreleased
+## Version 0.6.0 - Unreleased
+
+- Added first-class GARfVDB scale-conditioned instance segmentation through the Python API and
+  `frgs instance-segmentation` command.
+- Added portable `.garfvdb` bundles using NanoVDB encoder fields, safetensors network weights, and a PLY Gaussian
+  carrier. Manifests carry an explicit `schema_version` used to select an exact reader for future compatibility.
+  Added product-aware `frgs show` and `frgs resume` dispatch.
+- Integrated GARfVDB preprocessing with the standard scene-transform pipeline. SAM2 supervision is represented by a
+  registered, namespaced scene attribute so it composes with other derived scene products without replacing the
+  scene cache or discarding existing attributes.
+- Added a method-neutral, versioned training-checkpoint envelope and resume-handler registry. `frgs resume` now
+  dispatches stable method IDs without relying on product extensions or falling through unknown methods to Gaussian
+  reconstruction.
 
 ### Gaussian Splatting API
 

README.md

@@ -2,7 +2,8 @@
 
 fVDB-Reality-Capture is a reality-capture toolbox built on top of [fVDB](https://fvdb-core.readthedocs.io). It
 provides high-level abstractions and APIs for common reality capture tasks, such as loading sensor data, reconstructing
-radiance fields, extracting meshes and point clouds, visualization, and exporting results across standard formats such
+radiance fields, learning scale-conditioned instance feature fields, extracting meshes and point clouds, visualization,
+and exporting results across standard formats such
 as PLY and USD (USDC/USDZ).
 
 By leveraging the power of fVDB, fVDB-Reality-Capture can scale reconstruction to very large or dense

docs/api/checkpoints.rst

@@ -0,0 +1,37 @@
+Training checkpoints
+====================
+
+Reality Capture training checkpoints use a method-neutral, versioned container. The container identifies the stable
+method ID and keeps method-owned optimizer, dataset, and model state inside ``state``. Product extensions such as
+``.garfvdb`` are not used for resume dispatch.
+
+Each method owns its method ID and any adapter for checkpoints written before the container was introduced. The
+``frgs resume`` command separately owns resume-handler registration and output defaults, keeping CLI policy out of
+the serialization layer.
+
+Container schema versus method
+------------------------------
+
+A checkpoint carries two independent versions — one for the outer container and one for the method payload inside
+it — on orthogonal axes:
+
+- ``schema`` / ``schema_version`` version the **container format itself**: the fixed set of top-level keys
+  (``schema``, ``schema_version``, ``method``, ``method_version``, ``state``) and how they are parsed. ``schema`` is
+  a constant string shared by every method, and ``schema_version`` is owned centrally by
+  :mod:`fvdb_reality_capture.checkpoints` and selects the reader used to parse the container. Bump it only when the
+  container structure changes, which affects every method at once. Nothing method-specific lives at this level.
+- ``method`` / ``method_version`` version the **method-owned payload** carried in ``state``. ``method`` is the stable
+  identifier of the training method that produced the checkpoint (for example ``radiance_fields.gaussian_splat`` or
+  ``instance_segmentation.garfvdb``) and is the dispatch key used on resume. ``method_version`` versions that
+  method's ``state`` layout and is owned by the method's package as a single source of truth, so the writer's
+  recorded version and the method's ``state_dict``/``from_state_dict`` cannot drift. Bump it only when that one
+  method's ``state`` changes, independently of other methods.
+
+Everything method-specific — model weights, optimizer and scheduler state, configuration, global step, and method
+metadata — lives inside ``state`` and is opaque to the container layer; only the method interprets it. Because the
+axes are independent, the container can stay at one ``schema_version`` while different methods evolve their
+``method_version`` separately, and restructuring the container advances ``schema_version`` regardless of any method.
+
+.. automodule:: fvdb_reality_capture.checkpoints
+   :members: TrainingCheckpoint, create_training_checkpoint, load_training_checkpoint,
+             parse_training_checkpoint, register_legacy_checkpoint_adapter

docs/api/frgs.rst

@@ -1,11 +1,12 @@
 ``frgs 🐸 CLI Tool``
 ==========================================
 
-fVDB-Reality-Capture provides a command-line interface (CLI) tool called ``frgs`` for tools related to Gaussian splatting.
+fVDB-Reality-Capture provides a command-line interface (CLI) called ``frgs`` for tools related to Gaussian Splatting
+and derived spatial products.
 ``frgs`` is pronounced "frogs" 🐸, and is short for  **F**\ vdb-\ **R**\ eality-capture **G**\ aussian **S**\ platting.
 
-This tool allows users to perform Gaussian-splatting-related tasks such as reconstruction, format conversions,
-mesh and point cloud extraction, and visualization directly from the command line, without needing to write any code.
+`frgs` provides reconstruction, derived instance feature fields, format conversion, mesh and point-cloud extraction,
+and visualization directly from the command line without requiring Python code.
 
 
 ``frgs convert``
@@ -28,6 +29,11 @@ mesh and point cloud extraction, and visualization directly from the command lin
 
 .. include:: frgs/mesh_basic.rst
 
+``frgs instance-segmentation``
+--------------------------------
+
+.. include:: frgs/instance_segmentation.rst
+
 ``frgs mesh-dlnr``
 ----------------------
 

docs/api/frgs/evaluate.rst

@@ -45,6 +45,6 @@
     │ -s, --save-images, --no-save-images                                                            │
     │     Whether to save the rendered images. Defaults to True. (default: True)                     │
     │ --device STR|DEVICE                                                                            │
-    │     Device to use for computation. Defaults to "cuda". (default: cuda)                         │
+    │     Device to use for computation. Defaults to "cuda:0". (default: cuda:0)                     │
     ╰────────────────────────────────────────────────────────────────────────────────────────────────╯
 

docs/api/frgs/instance_segmentation.rst

@@ -0,0 +1,15 @@
+.. code-block:: text
+
+   usage: frgs instance-segmentation [-h] [INSTANCE-SEGMENTATION OPTIONS] PATH
+
+   Train a GARfVDB scale-conditioned instance feature field from an existing Reality Capture reconstruction.
+
+   Example:
+
+       frgs instance-segmentation ./colmap_dataset \
+           --reconstruction-path scene.ply \
+           --out-path scene.garfvdb
+
+   Required inputs are the original posed-image dataset and the matching Reality Capture PLY or checkpoint.
+   The output is a portable directory containing NanoVDB encoder grids, safetensors network weights, and the
+   exact filtered Gaussian model.

docs/api/frgs/mesh_basic.rst

@@ -67,6 +67,6 @@
    │ -o PATH, --output-path PATH                                                                    │
    │                         Path to save the extracted mesh (default is "mesh.ply"). (default:     │
    │                         mesh.ply)                                                              │
-   │ -d STR, --device STR    Extract a mesh from a Gaussian Splat reconstruction. (default: cuda)   │
+   │ -d STR, --device STR    Device to use for computation (default is "cuda:0"). (default: cuda:0) │
    ╰────────────────────────────────────────────────────────────────────────────────────────────────╯
 

docs/api/frgs/mesh_dlnr.rst

@@ -66,6 +66,6 @@
    │ -o PATH, --output-path PATH                                                                    │
    │                         Path to save the extracted mesh (default is "mesh.ply"). (default:     │
    │                         mesh.ply)                                                              │
-   │ -d STR, --device STR    Extract a mesh from a Gaussian Splat reconstruction. (default: cuda)   │
+   │ -d STR, --device STR    Device to use for computation (default is "cuda:0"). (default: cuda:0) │
    ╰────────────────────────────────────────────────────────────────────────────────────────────────╯
 

docs/api/frgs/points.rst

@@ -58,6 +58,6 @@
     │ -o PATH, --output-path PATH                                                                    │
     │                         Path to save the extracted mesh (default is "mesh.ply"). (default:     │
     │                         points.ply)                                                            │
-    │ -d STR, --device STR    Device to use for computation (default is "cuda"). (default: cuda)     │
+    │ -d STR, --device STR    Device to use for computation (default is "cuda:0"). (default: cuda:0) │
     ╰────────────────────────────────────────────────────────────────────────────────────────────────╯
 

docs/api/frgs/reconstruct.rst

@@ -42,7 +42,7 @@
     │ -d STR|DEVICE, --device STR|DEVICE                                                           │
     │     Which device to use for reconstruction. Must be a cuda device. You can pass in a         │
     │     specific device index via cuda:N where N is the device index, or "cuda" to use the       │
-    │     default cuda device. CPU is not supported. Default is "cuda". (default: cuda)            │
+    │     default cuda device. CPU is not supported. Default is "cuda:0". (default: cuda:0)        │
     │ -v, --verbose, --no-verbose                                                                  │
     │     If set, show verbose debug messages. (default: False)                                    │
     │ -nc INT INT INT, --nchunks INT INT INT                                                       │

docs/api/frgs/resume.rst

@@ -2,9 +2,8 @@
 
     usage: frgs resume [-h] [RESUME OPTIONS] PATH
 
-    Resume reconstructing a 3D Gaussian Splat radiance field from a checkpoint. This command loads a
-    model checkpoint and continues reconstruction from that point. The dataset used to create the
-    checkpoint must be at the same path as when the checkpoint was created.
+    Resume a Reality Capture training run. The versioned checkpoint container identifies the method and dispatches
+    its registered resume handler. Source dataset paths recorded by the checkpoint must remain available.
 
     Example usage:
 
@@ -13,7 +12,7 @@
 
     ╭─ positional arguments ───────────────────────────────────────────────────────────────────────╮
     │ PATH                                                                                         │
-    │     Path to the checkpoint file containing the Gaussian Splat radiance field. (required)     │
+    │     Path to a versioned Reality Capture training checkpoint. (required)                      │
     ╰──────────────────────────────────────────────────────────────────────────────────────────────╯
     ╭─ options ────────────────────────────────────────────────────────────────────────────────────╮
     │ -h, --help                                                                                   │
@@ -32,12 +31,13 @@
     │ -d STR|DEVICE, --device STR|DEVICE                                                           │
     │     Which device to use for reconstruction. Must be a cuda device. You can pass in a         │
     │     specific device index via cuda:N where N is the device index, or "cuda" to use the       │
-    │     default cuda device. CPU is not supported. Default is "cuda". (default: cuda)            │
+    │     default cuda device. CPU is not supported. Default is "cuda:0". (default: cuda:0)        │
     │ -v, --verbose, --no-verbose                                                                  │
     │     If set, show verbose debug messages. (default: False)                                    │
     │ -o PATH, --out-path PATH                                                                     │
-    │     Path to save the output PLY file. Defaults to `out.ply` in the current working           │
-    │     directory. Path must end in .ply, .usdc, or .usdz. (default: out_resumed.ply)            │
+    │     Output product path. The default is selected by the checkpoint method. (default: None)   │
+    │ -r {None}|PATH, --reconstruction-path {None}|PATH                                            │
+    │     Override a reconstruction referenced by a derived-product checkpoint. (default: None)   │
     ╰──────────────────────────────────────────────────────────────────────────────────────────────╯
     ╭─ io options ─────────────────────────────────────────────────────────────────────────────────╮
     │ Configure saving and logging metrics, images, and checkpoints.                               │

docs/api/frgs/show.rst

@@ -2,8 +2,8 @@
 
     usage: frgs show [-h] [SHOW OPTIONS] PATH
 
-    Visualize a Gaussian splat radiance field in a saved PLY or checkpoint file. This will plot the
-    splats in an interactive viewer shown in a browser window.
+    Visualize a saved Reality Capture product. Supported inputs are Gaussian PLY/checkpoint files
+    and portable GARfVDB bundle directories.
 
     # Example usage:
 
@@ -14,8 +14,7 @@
         frgs show model.pt --viewer-port 8888
 
     ╭─ positional arguments ───────────────────────────────────────────────────────────────────────╮
-    │ PATH                    Path to the input PLY or checkpoint file. Must end in .ply, .pt, or  │
-    │                         .pth. (required)                                                     │
+    │ PATH                    Path to a PLY, checkpoint, or .garfvdb bundle. (required)            │
     ╰──────────────────────────────────────────────────────────────────────────────────────────────╯
     ╭─ options ────────────────────────────────────────────────────────────────────────────────────╮
     │ -h, --help              show this help message and exit                                      │
@@ -25,6 +24,5 @@
     │                         The port to expose the viewer server on. (default: 127.0.0.1)        │
     │ -v, --verbose, --no-verbose                                                                  │
     │                         If True, then the viewer will log verbosely. (default: False)        │
-    │ --device STR|DEVICE     Device to use for computation (default is "cuda"). (default: cuda)   │
+    │ --device STR|DEVICE     Device to use for computation (default is "cuda:0"). (default: cuda:0) │
     ╰──────────────────────────────────────────────────────────────────────────────────────────────╯
-

docs/api/instance_segmentation.rst

@@ -0,0 +1,39 @@
+Instance segmentation
+=====================
+
+Portable bundle format
+----------------------
+
+``.garfvdb`` manifests use the schema name ``fvdb_reality_capture.garfvdb`` and an integer ``schema_version``.
+The current version is available as :data:`ARTIFACT_SCHEMA_VERSION`. Loaders select an exact version-specific
+reader and raise :class:`GARfVDBArtifactVersionError` for unsupported versions.
+
+GARfVDB
+-------
+
+.. currentmodule:: fvdb_reality_capture.instance_segmentation
+
+.. autoclass:: GARfVDB
+   :members:
+
+.. autoclass:: GARfVDBTrainer
+   :members: new, from_state_dict, from_checkpoint_file, train, to_product, state_dict
+
+.. autoclass:: GARfVDBConfig
+   :members:
+
+.. autoclass:: GARfVDBTrainingConfig
+   :members:
+
+.. autoclass:: GARfVDBTransformConfig
+   :members:
+
+.. autoclass:: GARfVDBMaskAttribute
+   :members:
+
+.. autoclass:: GenerateGARfVDBMasks
+   :members:
+
+.. autoexception:: GARfVDBArtifactError
+
+.. autoexception:: GARfVDBArtifactVersionError

docs/api/transforms.rst

@@ -35,4 +35,7 @@
 
 .. autoclass:: fvdb_reality_capture.transforms.Identity
     :members:
+
+.. autoclass:: fvdb_reality_capture.transforms.SceneTransformConfig
+    :members:
     :special-members: __init__, __call__

docs/index.rst

@@ -55,6 +55,7 @@ Features
 - Extraction of high-quality meshes and point clouds from reconstructed radiance fields, with support for various output
   formats.
 - Visualization tools for inspecting sensor data, radiance fields, and reconstruction results.
+- Scale-conditioned 3D instance feature fields using GARfVDB, stored as portable NanoVDB-based artifacts.
 - Extensible design that allows users to easily add new functionality and customize existing components.
 - Comprehensive documentation and tutorials to help users get started quickly.
 - Automatic scaling to multiple GPUs in a single machine for large-scale reconstructions
@@ -96,6 +97,7 @@ A common reality capture pipeline typically resembles the figure below:
    tutorials/sensor_data_loading_and_manipulation
    tutorials/radiance_field_and_mesh_reconstruction
    tutorials/frgs
+   tutorials/instance_segmentation
 
 .. toctree::
    :maxdepth: 1
@@ -107,7 +109,9 @@ A common reality capture pipeline typically resembles the figure below:
    api/sfm_scene
    api/tools
    api/transforms
+   api/checkpoints
    api/frgs
+   api/instance_segmentation
 
 .. toctree::
    :maxdepth: 1

docs/requirements.txt

@@ -21,6 +21,7 @@ opencv-python-headless
 pycolmap>=3.11
 pyproj
 requests
+safetensors
 scikit-image
 scipy
 torch==2.8.0+cpu