Skip to content

Commit a38ea5d

Browse files
author
dukaijie
committed
docs: add DISCOVERSE 3DGS scene tutorial + AndroidTwin 4-cam rollout demo
- scripts/data_collection/download_discoverse_scenes.sh: pull packed PLY from tatp/DISCOVERSE-models on HF, decompress SuperSplat -> standard 3DGS layout consumed by GaussianSensor. Default lab3, --all for the four indoor scenes (lab3, flower_table, discover_operation_studio, tsimf_library_0). Assets remain gitignored - this script is the only source of truth. - scripts/data_collection/decompress_supersplat.py: PlayCanvas packed PLY -> x/y/z + f_dc_* + scale_* + rot_* + opacity. Required by the download script (sibling lookup). - docs/discoverse_scenes.md: scene catalog, why-unpack rationale, render example, four non-obvious gotchas (don't render packed, floor 5th-pct, GS != MuJoCo intrinsics, GS coverage limited to training-cam hull). - README.md: new showcase "DISCOVERSE 3DGS rooms x AndroidTwin multi-cam rollout" right after the kitchen demo. Documents the single shared world_T_gs alignment + the OpenGL->OpenCV Y-flip required for level-horizon world cams. - docs/media/discoverse_lab3/{4cam_composite,head_cam,world_cam, world_back_cam,world_side_cam}.mp4: 920 KB total demo clips.
1 parent 9ba015d commit a38ea5d

9 files changed

Lines changed: 468 additions & 0 deletions

File tree

README.md

Lines changed: 53 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -95,6 +95,59 @@ uv run at-eval \
9595
(`MuGSRecorder`) lives at `androidtwin/envs/mugs_recorder.py` in the
9696
AndroidTwin repo and uses MuGS's standalone `GaussianSensor` API.
9797

98+
### DISCOVERSE 3DGS rooms × AndroidTwin multi-cam rollout
99+
100+
Four cameras, one rollout, one shared `world_T_gs` rigid alignment. The
101+
G1 + Inspire FTP stands via AMO inside the DISCOVERSE `lab3` 3DGS room;
102+
each frame is captured from the robot's first-person `head_cam` plus
103+
three world cams (front-diagonal, over-the-shoulder, low side hero).
104+
The composite below stitches all four panes — every panel is a hybrid
105+
MuGS render (3DGS bg + MuJoCo fg, alpha-blended per camera).
106+
107+
[![4-cam DISCOVERSE lab3 composite](docs/media/discoverse_lab3/4cam_composite.mp4)](docs/media/discoverse_lab3/4cam_composite.mp4)
108+
109+
| Camera | View |
110+
|---|---|
111+
| `robot/head_cam` (first-person, D435 mount, 47° down, 42° vfov) | [head_cam.mp4](docs/media/discoverse_lab3/head_cam.mp4) |
112+
| `world_cam` (front-diagonal) | [world_cam.mp4](docs/media/discoverse_lab3/world_cam.mp4) |
113+
| `world_back_cam` (over-the-shoulder back) | [world_back_cam.mp4](docs/media/discoverse_lab3/world_back_cam.mp4) |
114+
| `world_side_cam` (low hero shot) | [world_side_cam.mp4](docs/media/discoverse_lab3/world_side_cam.mp4) |
115+
116+
Two pieces of plumbing make this work compared to the kitchen demo
117+
above:
118+
119+
- **One alignment for all cams.** `MuGSRecorder` accepts a `world_T_gs`
120+
4×4 transform that maps MJ-world → GS-world. Every simultaneous camera
121+
applies the same transform to its live MJ pose, so the four views see
122+
the same room from coherent perspectives without per-cam initial-pose
123+
snapping.
124+
- **OpenGL → OpenCV Y-flip.** gsplat is OpenCV (+Y down). MuGS internally
125+
negates column 2 of the cam rotation (Z forward), so the recorder
126+
pre-negates column 1 in the `world_T_gs` branch. Without this, every
127+
level-horizon world cam renders upside-down; the first-person head cam
128+
survives unflipped only because its 47° pitch coincidentally lines up.
129+
130+
Get the scene:
131+
132+
```bash
133+
bash scripts/data_collection/download_discoverse_scenes.sh # default: lab3
134+
```
135+
136+
Render the demo (from the AndroidTwin repo, scene path adjusted):
137+
138+
```bash
139+
.venv/bin/python examples/discoverse_room_rollout.py \
140+
--bg-ply /path/to/MuGS/assets/scenes/discoverse_unpacked/lab3/point_cloud.ply \
141+
--out-dir outputs/discoverse_lab3 \
142+
--num-steps 120 --width 480 --height 360 \
143+
--gs-fx 380 --gs-fy 380 --yaw-deg 180
144+
```
145+
146+
See [`docs/discoverse_scenes.md`](docs/discoverse_scenes.md) for the
147+
scene catalog, alignment knobs (`yaw_deg`, GS focal, floor 5%-percentile,
148+
bbox-center vs median), and the training-cam convex-hull caveat that
149+
makes some camera placements render dim/noisy.
150+
98151
## Features
99152

100153
### Core Rendering

docs/discoverse_scenes.md

Lines changed: 97 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,97 @@
1+
# DISCOVERSE 3DGS scenes
2+
3+
How to fetch and render the DISCOVERSE indoor 3DGS captures used by the
4+
multi-camera AndroidTwin demo (see [the showcase
5+
section](../README.md#discoverse-3dgs-rooms--androidtwin-multi-cam-rollout)
6+
of the top-level README).
7+
8+
No scenes are tracked in this repo — `assets/scenes/*.ply` is `.gitignore`d.
9+
The download script is the only path.
10+
11+
## What you get
12+
13+
Four indoor 3DGS captures from the DISCOVERSE project:
14+
15+
| Scene | Description | Packed (HF) | Unpacked |
16+
|-----------------------------|--------------------------------------|------------:|---------:|
17+
| `lab3` | Robotics lab corner (cabinet, desk) | ~5.5 MB | ~23 MB |
18+
| `flower_table` | Indoor scene with a flower table | ~7.9 MB | ~33 MB |
19+
| `discover_operation_studio` | Operation studio room | ~8.4 MB | ~36 MB |
20+
| `tsimf_library_0` | Library reading area | ~22 MB | ~92 MB |
21+
22+
Source: [`tatp/DISCOVERSE-models`](https://huggingface.co/tatp/DISCOVERSE-models)
23+
on Hugging Face, paths under `3dgs/scene/<name>/point_cloud.ply`.
24+
25+
## Why the unpack step
26+
27+
DISCOVERSE ships scenes in PlayCanvas **SuperSplat compressed** PLY format
28+
(chunked uint32 packed fields), not the official 3DGS schema. MuGS's
29+
`GaussianSensor` only consumes the unpacked layout (`x/y/z`, `f_dc_*`,
30+
`scale_*`, `rot_*`, `opacity`), so the script runs
31+
`decompress_supersplat.py` once per scene right after download.
32+
33+
## Quick start
34+
35+
```bash
36+
# default: lab3 only
37+
bash scripts/data_collection/download_discoverse_scenes.sh
38+
39+
# everything
40+
bash scripts/data_collection/download_discoverse_scenes.sh --all
41+
42+
# pick a subset
43+
bash scripts/data_collection/download_discoverse_scenes.sh lab3 flower_table
44+
```
45+
46+
Outputs land in:
47+
48+
```
49+
assets/scenes/discoverse/<scene>/point_cloud.ply # packed (archival)
50+
assets/scenes/discoverse_unpacked/<scene>/point_cloud.ply # unpacked (renderable)
51+
```
52+
53+
Override with `--dst-root <dir>` or env `DISCOVERSE_SCENE_ROOT=…`. The
54+
unpacked directory always appends `_unpacked` to the chosen root.
55+
56+
## Dependencies
57+
58+
- `curl` (or `wget`) for download.
59+
- `python3` with `numpy` and `plyfile` for the decompressor.
60+
61+
## Render with MuGS
62+
63+
```python
64+
from mugs.sensors import GaussianSensor, GaussianSensorConfig
65+
66+
cfg = GaussianSensorConfig(
67+
width=480,
68+
height=360,
69+
background_ply_path="assets/scenes/discoverse_unpacked/lab3/point_cloud.ply",
70+
render_mode="3dgs_only", # or "hybrid" with a MuJoCo foreground
71+
)
72+
sensor = GaussianSensor(cfg)
73+
```
74+
75+
Pair with `MuGSRecorder` (in the AndroidTwin repo) for a full
76+
hybrid 3DGS + MuJoCo + MuJoCo-cam-tracking pipeline.
77+
78+
## Gotchas worth knowing once
79+
80+
1. **Don't render the packed PLY directly.** `GaussianSensor` will load it
81+
and silently render noise. Always go through the unpack step.
82+
2. **Use `bbox.center` for xy, `percentile(z, 5)` for floor.** COLMAP
83+
leaves a small number of sub-floor splats that drag `bbox.min[2]` ~1 m
84+
below the actual floor; the 5th-percentile reading is robust. The
85+
median xy is biased toward the densest splat cluster (e.g. lab3
86+
`median_y=+0.72` vs `bbox_center_y=+0.20`) which can push virtual
87+
cameras outside the training-cam convex hull — use the bbox center.
88+
3. **`gs_intrinsics ≠ MuJoCo intrinsics`.** Feed the GS scene's training
89+
focal length (lab3 ≈ 380 px at 480×360) to the renderer, not the
90+
MuJoCo camera's own `fx/fy`. A mismatch shows up as squashed or
91+
stretched backgrounds.
92+
4. **GS coverage ≠ scene bbox.** Each capture only renders cleanly
93+
*inside the convex hull of the training cameras*. lab3's training
94+
cams are biased toward the GS `-X` half — virtual cameras placed in
95+
the `+X` half will render dim/blurry noise. Use AndroidTwin's
96+
`examples/dump_cam_params.py` to isolate a single camera's GS-only
97+
render before composing a full multi-cam rollout.
443 KB
Binary file not shown.
51.7 KB
Binary file not shown.
113 KB
Binary file not shown.
89.9 KB
Binary file not shown.
197 KB
Binary file not shown.
Lines changed: 196 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,196 @@
1+
"""Decompress SuperSplat / PlayCanvas packed 3DGS PLY → official 3DGS PLY.
2+
3+
DISCOVERSE assets are stored in the PlayCanvas compressed format
4+
(`element chunk` + `element vertex` with uint32 packed_* fields).
5+
This script converts a single file or directory of such PLYs into
6+
the standard 3DGS layout (`x/y/z`, `f_dc_*`, `scale_*`, `rot_*`,
7+
`opacity`) that `mugs.sensors.GaussianSensor._load_official_ply`
8+
already understands.
9+
10+
Bit layout (per playcanvas engine `gsplat-compressed-data.js`):
11+
packed_position : uint32 11/10/11 bits → lerp(chunk min,max)
12+
packed_scale : uint32 11/10/11 bits → lerp(chunk min,max) (log-space)
13+
packed_rotation : uint32 2/10/10/10 → smallest-three quat
14+
packed_color : uint32 8/8/8/8 → R,G,B,A in [0,1]
15+
"""
16+
17+
from __future__ import annotations
18+
19+
import argparse
20+
import math
21+
from pathlib import Path
22+
23+
import numpy as np
24+
from plyfile import PlyData, PlyElement
25+
26+
27+
SH_C0 = 0.28209479177387814
28+
29+
30+
def _unpack_unorm(values: np.ndarray, shift: int, bits: int) -> np.ndarray:
31+
mask = (1 << bits) - 1
32+
return ((values >> shift) & mask).astype(np.float64) / float(mask)
33+
34+
35+
def _smallest_three_quat(packed: np.ndarray) -> np.ndarray:
36+
sel = (packed >> 30) & 0x3
37+
a = _unpack_unorm(packed, 20, 10) - 0.5
38+
b = _unpack_unorm(packed, 10, 10) - 0.5
39+
c = _unpack_unorm(packed, 0, 10) - 0.5
40+
s = math.sqrt(2.0)
41+
a *= s
42+
b *= s
43+
c *= s
44+
d2 = 1.0 - (a * a + b * b + c * c)
45+
d = np.sqrt(np.clip(d2, 0.0, None))
46+
47+
quats = np.zeros((packed.shape[0], 4), dtype=np.float64)
48+
# Convention used by playcanvas engine: ordering [x, y, z, w], the
49+
# selector indexes the *largest* magnitude component. Official 3DGS
50+
# PLYs store rot_0..rot_3 as [w, x, y, z], so map accordingly.
51+
# playcanvas index 0->x, 1->y, 2->z, 3->w --> official wxyz idx
52+
pc_to_wxyz = np.array([1, 2, 3, 0])
53+
for pc_i in range(4):
54+
mask = sel == pc_i
55+
if not mask.any():
56+
continue
57+
out_idx = pc_to_wxyz[pc_i]
58+
quats[mask, out_idx] = d[mask]
59+
remaining_pc = [j for j in range(4) if j != pc_i]
60+
remaining_out = pc_to_wxyz[remaining_pc]
61+
quats[mask, remaining_out[0]] = a[mask]
62+
quats[mask, remaining_out[1]] = b[mask]
63+
quats[mask, remaining_out[2]] = c[mask]
64+
65+
quats /= np.linalg.norm(quats, axis=1, keepdims=True) + 1e-12
66+
return quats
67+
68+
69+
def decompress_supersplat(plydata: PlyData) -> dict[str, np.ndarray]:
70+
if "chunk" not in {e.name for e in plydata.elements}:
71+
raise ValueError("not a compressed SuperSplat PLY (missing 'chunk' element)")
72+
73+
chunks = plydata["chunk"]
74+
verts = plydata["vertex"]
75+
n = len(verts)
76+
chunk_size = 256
77+
chunk_idx = np.minimum(np.arange(n) // chunk_size, len(chunks) - 1)
78+
79+
def per_vertex(field: str) -> np.ndarray:
80+
return chunks[field][chunk_idx]
81+
82+
pmin = np.stack([per_vertex("min_x"), per_vertex("min_y"), per_vertex("min_z")], 1)
83+
pmax = np.stack([per_vertex("max_x"), per_vertex("max_y"), per_vertex("max_z")], 1)
84+
smin = np.stack(
85+
[per_vertex("min_scale_x"), per_vertex("min_scale_y"), per_vertex("min_scale_z")],
86+
1,
87+
)
88+
smax = np.stack(
89+
[per_vertex("max_scale_x"), per_vertex("max_scale_y"), per_vertex("max_scale_z")],
90+
1,
91+
)
92+
93+
pp = verts["packed_position"].astype(np.uint32)
94+
pn = np.stack(
95+
[_unpack_unorm(pp, 21, 11), _unpack_unorm(pp, 11, 10), _unpack_unorm(pp, 0, 11)],
96+
1,
97+
)
98+
positions = pmin + pn * (pmax - pmin)
99+
100+
ps = verts["packed_scale"].astype(np.uint32)
101+
sn = np.stack(
102+
[_unpack_unorm(ps, 21, 11), _unpack_unorm(ps, 11, 10), _unpack_unorm(ps, 0, 11)],
103+
1,
104+
)
105+
log_scales = smin + sn * (smax - smin)
106+
107+
quats_wxyz = _smallest_three_quat(verts["packed_rotation"].astype(np.uint32))
108+
109+
pc = verts["packed_color"].astype(np.uint32)
110+
r = _unpack_unorm(pc, 24, 8)
111+
g = _unpack_unorm(pc, 16, 8)
112+
b = _unpack_unorm(pc, 8, 8)
113+
alpha = _unpack_unorm(pc, 0, 8)
114+
color_bound_fields = chunks.data.dtype.names
115+
if "min_r" in color_bound_fields:
116+
rmin = per_vertex("min_r")
117+
rmax = per_vertex("max_r")
118+
gmin = per_vertex("min_g")
119+
gmax = per_vertex("max_g")
120+
bmin = per_vertex("min_b")
121+
bmax = per_vertex("max_b")
122+
r = rmin + r * (rmax - rmin)
123+
g = gmin + g * (gmax - gmin)
124+
b = bmin + b * (bmax - bmin)
125+
colors_rgb = np.stack([r, g, b], 1)
126+
colors_rgb = np.clip(colors_rgb, 1e-6, 1.0 - 1e-6)
127+
alpha = np.clip(alpha, 1e-6, 1.0 - 1e-6)
128+
129+
# Convert color back to SH DC (the official format stores SH DC,
130+
# then renderer does 0.5 + SH_C0 * dc). Inverse:
131+
f_dc = (colors_rgb - 0.5) / SH_C0
132+
# Opacity is stored pre-sigmoid in official format; renderer does sigmoid.
133+
opacity_pre = np.log(alpha / (1.0 - alpha))
134+
135+
return {
136+
"xyz": positions.astype(np.float32),
137+
"f_dc": f_dc.astype(np.float32),
138+
"scales_log": log_scales.astype(np.float32),
139+
"rot_wxyz": quats_wxyz.astype(np.float32),
140+
"opacity_pre": opacity_pre.astype(np.float32),
141+
}
142+
143+
144+
def write_official_ply(out_path: Path, gaussians: dict[str, np.ndarray]) -> None:
145+
n = gaussians["xyz"].shape[0]
146+
dtype = [
147+
("x", "f4"), ("y", "f4"), ("z", "f4"),
148+
("nx", "f4"), ("ny", "f4"), ("nz", "f4"),
149+
("f_dc_0", "f4"), ("f_dc_1", "f4"), ("f_dc_2", "f4"),
150+
("opacity", "f4"),
151+
("scale_0", "f4"), ("scale_1", "f4"), ("scale_2", "f4"),
152+
("rot_0", "f4"), ("rot_1", "f4"), ("rot_2", "f4"), ("rot_3", "f4"),
153+
]
154+
arr = np.zeros(n, dtype=dtype)
155+
arr["x"], arr["y"], arr["z"] = gaussians["xyz"].T
156+
arr["f_dc_0"], arr["f_dc_1"], arr["f_dc_2"] = gaussians["f_dc"].T
157+
arr["opacity"] = gaussians["opacity_pre"]
158+
arr["scale_0"], arr["scale_1"], arr["scale_2"] = gaussians["scales_log"].T
159+
arr["rot_0"], arr["rot_1"], arr["rot_2"], arr["rot_3"] = gaussians["rot_wxyz"].T
160+
el = PlyElement.describe(arr, "vertex")
161+
PlyData([el], byte_order="<").write(str(out_path))
162+
163+
164+
def convert_file(src: Path, dst: Path) -> dict:
165+
src = src.resolve()
166+
plydata = PlyData.read(src)
167+
g = decompress_supersplat(plydata)
168+
dst.parent.mkdir(parents=True, exist_ok=True)
169+
write_official_ply(dst, g)
170+
bbox = g["xyz"].max(0) - g["xyz"].min(0)
171+
return {
172+
"src": str(src),
173+
"dst": str(dst),
174+
"n": g["xyz"].shape[0],
175+
"bbox_m": tuple(round(float(x), 3) for x in bbox),
176+
"center_m": tuple(round(float(x), 3) for x in g["xyz"].mean(0)),
177+
}
178+
179+
180+
def main() -> None:
181+
ap = argparse.ArgumentParser()
182+
ap.add_argument("src", type=Path, help="Compressed ply file or directory")
183+
ap.add_argument("dst", type=Path, help="Output ply file or directory")
184+
args = ap.parse_args()
185+
186+
if args.src.is_dir():
187+
for ply in sorted(args.src.glob("*.ply")):
188+
info = convert_file(ply, args.dst / ply.name)
189+
print(info)
190+
else:
191+
info = convert_file(args.src, args.dst)
192+
print(info)
193+
194+
195+
if __name__ == "__main__":
196+
main()

0 commit comments

Comments
 (0)