Sim(3) registration of 3D line maps across sensor modalities. ScalePluckerNet
is a scale-aware Plücker line matcher trained on fully synthetic, SLAM-calibrated
Sim(3) data; paired with the line-based Sim(3) estimator in lib/sim3_solver.py
it registers a scale-ambiguous monocular SLAM map onto a metric RGB-D or LiDAR
line map — recovering scale, rotation and translation from line geometry alone.
Explanation, theory, and full results live on the project page. A runnable Colab demo notebook registers a real bundled 7-Scenes map pair end-to-end on CPU. Both live in the parent SCALAR repo; this README covers only the codebase.
The encoder is a single affine-Grassmannian branch. Every line becomes a
2-plane in R^4; vec(P) is the 10 unique entries of the projector P = Y Yᵀ
with √2 on the off-diagonals, so Euclidean distance on vec(P) is the
chordal Grassmann distance (GraffMatch Eq. 3-4, Lusk et al. RA-L 2022).
per line i, over its k=10 nearest neighbours j in Graff(1,3):
edge_ij = cat(vec(P)_j − vec(P)_i, vec(P)_i) 20-D
node_i = MLP( mean_j Conv2d(edge_ij) ) 20→32→128
then 12 alternating self/cross attention layers (128 ch, 4 heads)
then L2-normalised descriptors → pairwise L2 → Sinkhorn (balanced, μ=0.1, 30 it)
There is no sign handling in the matcher, and none is needed. Flipping
[m;d] → [−m;−d] is the same line: it leaves p0 = d × m untouched and only
sends c1 → −c1, so the subspace — and hence P — is unchanged. Measured
exactly 0.000e+00 under random per-line flips. The k-NN graph and the
equivariant origin are built from foot points, which are sign-invariant too.
(The ambiguity is real: 47.3% of true cross-modal matches carry opposite Plücker
sign. A max-|component| hemisphere canon reduced that to a 7% residual seam; the
Grassmannian removes it by construction.)
There is no σ normalisation. Per-cloud σ is a statistic that assumes both clouds cover comparable content — measured 21.5% error at full overlap, 35.8% at 1/8 crop. The matcher's only normalisation is a moment whitening by the query moment std, which is invariant to the choice of world units while preserving the query/reference scale ratio.
sign_inv (sign-even embedding), dual_knn, geo_edge/geo_knn/graff_knn,
graph wavelets, geometric attention bias, 6 GNN layers, max/min pooling,
separate node+edge branches, explicit principal-angle edge channels, the
SuperGlue dustbin, and a size-adaptive k. Each was measured neutral or worse;
the verdicts and numbers are in lib/model.py and the parent repo's CLAUDE.md.
conda activate torch5090 # Python 3.11, PyTorch 2.6, CUDA
pip install easydict msgpack # + numpy, torch, matplotlibSelf-contained: network and solver both live in lib/; no external service or
dataset is required to register two maps.
lib/model.py the matcher — ONE encoder, no architecture flags
lib/sim3_solver.py THE Sim(3) estimator, one self-contained file
lib/{trainer,loss,dataloader,utils}.py training machinery
generate_synthetic.py calibrated Sim(3) pair generator
train.py training entry point
(dataset flags + scenario table now live in the parent repo:
docs/scalepluckernet_dataset_flags.md)
before regenerating or comparing against a checkpoint
output/ checkpoints
from lib.sim3_solver import Sim3Solver, solve_sim3
solver = Sim3Solver((q1, q2), (r1, r2)) # (N,3) segment endpoints each
p_q, p_r = solver.matcher_input() # what the network consumes
R, t, s = solve_sim3(solver, prob) # prob = matcher output# 1) generate data -- FLAGS MATTER, see ../../docs/scalepluckernet_dataset_flags.md
FOUND=1 INDOOR=1 MONO=1 DEALIAS=1 python generate_synthetic.py \
--name synthetic_found8 --n_train 300000 --n_valid 3000 --workers 12
# 2) train (no architecture flags -- there is only one configuration)
python train.py --dataset synthetic_found8 --bucket_batch --match_w 0.2 \
--batch 1 --iter_size 32 --lr 5e-4 --gamma 0.99 \
--train_epoch_size 32000 --workers 4 --name my_run
# resume (restores optimizer + schedule):
python train.py ... --resume output/synthetic_found8/my_run/checkpoint.pth
# optional: generate pairs on the fly instead of loading a .pkl dataset
python train.py ... --live --live_workers 6 --workers 0--live works but was measured to buy nothing: infinite unique pairs climb
at the same rate as the fixed 300k set (+0.0115 vs +0.0128 val/epoch) and score
the same on the real benchmark, at ~48% slower epochs. 300k is past the point
where repetition matters.
The scenario mix is a declarative table (_SPEC in generate_synthetic.py),
not a chain of hand-written branches: each entry names a STRUCTURE (which line
pool) and a REGIME (sizes, overlap, noise). The street constants are fitted to
measured KITTI statistics -- do not "tidy" them.
SOUND=1 (opt-in; SOUND=0 reproduces synthetic_found8 exactly, same RNG
stream) fixes two measured coverage defects. found8 put only 8.0% of its
pairs in the 7-Scenes operating regime because uniform-in-angle rotation has
median 90 deg while the benchmarks sit at 1-29 deg, and because the indoor
scenarios drew scale log-uniform over 6 octaves. SOUND replaces the rotation
law with a MIXTURE (half uniform-angle, half log-uniform) -- not a cap;
v19 capped at 90 deg and got 4.9% inlier retention at 180 deg vs v41's 91.2%.
Support is unchanged: still full SO(3), still 8.8% of pairs above 150 deg.
regime found8 SOUND
7-Scenes (s 1.19-3.06, rot<=58) 8.0% 15.5%
KITTI mono_best (s 8-42, rot<=12) 10.4% 14.7%
median rotation 80.0 32.3 deg
LOCAL=1 (opt-in) makes SPATIAL LOCALITY a regime axis. Measured: only
street_submap concentrated its correspondences (0.28 of the reference
radius); the generic submap scenario was 0.98, i.e. scattered -- it modelled
a submap as fewer query lines, not as a localized region of a bigger cloud.
LOCAL lets any pool be drawn windowed (LOCAL_P=0.25) with the reference
forced 2.5-8x the query, taking submap-regime pairs from 7.8% to 28.8% of the
mix across all structure types rather than streets only.
The generator COVERS the deployment regimes; it is never CALIBRATED to them.
The claim is zero-shot, so measuring a target dataset's regime is a legitimate
check that the cell is populated, but must not become a loop that reshapes the
bands to fit it. See ../../docs/scalepluckernet_dataset_flags.md.
TensorBoard is always written to output/<dataset>/<run>/. wandb needs a
separate mirror process -- train.py's own WANDB=1 path is a no-op here
because wandb is not installed in the torch5090 env and this directory's
wandb/ shadows the import anyway:
# live: holds ONE run open, run stays "running" until the trainer exits
python tools/wandb_live.py <run-name> --match "[t]rain\.py --dataset <dataset>"
# finished runs: one-shot snapshot, updates the same run id in place
python tools/wandb_backfill.py <run-name>Both live in the PARENT repo's tools/ and must run under base python from
the parent repo root, never from this directory. Run ids are deterministic
(md5 of the name) so re-running updates in place; a .wandb_id file in the run
directory overrides that, which is the only recovery if a run was ever deleted
(wandb permanently burns a deleted id).
--train_epoch_size is NOT saved in the checkpoint config. Its absence
proves nothing. Recover the real epoch size from the TensorBoard step range:
train/* starts at -(iters_per_epoch). Runs here use 1000 iters/epoch
(32000 pairs); omitting the flag silently gives 9375 and changes both
epoch-for-epoch comparability and the per-epoch LR decay.
Checkpoint selection: never use trainer val. It has disagreed with the real benchmark five times in this project — including one arm 4.3 points behind on val that tied on the benchmark, and a +0.66 val gain that scored slightly worse. Select on the benchmarks below.
# cache the matcher forward once, then A/B solvers on the same cache
python tools/cache_matcher.py <ckpt> cache_7s.pkl --dataset 7scenes
python tools/full_results.py # the main results table
python tools/judge_ckpt.py <ckpt> # real @5 + alias auditMatcher forward passes are GPU-numerics-sensitive and do not reproduce bit-exactly across sessions — cache the probabilities once and compare solvers on the same cache. RANSAC seed noise is ±2 pairs on 7-Scenes, so report mean±std over ≥5 seeds; 3-seed "bests" have collapsed under re-measurement four separate times.
All code here uses [m, d] order (moment first); the original PlueckerNet uses
[d, m]. Sim(3) law: d' = R d, m' = s·R·m + t × d'. Lines are sign-ambiguous
([m,d] ~ [−m,−d]). The matcher is invariant by construction (above). The
solver handles it explicitly and still needs to: it canonicalises query
line 1 and proposes both signs of line 2, letting the G(2,4) criterion
arbitrate — a residual-based sign test is provably blind at 90°, since flipping
line 2 maps the inter-line angle θ → 180−θ (measured 66.3% agreement in the
80–90° bin versus 100% at 30–60°).