Is there any suggestions to train large scene_scale dataset with MCMC?

[a11enL]

gsplat 1.4.0

python3 examples/simple_trainer.py mcmc --use_bilateral_grid --data_factor 1 --data_dir data/test123/ --result_dir exports/test123/

part of logs

...
[Parser] 120 images, taken by 120 cameras.
Scene scale: 2009.6237303032315
Model initialized. Number of GS: 159431
...
loss=0.475| sh degree=0| : 100%|██▋| 299/30000 [00:25< ......
......
loss=0.185| sh degree=3| : 25%|██████████████████████▌ | 7600/30000 [13:15<39:05, 9.55it/s]
Traceback (most recent call last):
File "/home/ubuntu/gsplat-1.4.0/examples/simple_trainer.py", line 1120, in
cli(main, cfg, verbose=True)
File "/home/ubuntu/gsplat-1.4.0/gsplat/distributed.py", line 360, in cli
return _distributed_worker(0, 1, fn=fn, args=args)
File "/home/ubuntu/gsplat-1.4.0/gsplat/distributed.py", line 295, in _distributed_worker
fn(local_rank, world_rank, world_size, args)
File "/home/ubuntu/gsplat-1.4.0/examples/simple_trainer.py", line 1065, in main
runner.train()
File "/home/ubuntu/gsplat-1.4.0/examples/simple_trainer.py", line 820, in train
self.cfg.strategy.step_post_backward(
File "/home/ubuntu/gsplat-1.4.0/gsplat/strategy/mcmc.py", line 128, in step_post_backward
n_relocated_gs = self._relocate_gs(params, optimizers, binoms)
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/gsplat-1.4.0/gsplat/strategy/mcmc.py", line 158, in _relocate_gs
relocate(
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/gsplat-1.4.0/gsplat/strategy/ops.py", line 278, in relocate
sampled_idxs = _multinomial_sample(probs, n, replacement=True)
File "/home/ubuntu/.local/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 116, in decorate_context
return func(*args, **kwargs)
File "/home/ubuntu/gsplat-1.4.0/gsplat/strategy/ops.py", line 31, in _multinomial_sample
assert not num_elements == 0, ('_multinomial_sample weights 0')
AssertionError: _multinomial_sample weights 0

It seems gsplat will crash or generate a nothing but noise ply file if change opacity_reg to 0.001, when dataset scene_scale is larger than 2000 or 10000. all of cases is running with high traning loss. I have three datasets from AI tool with this situation here, both these datasets work well with colmap without any errors or warnings. and I nerver encounter such situation with traditional colmap dataset. thanks.

[a11enL]

Current the problem is that loss doesn't converge, it stopped on 0.1 or 0.2.
I have tried noise-lr, opacity_reg, scale_reg and filter theose zero-alive GS cases etc.

This is the dataset with above situation

120 images
120 cameras
159431 points
Source: colmap image_undistorter

download, 183M, Google Drive
https://drive.google.com/file/d/1pn8b74AAGobQI-Bxv6arLA-B6-l8IIWQ/view?usp=sharing