[Question/Bug] Assertion `channels == 3` prevents rendering RGB+Depth (4 channels) in `with_eval3d` + `with_ut` mode

[vairleon]

I noticed that in Rasterization.cpp, inside the rasterize_to_pixels_from_world_3dgs_fwd function, there is a hard assertion at Line 744 enforcing 3 channels:

// Rasterization.cpp Line 744
assert (channels == 3); // only support RGB for now

I am currently attempting to render RGB + Depth, which results in channels = 4. Consequently, this assertion fails and causes the program to crash. This occurs when using the configuration: with_eval3d + with_ut ： python: /home/lenovo/Projects/gaussians/honor_vv/submodules/gsplat/gsplat/cuda/csrc/Rasterization.cpp:974: std::tuple<at::Tensor, at::Tensor, at::Tensor> gsplat::rasterize_to_pixels_from_world_3dgs_fwd(at::Tensor, at::Tensor, at::Tensor, at::Tensor, at::Tensor, std::optionalat::Tensor, std::optionalat::Tensor, uint32_t, uint32_t, uint32_t, at::Tensor, std::optionalat::Tensor, at::Tensor, CameraModelType, UnscentedTransformParameters, ShutterType, std::optionalat::Tensor, std::optionalat::Tensor, std::optionalat::Tensor, FThetaCameraDistortionParameters, at::Tensor, at::Tensor): Assertion `channels == 3' failed.
Aborted (core dumped)

Is there a specific technical reason why we strictly assert channels == 3 here? If the downstream logic supports it, is it safe to remove this assertion or change it to channels >= 3 to allow for Depth or other attributes?
Any insights would be appreciated. Thanks!

[marslicy]

According to their document, if you want to render D dimensional feature, you should set the sh degree as None.
https://docs.gsplat.studio/main/apis/rasterization.html

Support N-D Features: If sh_degree is None, the colors is expected to be with shape […, N, D] or […, C, N, D], in which D is the channel of the features to be rendered. The computation is slow when D > 32 at the moment. If sh_degree is set, the colors is expected to be the SH coefficients with shape […, N, K, 3] or […, C, N, K, 3], where K is the number of SH bases. In this case, it is expected that , where sh_degree controls the activated bases in the SH coefficients.

[vairleon]

According to their document, if you want to render D dimensional feature, you should set the sh degree as None. https://docs.gsplat.studio/main/apis/rasterization.html

Support N-D Features: If sh_degree is None, the colors is expected to be with shape […, N, D] or […, C, N, D], in which D is the channel of the features to be rendered. The computation is slow when D > 32 at the moment. If sh_degree is set, the colors is expected to be the SH coefficients with shape […, N, K, 3] or […, C, N, K, 3], where K is the number of SH bases. In this case, it is expected that , where sh_degree controls the activated bases in the SH coefficients.

Thanks for your reply. I think they have dealt with the 'SH+Depth' mode inside the rasterization function when you set the mode to 'RGB+D'. I'm using their given render mode instead of passing custom D feature channels. And this bug only occurs with passing with_eval3d + with_ut options.