Dimension mismatch between rendered image and reference image during nerf training

[mniverthi]

In scripts/run.py at line 269 we have

diffimg = np.absolute(image - ref_image)

This results in a dimension mismatch between the two arrays, which results in an error. I think this is due to line 169 in src/python_api.cu which allocates a height x width x 4 buffer to store the new image. I'm not entirely sure why this is the case.

py::array_t<float> result({height, width, 4});

Changing lines 269-270 in scripts/run.py seemed to fix the issue, but I'm not sure if this should've been done.

diffimg = np.absolute(image[..., :-1] - ref_image)
# diffimg[...,3:4] = 1.0

[Tom94]

Hi there, instant-ngp always operates on RGBA (since opacity is an inherent part of NeRF), and so always produces RGBA images. If you have a non-transparent background color specified, the rendered image's alpha channel will always be 1, though (because it got composited with the opaque background), and your solution is perfectly valid.

To generalize the code to both RGB and RGBA datasets, run.py could be extended with functionality to detect RGB datasets and to automatically append an opaque alpha channel to the reference image in that case.

[mniverthi]

Ok, thanks for the clarification. I didn't realize that RGBA was being used in this case.