Does the python api gives a way to train datasets batch by batch?

[StarsTesla]

#221 As I last time mentioned the problem of "Out of Memory", and I read the former issues #189 talking about that, so as we have a lot of images shot from some professional camera, except to improve my GPU memory, I wonder if there is way to train those pictures batch by batch instead of pushing all data into GPU RAM? Or does the python api had done that?

[Tom94]

This is currently not possible, although it would be nice to eventually be able to pass a bunch of rays with associated target RGB colors through the Python API.

[pwais]

@Tom94 Has anybody tested yet how much the image -> rays conversion latency compares with the rest of the training loop latency? That might help shape how all NeRF minibatching gets implemented... it's kinda mixed in pytorch land. Also since this project has such wide GPU support, could be easier to extrapolate from the 3090 to older PCIe 3.x cards.

[Tom94]

If all the data is already on the GPU, converting from image coordinates to rays is basically free.

Based on profiles of our training code, I would classify the major components of

• 10% density grid updates
• 40% time spent compacting the rays & computing the loss + gradient
• 50% time spent in the Encoding+NN (inferencing before compaction + forward/backward passes after compaction)
• Add a 15% overhead for computing input gradients (when extrinsics/intrinsics are to be optimized)

Regarding the original question: I'm unsure whether the CPU side preparing the data wouldn't become a bottleneck if it became the source of data rather than randomly sampling from existing GPU buffers.