Generative Transposed Convolution

[lishiyu005]

Dear developers,

I greatly appreciate this good work.

Do you provide 3D generative transposed convolution operation that allows to create new possible coordinates as in Minkowski Engine: https://github.com/NVIDIA/MinkowskiEngine/blob/b71caef9b45ab32c02d2c1b5f48f13c1eb5df526/MinkowskiEngine/MinkowskiConvolution.py#L539

If not do you plan to add a shape completion task similar to https://github.com/NVIDIA/MinkowskiEngine/blob/b71caef9b45ab32c02d2c1b5f48f13c1eb5df526/examples/completion.py?

Thank you very much.

[swahtz]

Hi @lishiyu005 ,

Thanks very much, we're glad that you find it useful!

Yes, we do supply functionality that is equivalent to the MinkowskiEngine generative transposed convolution idea. In fact, fixing up some issues in this functionality was part of the recent PR #726 and will be in the imminent v0.6 release, so I'd recommend you build from source or install nightly builds to try this out.

ConvolutionPlan.from_grid_batch_transposed(kernel_size, stride, source_grid, target_grid=None) selects the COMPLETE topology policy and builds the target as source_grid.conv_transpose_grid(kernel_size, stride). New voxels are generated everywhere any kernel tap can reach, which is the "generative" behavior .

The target grid's voxels can be masked by building a GridBatch from a masked ijk by gathering features with inject_from. Those masked features can be a classifier of which voxels to mask off and can be differentiable. However, we don't have a one-call layer to do this, though it would be very useful… in fact we had examples that did this for the older API as part of a shape completion example but they're old enough now that they don't structurally work but the idea is the same.

Let me go and write this layer and integrate it into an example, as you suggest, since the logic for doing this is common to these types of problems but also perhaps a bit tricky to do right. I can then add it to our nn module and as an example script which will update our old structure prediction example.