Generate 3D inpainted mesh from an existing depthmap file

[diegomagikal]

Hi,

How can I generate 3D inpainted mesh from an existing depthmap file? Currently, the UI allows generating a .ply file from an image only. I intend to use the video generator feature using a depthmap created or fine-tuned on another software.

Thanks

[AugmentedRealityCat]

The 3d inpainted mesh generation and the video generation code are both part of a previous project that has been adapted to work with this extension. There is a Colab you can use online to generate depthmap+PLYmodel+4videos, and the code was also adapted to run on windows by @donlinglok

https://github.com/donlinglok/3d-photo-inpainting

https://colab.research.google.com/drive/1706ToQrkIZshRSJSHvZ1RuCiM__YX3Bz

So you could use either the Colab or the windows version to generate a mesh from a custom depth map by putting it in the proper folder and setting the parameters accordingly. Here are the instructions to make it use a moon depth map that has been pre-generated - you could just replace it with your own depth map.

Remember to set the following two flags as listed below if user wants to use manually edited depth/moon.png as input for 3D Photo.

depth_format: '.png'
require_midas: False

Now, the question that remains is: would it be possible to modify those exact same parameters in the current A1111 DepthMap extension to achieve similar behaviour ? It might well be, but I haven't tested it and I cannot guarantee it either. But it's worth a try - it would certainly be more convenient than installing a windows version, or using the Colab.