[REQUEST] Implementing Depthmap Anything V2 model?

[eyeEmotion]

Hi,

is there any intention of implementing the new Depth Anything v2 model? (Base and Large).

I'm currently using it via cmd-prompt, and it has a lot more detail, and is a little bit faster at generating depthmaps too. Although in certain areas, it does take a step backwards though in comparison to v1. But still a vast improvement overal.

Cheers.

PS: sorry, I don't know how to put labels on there requests.

[petermg]

Hi,

is there any intention of implementing the new Depth Anything v2 model? (Base and Large).

I'm currently using it via cmd-prompt, and it has a lot more detail, and is a little bit faster at generating depthmaps too. Although in certain areas, it does take a step backwards though in comparison to v1. But still a vast improvement overal.

Cheers.

PS: sorry, I don't know how to put labels on there requests.

I'm also hoping for this!

[semjon00]

I am hoping somwbody could impöement it. I am busy currently.

[graemeniedermayer]

If @petermg @eyeEmotion @semjon00 wanted to test it out #458 in pull request. Might need some finessing but it seems to be working.

[petermg]

If @petermg @eyeEmotion @semjon00 wanted to test it out #458 in pull request. Might need some finessing but it seems to be working.

Awesome! How do I install it instead of the main branch?

[petermg]

Thought I got it installed, it shows up as an option but it gives the error:

ModuleNotFoundError: No module named 'depth_anything_v2'

[petermg]

Here's the full error:

Please report this issue here. Make sure to provide the full stacktrace:
Traceback (most recent call last):
File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\common_ui.py", line 548, in run_generate
input_i, type, result = next(gen_obj)
File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\core.py", line 329, in core_generation_funnel
raise e
File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\core.py", line 130, in core_generation_funnel
model_holder.ensure_models(inp[go.MODEL_TYPE], device, inp[go.BOOST], inp[go.TILING_MODE])
File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\depthmap_generation.py", line 68, in ensure_models
self.load_models(model_type, device, boost, tiling_mode)
File "F:\SD\stable-diffusion-webui\extensions\stable-diffusion-webui-normalmap-script\src\depthmap_generation.py", line 233, in load_models
from depth_anything_v2.depth_anything_v2.dpt import DepthAnythingV2
ModuleNotFoundError: No module named 'depth_anything_v2'

[graemeniedermayer]

You might need to do a extra git pull.

If not check that in the extension file stable-diffusion-webui-depthmap-script folder there is a depth_anything_v2 folder that isn't empty.

I might have also made a pathing error somewhere.

[petermg]

You might need to do a extra git pull.

If not check that in the extension file stable-diffusion-webui-depthmap-script folder there is a depth_anything_v2 folder that isn't empty.

I might have also made a pathing error somewhere.

Still getting the error. Here is what that folder contains on my system:

also here is the error:

[petermg]

and when I did a git pull

[graemeniedermayer]

Thank you for all the information! That is how the folder should look, so I'm a little confused. I made some small edits, but I'm not confident they'll fix it. I'll have to think about it more.

If you have any other extensions that use depth_anything_v2 you could try changing the line to from depth_anything_v2.dpt import DepthAnythingV2 and seeing if there's any change. You could also try temporarily turning off all the other extensions.

My import structure is very similar to the marigold import. Does marigold work for you?

[petermg]

Thank you for all the information! That is how the folder should look, so I'm a little confused. I made some small edits, but I'm not confident they'll fix it. I'll have to think about it more.

If you have any other extensions that use depth_anything_v2 you could try changing the line to from depth_anything_v2.dpt import DepthAnythingV2 and seeing if there's any change. You could also try temporarily turning off all the other extensions.

My import structure is very similar to the marigold import. Does marigold work for you?

Yes Marigold works no problem. Maybe I installed it wrong? I went to EXTENSIONS tab, UNCHECKED the official depth map extension and installed this one via INSTALL FROM URL and used this as the url:
https://github.com/graemeniedermayer/stable-diffusion-webui-normalmap-script
and for branch put depthanything_v2
It says it's installed. But maybe I did something wrong? Here is my extension tab.

[petermg]

Ok so here is something weird, BEFORE I update to the update you just did, I have the DEPTH Tab as seen here:

But after updating to your update just now, it's totally gone!

[petermg]

ok I deleted the extensions inside the folder and installed just your version and the depth tab is back but still the same exact error. I'll see if I can figure out how to do what you mentioned about changing the line to from depth_anything_v2.dpt import DepthAnythingV2 and seeing if there's any change.

[petermg]

I am unable to find the file "depth_anything_v2.dpt". Should I have it somewhere?

[graemeniedermayer]

the problem import line is in src/depthmap_generation.py file.

Oh I forgot to push my last updates.

[petermg]

the problem import line is in src/depthmap_generation.py file.

Oh I forgot to push my last updates.

THAT WORKED!!!
HOWEVER I did end up needing to manually delete the original installation of the extension because when I just updated it, the "Depth" tab went away again. When I deleted the folder manually and reinstalled the extension, then it worked.

[petermg]

So for some reason YOUR version is working BETTER than if I just use depth anything v2 from here
https://github.com/DepthAnything/Depth-Anything-V2
from the command line!? The one from there using the command line produces a depth map with only 8 bit precision, resulting in stair stepping when trying to apply this to a 2D plane, for example:

You can see the stair stepping in that one, which was done by the command line version, and produces a depth map of only 8 bit integer.
HOWEVER, when I process the same image using your extension of Depth Anything V2 the result I get is:

Do you have any idea why this is? Like what command or line in what python script changes the bit depth accuracy? Also, not that it's needed but is it possible to increase the bit depth precision even more just to see if it would make any difference?

[graemeniedermayer]

Awesome! A lot of scripts downsample to 8bit arrays while saving images. I'm not sure why they do this.

For precision at a certain point it probably just makes more sense to work with pointclouds or float arrays because then they keep more information. This didn't make sense earlier because there wasn't enough detail anyways, but I think monodepth is starting to get precise enough that it might make sense to think about switching out of images for 3d work.

(the line of code in run.py that down samples the array)

depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0
depth = depth.astype(np.uint8)
if args.grayscale:
     depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
else:
     depth = (cmap(depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8)

[petermg]

Awesome! A lot of scripts downsample to 8bit arrays while saving images. I'm not sure why they do this.

For precision at a certain point it probably just makes more sense to work with pointclouds or float arrays because then they keep more information. This didn't make sense earlier because there wasn't enough detail anyways, but I think monodepth is starting to get precise enough that it might make sense to think about switching out of images for 3d work.

(the line of code in run.py that down samples the array)

depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0
depth = depth.astype(np.uint8)
if args.grayscale:
     depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
else:
     depth = (cmap(depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8)

Thank you, how can I modify this so that I can increase the precision?

[petermg]

Awesome! A lot of scripts downsample to 8bit arrays while saving images. I'm not sure why they do this.

For precision at a certain point it probably just makes more sense to work with pointclouds or float arrays because then they keep more information. This didn't make sense earlier because there wasn't enough detail anyways, but I think monodepth is starting to get precise enough that it might make sense to think about switching out of images for 3d work.

(the line of code in run.py that down samples the array)

depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0
depth = depth.astype(np.uint8)
if args.grayscale:
     depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
else:
     depth = (cmap(depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8)

Ok so I changed to this:

 for k, filename in enumerate(filenames):
        print(f'Progress {k+1}/{len(filenames)}: {filename}')
        
        raw_image = cv2.imread(filename)
        
        depth = depth_anything.infer_image(raw_image, args.input_size)
        
        depth = (depth - depth.min()) / (depth.max() - depth.min()) * 2048.0
        depth = depth.astype(np.uint16)
        
        if args.grayscale:
            depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
        else:
            depth = (cmap(depth)[:, :, :3] * 2048)[:, :, ::-1].astype(np.uint16)
        
        if args.pred_only:
            cv2.imwrite(os.path.join(args.outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), depth)
        else:
            split_region = np.ones((raw_image.shape[0], 50, 3), dtype=np.uint16) * 2048
            combined_result = cv2.hconcat([raw_image, split_region, depth])
            
            cv2.imwrite(os.path.join(args.outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), combined_result)

from:

    for k, filename in enumerate(filenames):
        print(f'Progress {k+1}/{len(filenames)}: {filename}')
        
        raw_image = cv2.imread(filename)
        
        depth = depth_anything.infer_image(raw_image, args.input_size)
        
        depth = (depth - depth.min()) / (depth.max() - depth.min()) * 255.0
        depth = depth.astype(np.uint8)
        
        if args.grayscale:
            depth = np.repeat(depth[..., np.newaxis], 3, axis=-1)
        else:
            depth = (cmap(depth)[:, :, :3] * 255)[:, :, ::-1].astype(np.uint8)
        
        if args.pred_only:
            cv2.imwrite(os.path.join(args.outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), depth)
        else:
            split_region = np.ones((raw_image.shape[0], 50, 3), dtype=np.uint8) * 255
            combined_result = cv2.hconcat([raw_image, split_region, depth])
            
            cv2.imwrite(os.path.join(args.outdir, os.path.splitext(os.path.basename(filename))[0] + '.png'), combined_result)

Now it creates files that are 16 bit but are very dark, look almost black, until I open them in GIMP and use the LEVELS tool. I also increase the precision in GIMP as high as it will go BEFORE I save the file, but I don't think I should HAVE to do this but it turns out better when I do. I'm crazy impressed with this so far. Would be nice to be able to automate this so that I can process multiple at a time. Here is my current command line that I run

python run.py --encoder vitl --img-path Inputpics --outdir Outputpics --pred-only --grayscale --input-size 1200

Increasing the input size makes a NOTICEABLE DIFFERENCE!

[graemeniedermayer]

Oh instead of 2048 you should be able to use 255*255 or 65025. This should make them less dark. The input size is interesting I should double check that I implemented that correctly.

[petermg]

Oh instead of 2048 you should be able to use 255*255 or 65025. This should make them less dark. The input size is interesting I should double check that I implemented that correctly.

THANKS!! that works PERFECTLY!!!

[petermg]

I don't really know much about the whole "BOOST" and "TILING" options, they always seem to make things WORSE when I've tried to use them, but at least for Depth Anything V2, I think what IS useful if you could implement it in this plugin version of it, is changing the input size, like manually selecting it on the sliders and also selecting the native input size checkbox.

[noobtoob4lyfe]

Now that there has been so much progress in this thread towards installing and using without errors, would it be possible to give a quick guide of how to install this for a less advanced user like myself? I would really love to give this a try.

[petermg]

Now that there has been so much progress in this thread towards installing and using without errors, would it be possible to give a quick guide of how to install this for a less advanced user like myself? I would really love to give this a try.

https://www.reddit.com/r/StableDiffusion/comments/1e02w8b/how_to_use_depth_anything_v2_in_automatic1111s/