Depthmap for Animation: keeping continuity between frames

[steampunkchris]

I love the output your script produces!
I create lots of animation using Deforum and need a decent depthmap to bring it to VR. The depthmaps generated by Deforum for its 3D movement are totally unusable. Currently the "Batch from Directory" treats each image as a separate entity and the results are a high amount of flicker/distortion between each frame. This occurs even if the subject is well defined. I have resorted to using a longer diffusion cadence and using FlowFrame to create tweens but the inconsistency is still nauseating LOL. Any suggestions or possible fixes?

[thygate]

All images are processed individually, so there cannot be any temporal consistency, there is no easy fix for this.

For the backgrounds you could try far clipping to clean them up a bit in some cases ..

An other option might be to post process the generated depth maps through some moving average filter to massage things out ..

[steampunkchris]

Thanks Bob! Already doing those options... is there any way for the depth map generation to use the preexisting frame as a basis for the current.. it would at least help objects from "popping" to the front and back in a frame =)

[thygate]

I'm afraid not, that's not how these networks work. The boost option gives the highest quality depth maps, but these will have the least amount of temporal consistency because of the merging. I think your best option is to use one of the recent midas networks in higher resolution without boost.

[thygate]

Check this video posted by Mickmumpitz on youtube : https://www.youtube.com/watch?v=0tFe9dashgI (time index 4:51)
He uses a tool called EbSynth, using optical flow analysis of the video it can transform an initial depth map of the first frame to generate much more consistent depth frames from the video.

[AugmentedRealityCat]

Would it be possible to use some optical flow tools to extract per-pixel movement from a video and save that data as a motion vector map showing what is moving where at each frame - much like EBSynth does I guess - but to apply this movement to the latent noise itself, prior to image generation ?

My intuition tells me this would get us much closer to a solution for generating steady animated content.

I came to this conclusion after making many tests based on the use of prerendered dept maps sequences as feeds for ControlNet (with the batch hack). Those tests show that even if you have a very steady and clean depth map sequence, the results will not follow well. If you let the seed number change every frame, then the interpretation of the depth map will also change from frame to frame, so it will be jumpy. If on the opposite you keep the exact same seed number for all frames, then your depth map might make the overall scene move while some details, like bright highlights or dark areas, try to remain in place. You can see that one influence, the depth map, is trying to make the object move one way, but that another influence, the latent noise, does its best to maintain everything in its current position.

My guess is Depth Maps and Latent Noise would produce better results if they could work together, and for that they must follow each other, hence the idea to use optical flow analysis and applying the extracted movement to the latent noise itself prior to step 1.

If you are willing to give this a try, let me know and I'll provide you with sample sequences of per-pixel motion vectors. At first we should just check if it works to animate the latent noise - maybe it's simply not possible, and if that's the case then we won't have worked on optical flow extraction for nothing !

[AugmentedRealityCat]

About the Latent Noise influence on image generation when you keep the same seed, here is some info:

That’s why for example using the same latent noise seed but different prompts tends to give images that are related to each-other at the level of overall composition, but not at the level of individual textures or small-scale patterns. The diffusion process doesn’t know how to change those long-wavelength features. And the longest wavelength feature is the average value of the image as a whole, which is also the feature that is least likely to vary between independent samples of the latent noise.

https://www.crosslabs.org//blog/diffusion-with-offset-noise