stereo image generation

[sina-masoud-ansari]

Hi there, thanks for your work on this. I was looking to create stereo images from the image + depth map and found this:
https://github.com/m5823779/stereo-image-generation

It's a bit out of date but I managed to get it working. I found the res101 depthmap output produces better results than the existing midas integration in that repo. I extracted the neccessary bits from stereo_generation_image.py and modified it to read the depth map output produced from your extension.

I've tested out the stereo image in an Oculus CV1 headset using Whirligig, images attached. If I use this frequently I'm happy to write a PR if you think it's a good addition to your plugin, otherwise in case you wanted to integrate it sooner, heres the necessary code to run it on the command line.

Cheers

import os
import cv2
import argparse
import numpy as np

MONITOR_W = 38.5


def write_depth(depth, bits=1, reverse=True):
    depth_min = depth.min()
    depth_max = depth.max()
    max_val = (2 ** (8 * bits)) - 1

    if depth_max - depth_min > np.finfo("float").eps:
        out = max_val * (depth - depth_min) / (depth_max - depth_min)
    else:
        out = 0
    if not reverse:
        out = max_val - out

    if bits == 2:
        depth_map = out.astype("uint16")
    else:
        depth_map = out.astype("uint8")

    return depth_map


def generate_stereo(left_img, depth, ipd):
    h, w, c = left_img.shape

    depth_min = depth.min()
    depth_max = depth.max()
    depth = (depth - depth_min) / (depth_max - depth_min)

    right = np.zeros_like(left_img)

    deviation_cm = ipd * 0.12
    deviation = deviation_cm * MONITOR_W * (w / 1920)

    print("\ndeviation:", deviation)

    for row in range(h):
        for col in range(w):
            col_r = col - int((1 - depth[row][col] ** 2) * deviation)
            # col_r = col - int((1 - depth[row][col]) * deviation)
            if col_r >= 0:
                right[row][col_r] = left_img[row][col]

    right_fix = np.array(right)
    gray = cv2.cvtColor(right_fix, cv2.COLOR_BGR2GRAY)
    rows, cols = np.where(gray == 0)
    for row, col in zip(rows, cols):
        for offset in range(1, int(deviation)):
            r_offset = col + offset
            l_offset = col - offset
            if r_offset < w and not np.all(right_fix[row][r_offset] == 0):
                right_fix[row][col] = right_fix[row][r_offset]
                break
            if l_offset >= 0 and not np.all(right_fix[row][l_offset] == 0):
                right_fix[row][col] = right_fix[row][l_offset]
                break

    return right_fix


def overlap(im1, im2):
    width1 = im1.shape[1]
    height1 = im1.shape[0]
    width2 = im2.shape[1]
    height2 = im2.shape[0]

    # final image
    composite = np.zeros((height2, width2, 3), np.uint8)

    # iterate through "left" image, filling in red values of final image
    for i in range(height1):
        for j in range(width1):
            try:
                composite[i, j, 2] = im1[i, j, 2]
            except IndexError:
                pass

    # iterate through "right" image, filling in blue/green values of final image
    for i in range(height2):
        for j in range(width2):
            try:
                composite[i, j, 1] = im2[i, j, 1]
                composite[i, j, 0] = im2[i, j, 0]
            except IndexError:
                pass

    return composite

def parse_args():
    parser = argparse.ArgumentParser(formatter_class=argparse.ArgumentDefaultsHelpFormatter)
    parser.add_argument("image", metavar="IMAGE", type=str, help="Image")
    parser.add_argument("depth", metavar="DEPTH", type=str, help="16bit depth map")
    parser.add_argument("--output-dir", type=str, help="Output dir", default=".")
    parser.add_argument("--ipd", type=float, default=6.5, help="Interpupillary distance (cm)")
    return parser.parse_args()


def main():
    args = parse_args()
    ipd = args.ipd

    left_img = cv2.imread(args.image)
    depth_map = cv2.imread(args.depth, cv2.IMREAD_ANYDEPTH)

    depth_map = cv2.blur(depth_map, (3, 3))

    right_img = generate_stereo(left_img, depth_map, ipd)
    stereo = np.hstack([left_img, right_img])
    anaglyph = overlap(left_img, right_img)

    name, ext = os.path.splitext(os.path.basename(args.image))
    cv2.imwrite(os.path.join(args.output_dir, f"{name}_stereo.png"), stereo)
    # cv2.imwrite(os.path.join(args.output_dir, f"{name}_anaglyph.png"), anaglyph)


if __name__ == "__main__":
    main()

[thygate]

That's a great find ! I will test it as soon as I find some time.

If that's all that's required, and the results are good, I would certainly want to include it in the script.

[sina-masoud-ansari]

Great! In my experience, switching the eye order in the app used to view the side-by-side image tends to make it 'pop' a lot more but also has some quite obvious layering going on. I'll keep looking around for mono to stereo workflows in case there's something better

[thygate]

Thanks for contributing, I have to say this community is awesome. Keep it coming =)

I've only had a little time to play with it so far, with mixed results. It all feels a bit massaged.
There is no paper, no source comments, no explanation in the repo readme, some magic numbers, and

except IndexError:
    pass

isn't inspiring confidence either ..

I need to play with it further and investigate deeper ..

edit: I'm converting this to a discussion, we can share our findings and results there.

[thygate]

I was testing with the anagllyph images only so far (cause i'm lazy), and I could not get it to pop out.

The overlap function was assuming BGR, and now after swapping the red and blue channel in that function the anaglyph seems to be correct and is popping out.

Now the results are actually pretty impressive, I'm going to do a bit more testing ..

[thygate]

Also swapping left and right images in the combine function now makes it really pop out.

[thygate]

I've included it in v0.2.9, I credited you in the readme. thanks again for the tip, the results are quite amazing !

You can now use boost to get high resolution depth maps and generate stereo images.

[thygate]

Original image from this discussion : #34 using boost and res101.

Very impressive results.

[sina-masoud-ansari]

That's awesome! I just tried it out on these and it's looking great. Thanks for getting it debugged, tested and integrated so quickly!

[thygate]

No problem, I wanted to see this in action too =)
And the results are pretty darn amazing, I'm wondering now how doable it would be to convert a video to 3D ..
I've done this with midas with just the depthmaps for viewing on lookingglass, but curious to see now how it would perform with boost, generate sbs video and watch on big screen ..

[sina-masoud-ansari]

Yeah after playing around some more on synthetic and real images, I'm really impressed and video is the next logical step :) Let me know if you need any help developing or testing. I thought I might have a look at improving this even further with a model that generates SBS images directly using this kind of data: https://github.com/leiainc/holopix50k for training.

[brianjorden]

(To paraphrase Blaise Pascal, if I had more time, I would have written a shorter message. TLDR: thanks, working on video, is anyone else?)

First off, thank you both (and ALL open source contributors) for all the work. I haven't contributed as much as I probably should have over the years. Also, I'm really new into python and AI/ML, but dove head first into a couple things in the past several weeks. So please let me know if it seems like I might be missing something or if it sounds like I'm on a bad path when I should know better.

Initial idea:
A few days ago, I started researching / experimenting with the idea of image generation for viewing with a VR headset. But, with the sorta 3D you get from a screenshot or when a VR180 video is paused. Was thinking of streaming a slide show sort of thing from Stable Diffusion that you can prompt from within the headset. Basically a holodeck, but in a VR180 slide show. Obviously, that is just until video generation is more practical.

First concept:
Originally thought I might try training a LORA for Stable Diffusion, but that seems incredibly optimistic to "just work" after researching a bit more. There are just several big hurdles that I'm not sure if that approach, at least alone, could solve quite yet. More importantly, I realized I needed to work more iteratively to start.

Early efforts:
My early experiments and prototypes have had mixed results. I've realized there are some really weird and complicated things going on with proper VR pictures/video. In the past few days I've been researching video formats, aspect ratios, camera lenses, FOV considerations, the damn elusive "right" amount of fisheye needes, stereoscopic images, depth maps, image and video parsers, VR apps for pics/videos, the very closed source players used on several VR video sites, etc, etc...

The realization:
The best results I've gotten so far have been when trying to create a "control" experiment for quality. I took an existing high quality VR video, grabbed a screenshot of just one half of the video, ran it through this project's add on script, and viewed with immerGallery on my Oculus Quest. I thought I was just finding a reasonable best case scenario to target for now. Instead, I realized it wasn't many more steps, just a lot more iterations, to convert a regular 2D video to VR180. So, I've started down the path of video...

Current effort:
Last night I tested a tiny 6 second clip of video, and it sure wasn't perfect, but it was a pretty solid first prototype to me at least. Again, I'm trying to stack the deck in my favor, but here is what I did to start (very simplified / abbreviated):

• Downloaded a VR180 video I knew was pretty good
• Wrote a little python script to extract images for each frame of video
• Wrote another python script to split the right eye / left eye portions of the images into different files
• Used this project to generate stereoscopic right-left images from JUST one eye
• Compared the originally extracted VR180 images with the newly generated ones with ImmerGallery (app helped with tweaking viewing style / format)
• Wrote yet another python script to reassemble the newly generated right-left images into a video
• Viewed and compared the original tiny clip and the newly generated one with SKYBOX VR Video Player (app helped with tweaking viewing style / format)

Still missing / to dos:
I felt like the first prototype was better than I could have hoped for, but there were some glaring mistakes, issues, and extra things I'm planning:

• I forgot to deal with audio at all, oops, know that might be tricky especially with videos that don't have nice clean frame rates
• There seems to be a fair bit of metadata needed to make a video/image file act correctly as VR180 without manual overrides in the viewing apps
• Speed is a serious issue, took me a little over 2 hours to run 360 of the 1440x1440 images I used through this project to get the side-by-side images, that was for only 6 seconds of video at 60fps
• Need to consider the differences in final quality based on frame rate, cutting this to 30 seems logical, but I think the higher frame rate really helped the "jitter" look a bit more like blur/morphing
• Going to research some of the methods mentioned in other threads about an initial depth map approach
• Fighting through some of the frustrating python dependency / environment issues, this ecosystem feels very chaotic and fragile by comparison to something like npm/node_modules in JS/TS
• Need to test performance using scripts from this project outside of SD Web UI
• Getting the fisheye portion as "right" as possible for any given 2D input image / video is proving to be rather tricky
• Realized I need to pad the images with some sort of background / blur to get them to a 1:1 aspect ratio
• Going to test padding the 2D images based on resolution, aspect ratio, and "typical FOV" of common combinations to try and deal with the size/scale issues
• Issues to deal with around wrangling >100k images to convert a 30 min 60fps video
• Want to figure out if I'm getting anywhere near the performance I should expect for the hardware I'm using (pretty sure I'm not, see ecosystem issues above lol)
• Going to do some more research into the couple companies offering 2D to 3D video conversion, but most seem to be walled gardens or just creating a sorta-3D effect for typical 2D viewing
• Lots of other things...

Why I'm posting all of this:
I decided to try to "work" in a bit more open source type way on this. I know others have probably already gotten much further down this path than I have, and some might be just starting. Ideally I'd love to see someone to say I'm wasting my time because this already exists in a form similar enough to how I want to use it. So far though, I haven't really found any other open source way to convert a 2D video to VR180.

A close second best outcome, to me, would be to chat a little bit with people who are also interested in and/or working on similar types of things. An unexpected bonus would be if anyone else was actually interested in collaborating a little bit. For now, my Twitter (brian_jorden) DMs are open, but anything like a Discord / Slack that anyone might suggest? If not, I could set something up / find somewhere if anyone is interested in chatting a bit.

Lastly, I'm happy to move / remove this thing if it is at all out of place in your opinion. Thanks again for all the efforts and I'll at least create a new repo with stuff I've used to create some prototypes once I get rid of a few really rough edges. Sorry for all the rambling lol

[brianjorden]

As an actually (somewhat) brief update this time:

I've made a ton of progress here, but have had to unravel some weird nuances around VR images and video in general. I'm pretty far down the path of creating a super streamlined and very opinionated automated process for converting a regular video into a VR video. Since my prior post I've come to learn a TON about tangentially related things like VR180 vs Flat 3D, metadata requirements, terrible native media players, fisheye weirdness, allowable resolutions, and how difficult / limited the options are for streaming.

This project is a great swiss-army-knife of approaches and options, but I believe I may end up spinning what I've been working on off into a separate open source project. I'm hoping to share a first version in the next week or so. I'd still absolutely love to even just chat with someone else about this and what I'm still planning. Anyone is free to reach out if interested.

[Jonseed]

This is very cool! Looking forward to testing this out.

One thing I thought I'd mention, the first images above in the OP of the man side-by-side seem to be reversed. They should be swapped. Our left eye would see less of his left ear (on the right side of his head in the image). As it is, we see more of it in the left image. Etc. So this makes the image reversed in its 3D depth. The other images in the thread look correct.

[sina-masoud-ansari]

Thanks for pointing that out! I think that has been fixed in the current version and explains why swapping the eye-order seemed to fix the depth in that example

[yingtian991221]

Hi, I'm recently also working on the same project. Could you give me some ideas about how long an inference of the depth-generation and reprojecting to new eyes take, on which desktop GPU?

[thygate]

@yingtian991221 The stereo image generation code runs on CPU, the depthmap generation can run on both. Currently it will compute the depthmap first and then compute the stereo image on CPU. It could be more streamlines to be faster.

I've processed a 720p video, frame-by-frame (batch mode) with the zoedepth model to generate stereo image pairs (eg. 3D video), and it takes around a second per frame on a 3rd gen RTX, and an older intel i5.

[semjon00]

This is a good idea! Would require some significant work, even considering the large refactor.

[yingtian991221]

Thanks! I thought about testing the whole stuff on a headset, but this result is much beyond a headset's capacity.