Plotting the bounding boxes in the point cloud

[YashRunwal]

Hello,

I am working on a project in which I have an RGB Image and the corresponding 2d bounding boxes for the objects in the image. I also have a respective point cloud and therefore, I can extract the depth information of the objects in the 2d image from the point cloud. I would like to convert these 2d bounding boxes to 3d boxes (cuboid) and then plot it inside the point cloud, thus indicating the position/orientation, i.e. pose of the object inside the point cloud.

However, I am stuck on the conversion of 2d boxes to 3d. Can anyone lend a hand perhaps?

Thank You.

[germanros1987]

Better late than never.

For starters, the multiple version of the "unproject" functions are currently in awkward places of Open3D. So we need to start by writing our own unproject function:

import numpy as np
import open3d as o3d

def unproject(points_2d, depths, intrinsics):
    # Extract the intrinsic parameters
    fx, fy = intrinsics.intrinsic_matrix[0, 0], intrinsics.intrinsic_matrix[1, 1]
    cx, cy = intrinsics.intrinsic_matrix[0, 2], intrinsics.intrinsic_matrix[1, 2]

    # Backproject each 2D point to 3D
    points_3d = []
    for (x, y), depth in zip(points_2d, depths):
        X = (x - cx) * depth / fx
        Y = (y - cy) * depth / fy
        Z = depth
        points_3d.append([X, Y, Z])
        
    return np.array(points_3d)

This is just a quick and dirty way to do the unproject, as a way of example.

Then use the unproject function to create an array with the 3D points representing the 4 points of your bounding box.

# Example intrinsic parameters. You need to know your camera intrinsics for the 2D points.
intrinsics = o3d.camera.PinholeCameraIntrinsic()
intrinsics.set_intrinsics(WIDTH, HEIGHT, F_Y, F_Y, C_X, C_Y)

# Example 2D points (in pixel coordinates) and their corresponding depths
points_2d = [[100, 150], [200, 150], [200, 250], [100, 250]]
depths = [2.0, 2.5, 3.0, 2.5]

# Backproject the 2D points to 3D
points_3d = backproject(points_2d, depths, intrinsics)

Now you can assume that the 3D bounding box is aligned with +Z. Therefore the missing 4 points defining the 3D bounding box must be:

points_3d_extra = points_3d + np.array([0, 0, delta_Z]).reshape(1,-1)
 eight_points_3d = np.vstack((points_3d, points_3d_extra))

where delta_Z is a free parameter defined by the user or somehow estimated from additional information.

After that, you can create a point cloud and extract an oriented bounding box.

# Convert to a point cloud
pcd = o3d.geometry.PointCloud()
pcd.points = o3d.utility.Vector3dVector(eight_points_3d)

# Compute the OBB
obb = pcd.get_oriented_bounding_box()

Finally, if you are able to obtain the orientation of the bounding box from additional information, you could apply rotate to the obb object.

Hope this helps.