Photos are two-dimensional, but autonomous vehicles and other technologies have to navigate the three-dimensional world. Researchers have developed a new method to help artificial intelli­gence extract 3D information from 2D images, making cameras more useful tools for these emerging technologies. “Existing techniques for extracting 3D information from 2D images are good, but not good enough,” says Tianfu Wu from North Carolina State University. “Our new method, called Mono­Xiver, can be used in conjunction with existing techniques – and makes them signi­ficantly more accurate.”

The work is particularly useful for appli­cations such as autonomous vehicles. That’s because cameras are less expensive than other tools used to navigate 3D spaces, such as lidar, which relies on lasers to measure distance. Because cameras are more affordable than these other techno­logies, designers of autonomous vehicles can install multiple cameras, building redundancy into the system. But that’s only useful if the AI in the autonomous vehicle can extract 3D navi­gational information from the 2D images taken by a camera. This is where MonoXiver comes in.

Existing techniques that extract 3D data from 2D images make use of bounding boxes. Specifically, these techniques train AI to scan a 2D image and place 3D bounding boxes around objects in the 2D image, such as each car on a street. These boxes are cuboids, which have eight points. The bounding boxes help the AI estimate the dimen­sions of the objects in an image, and where each object is in relation to other objects. In other words, the bounding boxes can help the AI determine how big a car is, and where it is in relation to the other cars on the road. However, the bounding boxes of existing programs are imperfect, and often fail to include parts of a vehicle or other object that appears in a 2D image.

The new MonoXiver method uses each bounding box as a starting point, or anchor, and has the AI perform a second analysis of the area surrounding each bounding box. This second analysis results in the program producing many additional bounding boxes surrounding the anchor. To determine which of these secondary boxes has best captured any missing parts of the object, the AI does two comparisons. One comparison looks at the geometry of each secondary box to see if it contains shapes that are consistent with the shapes in the anchor box. The other comparison looks at the appearance of each secondary box to see if it contains colors or other visual charac­teristics that are similar to the visual charac­teristics of what is within the anchor box.

“One significant advance here is that MonoXiver allows us to run this top-down sampling technique – creating and analyzing the secondary bounding boxes – very effi­ciently,” Wu says. To measure the accuracy of the MonoXiver method, the researchers tested it using two datasets of 2D images: the well-established KITTI dataset and the more challenging, large-scale Waymo dataset. “We used the MonoXiver method in conjunction with MonoCon and two other existing programs that are designed to extract 3D data from 2D images, and MonoXiver significantly improved the per­formance of all three programs,” Wu says. “We got the best performance when using MonoXiver in conjunction with MonoCon.

“It’s also important to note that this improve­ment comes with relatively minor computational overhead,” Wu says. “For example, MonoCon, by itself, can run at 55 frames per second. That slows down to 40 frames per second when you incorporate the MonoXiver method – which is still fast enough for practical utility. “We are excited about this work, and will continue to evaluate and fine-tune it for use in autonomous vehicles and other appli­cations,” Wu says. (Source: NCSU)

Reference: X. Liu et al.: Monocular 3D Object Detection with Bounding Box Denoising in 3D by Perceiver, arXiv 2304.01289

Link: Visual Modeling, Computing and Learning (T. Wu), North Carolina State University, Raleigh, USA