Researchers at the UC Davis College of Engi­neering have developed a camera that uses a thin microlens array and new image processing algorithms to capture three dimensional information about objects in a scene with a single exposure. The camera could be useful for a variety of appli­cations such as industrial part inspection, gesture recog­nition and collecting data for 3D display systems. “We consider our camera lensless because it replaces the bulk lenses used in conventional cameras with a thin, lightweight microlens array made of flexible polymer,” said research team leader Weijian Yang, assistant professor of electrical and computer engi­neering at UC Davis. “Because each microlens can observe objects from different viewing angles, it can accomplish complex imaging tasks such as acquiring 3D infor­mation from objects partially obscured by objects closer to the camera.”

Because the camera learns from existing data how to digitally reconstruct a 3D scene, it can produce 3D images in real time. “This 3D camera could be used to give robots 3D vision, which could help them navigate 3D space or enable complex tasks such as mani­pulation of fine objects,” said Yang. “It could also be used to acquire rich 3D information that could provide content for 3D displays used in gaming, enter­tainment or many other appli­cations.” The new camera grew out of previous work in which the researchers developed a compact micro­scope that can image 3D micro­scopic structures for biomedical applications. “We built the micro­scope using a microlens array and thought that a similar concept could be applied for imaging macro­scopic objects,” said Yang.

The individual lenses in the new camera allow it to see objects from different angles or perspectives, which provides depth information. Although other research groups have developed cameras based on single layer microlens arrays, it has been difficult to make them practical because of extensive cali­bration processes and slow reconstruction speeds. To make a more practical 3D camera for macroscopic objects, the researchers considered the microlens array and the recon­struction algorithm together rather than approaching each separately. They custom designed and fabricated the microlens array, which contains 37 small lenses distributed in a circular layer of polymer that is just 12 millimeters in diameter. The recon­struction algorithm they developed is based on a highly efficient arti­ficial neural network that learns how to map information from the image back to the objects in a scene.

“Many existing neural networks can perform designated tasks, but the underlying mechanism is difficult to explain and understand,” said Yang. “Our neural network is based on a physical model of image recon­struction. This makes the learning process much easier and results in high quality recon­structions.” Once the learning process is complete, it can reconstruct images containing objects that are at different distances away from the camera at a very high speed. The new camera doesn’t need cali­bration and can be used to map the 3D locations and spatial profiles or outlines of objects.

After performing numerical simu­lations to verify the camera’s performance, the researchers performed 2D imaging that showed perceptually pleasing results. They then tested the camera’s ability to perform 3D imaging of objects at different depths. The resulting 3D recon­struction could be refocused to different depths or distances. The camera also created a depth map that agreed with the actual object arrangement. “In a final demons­tration we showed that our camera could image objects behind the opaque obstacles,” said Yang. “To the best of our knowledge, this is the first demonstration of imaging objects behind opaque obstacles using a lensless camera.”

The researchers are currently working to reduce artifacts, or errors, that appear in the 3D reconstructions and to improve the algorithms to gain even higher quality and speed. They also want to miniaturize the overall device footprint so it could fit into a cellphone, which would make it more portable and enable more appli­cations. “Our lensless 3D camera uses computational imaging, an emerging approach that jointly optimizes imaging hardware and object recon­struction algorithms to achieve desired imaging tasks and quality,” said Yang. “With the recent development of low-cost, advanced micro-optics manu­facturing techniques as well as advancements in machine learning and compu­tational resources, computational imaging will enable many new imaging systems with advanced func­tionality.” (Source: UC Davis / Optica)

Reference: F. Tian & W. Yang: Learned lensless 3D camera, Opt. Exp. 30, 34479 (2022); DOI: 10.1364/OE.465933

Link: Yang Research Laboratory, Dept. of Electrical and Computer Engineering, University of California, Davis, USA