Researchers have used inkjet printing to create a compact multispectral version of a light field camera. The camera, which fits in the palm of the hand, could be useful for many appli­cations including auto­nomous driving, classi­fication of recycled materials and remote sensing. 3D spectral information can be useful for classifying objects and materials; however, capturing 3D spatial and spectral information from a scene typically requires multiple devices or time-intensive scanning processes. The new light field camera solves this challenge by simul­taneously acquiring 3D information and spectral data in a single snapshot.

“To our knowledge, this is the most advanced and integrated version of a multi­spectral light field camera,” said Uli Lemmer from the Karlsruhe Institute of Techno­logy in Germany. “We combined it with new AI methods for reconstructing the depth and spectral properties of the scene to create an advanced sensor system for acquiring 3D information.” The new camera and image reconstruction methods can be used to distinguish objects in a scene based on their spectral charac­teristics. Using inkjet printing to make the camera’s key optical components allows it to be easily customized or manu­factured in large volumes.

“Reconstructed 3D data from camera images are finding widespread use in virtual and augmented reality, autonomous cars, robotics, smart home devices, remote sensing and other applications,” said Michael Heizmann, a member of the research team. “This new techno­logy could, for example, allow robots to better interact with humans or improve the accuracy of classifying and separating materials in recycling. It could also be potentially used to classify healthy and diseased tissues.” Light field cameras, or plenoptic cameras, are specialized imaging devices that capture the direction and intensity of light rays. After image acquisition, compu­tational processing is used to reconstruct 3D image information from the acquired data. These cameras typically use microlens arrays that are aligned with the pixels of a high-resolution camera chip.

To create a multispectral light field camera, the researchers used inkjet printing to deposit a single droplet of material to form each individual lens on one side of ultrathin microscope slides and then printed fully aligned color filter arrays on the opposite side of the microscope slides. The resulting optical component was integrated directly onto a CMOS camera chip. The inkjet printing method allowed precise alignment between the optical components, signi­ficantly reducing the manu­facturing complexity and enhancing effi­ciency. Because this setup produces spectral and depth information that are interwoven in the camera image, the researchers developed methods to separate each component. They found that an approach based on deep learning worked best for extracting the desired information directly from the acquired measure­ments.

“Tackling the challenge of creating a multispectral light field camera was only possible by combining recent advances from manu­facturing, system design and AI-based image reconstruction,” said Qiaoshuang Zhang. “This work pushes the boundaries of inkjet printing – a versatile method with high precision and industrial scalability – for manu­facturing photonic components.”

The researchers tested the camera by recording a test scene that contained multicolor 3D objects at different distances. The image recon­struction algorithm was trained and tested on many synthetic and real multispectral images. The results demonstrate that the prototype camera can simul­taneously acquire 3D spatial and spectral information and that different objects can be imaged and dis­tinguished by their different spectral composition and depth information within one single snapshot. Now that they have completed this first proof-of-concept, the researchers are exploring various applications where a light field camera with an ability to acquire multispectral information could be useful. (Source: Optica)

Reference: Q. Zhang et al.: Compact multispectral light field camera based on an inkjet-printed microlens array and color filter array, Opt. Exp. 32, 23510 (2024); DOI: 10.1364/OE.521646

Link: Light Technology Institute, Karlsruhe Institute of Technology (KIT), Karlsruhe, Germany