An international business–academia colla­boration has yielded a new type of projector that can project RGB and invisible infrared images simul­taneously and independently at a high speed of almost 1,000 fps. With its high image alignment accuracy enabled by a custom optics engine, this projector will help closely integrate invi­sible sensing and visible display for a wide range of novel appli­cations, such as dynamic projection mapping.

Modern projectors and related techniques, such as projection mapping, allow us to visually augment surfaces, opening up vast possi­bilities for amusement attractions, concerts, stages, digital signage, task assistance, and more. In many of these appli­cations, the projection target is not just limited to a flat, white, and static screen; the projected image has to be controlled in real time to match the surface of complex moving targets and make their augmented appearance more con­vincing. Techniques used for such purposes are referred to as dynamic projection mapping (DPM) and have gathered much attention over the past few years.

They rely on visual sensing methods to detect the target surface and require high speed, both in sensing and projection frame rate, to properly merge the projected image and its target without noticeable mis­alignment. Unfor­tunately, conven­tional DPM has limited uses because it either needs markers attached to the target or has to be restricted to a single plane to meet the high speed requirements. Recently developed projection mapping techniques can achieve a very realistic appearance, even in dynamic scenes. However, they suffer from critical limi­tations, such as the need for markers on moving pro­jection targets.

On the other hand, projectors have also played important roles in visual sensing appli­cations; projector–camera systems, which project a reference image onto the target surface and then capture it, can efficiently acquire much spatial infor­mation without using markers. Nevertheless, this type of sensing has not been used for DPM because no existing device has ever met the demands. That is, no device can handle display and sensing simul­taneously with high perfor­mance and without them interfering with each other. 

Fortunately, things might soon change on this front thanks to the new type of projector recently developed by a team led by Yoshi­hiro Wata­nabe at Tokyo Institute of Technology (Tokyo Tech), Takeshi Yuasa at Tokyo Electron Device Limited, Uwe Lippmann at Fraunhofer Institute for Applied Optics and Precision Engineering, and Petra Aswendt at ViALUX. This projector can simul­taneously project visible RGB and invisible infrared (IR) images at a high frame rate of 925 fps. “Using our projector, we can construct a system that can sense its target surface using invisible wavelengths based on a projector–camera confi­guration which is a long-established computer-vision technology,” explains Watanabe, “at the same time, the system can adaptively mani­pulate the images to display in visible wavelengths based on its sensing results.”

At the core of this projector lie two DLP (Digital Light Processing) digital micro­mirror devices, which can handle 24-bit RBG and 8-bit IR images simul­taneously and independently. The team also developed a novel optical engine to align the images coaxially with excellent precision while allowing for a compact confi­guration capable of high-power projection. The developed projector has a unique optics engine and two digital micro­mirror devices that enable it to project RGB and IR images with excellent alignment accuracy. 

As a first appli­cation case, the team plans to introduce their novel projector for a new type of DPM system, which will be capable of mani­pulating the appearance of the entire scene based on the sensed shape in a markerless way. “Conven­tional DPM approaches are limited in that they can only augment the appearance of targets with markers,” states Watanabe, “The new type of mapping enabled by our projector will hopefully extend the fields of application of DPM.” Moreover, while only IR projection will be used for shape sensing in their first project, a projector–camera confi­guration using the entire system could be designed to acquire deeper spatial and physical infor­mation, thereby greatly enhancing projection mapping. (Source: Tokyo Tech)

Reference: U. Lippmann et al.: High-Speed RGB+IR Projector based on Coaxial Optical Design with Two Digital Mirror Devices, PRJ2-4L, 28th International Display Workshops, December 2021

Link: Fraunhofer Institute for Applied Optics and Precision Engineering, Munich, Germany • Watanabe Lab, Tokyo Institute of Technology, Tokyo, Japan