Researchers have developed the first lidar-based augmented reality head-up display for use in vehicles. Tests on a proto­type version of the techno­logy suggest that it could improve road safety by seeing through objects to alert of potential hazards without distracting the driver. The technology, developed by researchers from the University of Cambridge, the University of Oxford and University College London (UCL), uses lidar data to create ultra high-definition holo­graphic repre­sentations of road objects which are beamed directly to the driver's eyes, instead of 2D windscreen projections used in most head-up displays.

While the technology has not yet been tested in a car, early tests, based on data collected from a busy street in central London, showed that the holo­graphic images appear in the driver's field of view according to their actual position, creating an augmented reality. This could be parti­cularly useful where objects such as road signs are hidden by large trees or trucks, for example, allowing the driver to see through visual obstruc­tions.

“Head-up displays are being incor­porated into connected vehicles, and usually project information such as speed or fuel levels directly onto the windscreen in front of the driver, who must keep their eyes on the road,” said lead author Jana Skir­newskaja, a Ph.D. candidate from Cambridge's Department of Engi­neering. “However, we wanted to go a step further by repre­senting real objects in as panoramic 3D projections.”

Skirnewskaja and her colleagues based their system on lidar a remote sensing method which works by sending out a laser pulse to measure the distance between the scanner and an object. Lidar is commonly used in agri­culture, archaeo­logy and geography, but it is also being trialled in auto­nomous vehicles for obstacle detection. The researchers scanned Malet Street, a busy street on the UCL campus in central London. Phil Wilkes, a geographer who normally uses Lidar to scan tropical forests, scanned the whole street using terrestrial laser scanning. Millions of pulses were sent out from multiple positions along Malet Street. The data was then combined with point cloud data, building up a 3D model.

“This way, we can stitch the scans together, building a whole scene, which doesn't only capture trees, but cars, trucks, people, signs, and everything else you would see on a typical city street,” said Wilkes. “Although the data we captured was from a sta­tionary platform, it's similar to the sensors that will be in the next generation of auto­nomous or semi-auto­nomous vehicles.” When the 3D model of Malet St was completed, the researchers then transformed various objects on the street into holo­graphic projections. The data, in the form of point clouds, was processed by separa­tion algorithms to identify and extract the target objects. Another algorithm was used to convert the target objects into computer-generated diffrac­tion patterns. These data points were implemented into the optical setup to project 3D holo­graphic objects into the driver's field of view.

The optical setup is capable of projecting multiple layers of holograms with the help of advanced algorithms. The holo­graphic projection can appear at different sizes and is aligned with the position of the represented real object on the street. For example, a hidden street sign would appear as a holo­graphic projection relative to its actual position behind the obstruc­tion, acting as an alert mechanism.

In future, the researchers hope to refine their system by personalising the layout of the head-up displays and have created an algo­rithm capable of projecting several layers of different objects. These layered holo­grams can be freely arranged in the driver's vision space. For example, in the first layer, a traffic sign at a further distance can be pro­jected at a smaller size. In the second layer, a warning sign at a closer distance can be displayed at a larger size.

“This layering technique provides an aug­mented reality experience and alerts the driver in a natural way,” said Skirnewskaja. “Every individual may have different pre­ferences for their display options. For instance, the driver's vital health signs could be projected in a desired location of the head-up display. Panoramic holo­graphic pro­jections could be a valuable addi­tion to existing safety measures by showing road objects in real time. Holograms act to alert the driver but are not a distrac­tion.” The researchers are now working to miniaturize the optical components used in their holo­graphic setup so they can fit into a car. Once the setup is complete, vehicle tests on public roads in Cambridge will be carried out. (Source: U. Cambridge)

Reference: J. Skirnewskaja et al.: LiDAR-derived digital holograms for automotive head-up displays, Opt. Exp. 29, 13681 (2021); DOI: 10.1364/OE.420740

Link: EPSRC Centre for Doctoral Training (CDT) in Connected Electronic and Photonic Systems, University of Cambridge, Cambridge, United Kingdom