Time-of-flight (ToF) sensors add depth information to images and the technology – along with Swir imaging, GS-CMOS sensors and on-chip ­polarisation – is helping move camera modules away from being merely devices that simply capture images, to systems that can provide valuable data for an array of detection and recognition applications. And this will prove vital in creating industrial IoT systems, where the demand for deeper levels of automation is growing rapidly.
And recent years has seen ToF sensors undergo a revolution, with the accuracy and range increasing significantly. And this will enable many new sectors and applications to benefit from the added information they deliver. Indeed, we are on the cusp of them becoming almost ubiquitous in many industrial settings.

What Is Time of Flight?

ToF sensors calculate depth by emitting light outside the visible spectrum –  typically generated by a laser or a LED – and measuring the time it takes for it to reflect off an object. This enables depth information to be added for each pixel, creating full-resolution depth maps at a fraction of the cost of a Lidar system.

The technology fills a unique niche. Unlike low-cost depth-sensing technologies such as passive stereoscopic vision, it has a much higher level of accuracy and isn’t affected when an object is faint against a background, for example white on white.
There are two main types: direct (dToF) and indirect (iToF). While similar in approach, these differ in the way they measure distance, with dToF using a more simple, direct measurement of time from light being emitted and a reflection being detected; and iToF measuring the distance by collecting reflected light to discern the phase shift between emitted and reflected light.
iToF sensors are especially effective in high-speed, high-resolution 3D imaging of objects at short and long distances. Indeed, they are arguably the best-suited depth-sensing technology, especially in industrial applications.

Improving the Accuracy of ToF Sensors

There are still many use cases which require the measurement of longer distances (e.g. for civil engineering), or shorter distances (e.g. for manufacturing) than ToF can deliver. As such recent work has focused on extending their operational range.

The accuracy, and therefore operational range, of a ToF system is set by two factors:

• how efficiently reflected light is captured (condensing efficiency)
• and the speed at which distance-measurement data are processed (rapid distance processing).

If we look first at condensing efficiency. One of the key recent advances in ToF sensors is in the architecture used. Advanced ToF sensors now use an architecture capable of high-speed modulation. This allows for greater distance-measurement accuracy in each individual pixel. Similarly, a small number of sensors are switching from front-illuminated sensors, where the light enters through the lens and passes through the metal-wiring layer before reaching the photosensitive layer, to a back-illuminated architecture, which swaps the metal wiring and photosensitive layer to improve the reflection signal readout precision.

By using these architectures in conjunction with a drive mode on the sensor, it is also possible to extend operational range. to allow both short range (down to a few dozen centimetres, Depending on specification and system environments) and long range (up to several tens of metres) functionality when creating high-precision depth maps in VGA resolution

And if we now look at rapid distance processing, this approach also has a second key benefit in that it captures a depth map for each frame acquired. This means higher frame rates (up to 60 fps at VGA resolution) are possible than, for example, in laser scanning range-finding, which is essential for moving objects such as robots and drones.

Furthermore, by recording depth information at the pixel level, processing on the sensor output (compared with stereoscopic 3D systems) is significantly reduced, making it suitable for applications where faster operation is sought, as in recognition by robots where responsiveness is essential.

This still, however, leaves an issue of noise. And a major focus of recent work on ToF sensors has been to incorporate low-noise technologies. By doing so, distance information can be obtained with much greater precision than with existing ToF sensors, and enabling its use in applications that had, thus far, proved difficult to deliver the required accuracy.

How ToF Sensors Will Be Used

Much has been written about ToF sensors falling cost and their integration into consumer applications, such as smartphones. And it can be tempting to overlook the role it has to play in enhancing industrial and manufacturing processes, or even in construction and agriculture. This would be wrong.

Let’s have a specifically look at these applications and how the increased accuracy, especially for close-proximity and longer-distance sensing increases, can aid them.

Industrial IoT: ToF sensors are also beginning to enter IoT. On the factory floor, this might mean intrusion detection systems for hazardous areas, and object location and detection for robot picking.

Logistics: ToF sensors are already being used in logistics to allow robots to move semi-autonomously through controlled environments. The increase in accuracy that iToF sensors bring will allow their use to be expanded in this sector. The longer-term (more reported on) element of this will be the support of enhanced collision detection and the recognition of the external environment to enable last-mile robots/delivery drones to be more common.

But before that, the depth information ToF cameras give enables logistics firms to optimise material handling lines by measuring positions, sizes, and loading rate and optimising loading of pallets/vehicles to ensure the transportation of goods becomes more efficient. 

B2B augmented reality systems: Augmented reality has been eagerly-adopted to speed the design and manufacturing process in many sectors, with this being well demonstrated by Ford’s rendering of design modifications, allowing a designer to examine how several minor changes would look without creating clay models for each.
The depth and precision that a ToF system brings enables this to be more accurate. Additionally, this information can be used to reducing the motion sickness that is common with VR/AR headset use… and prevent those wearing VR headsets from walking into objects.

Civil engineering: A lot has already been written about autonomous vehicles and the need for additional sensors to help the car understand the world around it: notably (in the case of ToF sensors) for pedestrian and cyclist detection/avoidance. The same principles can be applied to heavy machinery, using ToF sensors to introduce smart technologies to detecting environmental features and objects: providing driver assistance, vehicle automation.They are also expected to be used to better monitor the progress of construction and the status of existing infrastructure.

Agriculture: ToF sensors are already starting to be used in quality control and productivity improvement for farming. If we look at livestock management, ToF (and the 3D information it gives) enables automated milking, body condition scoring, feed monitoring, and iToF and the increased accuracy it gives will improve this greatly.

Additionally, the 3D data from ToF sensors also supports applications like automatic fruit picking by acquiring targets’ positions, and the ability to control farming equipment to improve efficiency.

Conclusion

ToF sensors have undergone a quiet revolution and the accuracy and range have increased significantly in recent years. This makes them the ideal sensor for depth information in both industrial and consumer applications; indeed, even handsets now have the technology embedded. As prices come down, and more iToF sensors – such as the IMX556/IMX570 – become available to the market this sector is likely to see significant growth and competition.

Author

Piotr Papaj, Sony Europe