Artificial intelligence, combined with a new biotechnology-based camera, allows it to detect pedestrians on the road 100 times faster than current car cameras. This major breakthrough in computer vision, made by researchers at the University of Zurich, could significantly improve the safety of driver assistance systems and self-driving cars.
A situation where a pedestrian suddenly steps out in front of a car, usually giving the driver only a fraction of a second to react, is a nightmare for any driver. Current car camera systems can warn the driver or activate emergency braking, but they are not yet fast or reliable enough for use in autonomous vehicles where there is no human behind the wheel.
What the new invention offers
Daniel Gehrig and David Scaramuzza from the Department of Computer Science at the University of Zurich have combined a new camera inspired by biological processes with artificial intelligence to create a system that can detect obstacles around a car much faster than current systems and with less computing power.
Most modern cameras are time-lapse, meaning they take pictures at intervals. Cameras used to assist drivers in cars are typically able to capture 30 to 50 frames per second, and an artificial neural network can be trained to recognize objects – pedestrians, cyclists and other cars – much faster.
While they are getting better, if something happens to standard cameras within the 20-30 milliseconds between two frames, the camera may see it too late. One option is to increase the frame rate, but that requires more real-time data processing and more computing power, “the new system is a visual detector that can detect objects as fast as a camera that takes 5,000 frames per second, but requires the same bandwidth as a standard 50-frame-per-second camera.
The new cameras take an innovative approach based on a different principle. Instead of a constant frame rate, they have smart pixels that capture information when fast movements are detected.
“This way, there are no blind spots between frames, which allows them to detect obstacles faster. They are also called neuromorphic cameras because they mimic the way human eyes perceive images,” says Scaramuzza, head of the robotics and perception group.
However, they also have their drawbacks, as they can miss slow-moving objects, and their images are difficult to convert into data used to train artificial intelligence algorithms.
The scientists’ innovation is that they are developing a hybrid system that combines the best of both technologies:
The first includes a standard camera that captures 20 images per second, which is a relatively low frame rate compared to current ones. The images are processed by an artificial intelligence system, a so-called convolutional neural network, trained to recognize cars and pedestrians.
The data from the event camera is combined with another type of artificial intelligence system, an asynchronous graph neural network, which is particularly well-suited to analyzing three-dimensional data that changes over time. The detections from the event camera are used to predict the standard camera’s detections and improve its performance.
The prospects are huge
During testing, the team compared their system to the best cameras and vision algorithms available on the automotive market today and found that it detects objects 100 times faster. This reduces the amount of data that needs to be transferred between the camera and the on-board computer, as well as the computing power required to process the images, without compromising accuracy. Importantly, the system effectively detects cars and pedestrians entering the field of view between two consecutive frames of the standard camera, which provides additional safety for both the driver and road users. This is especially important at high speeds. The scientists believe that in the future this method can be further improved by integrating cameras with LiDAR sensors, similar to those used in self-driving cars. Hybrid systems of this type could become a cornerstone for the safety of autonomous driving.
Illustrative photo: pexels-jakubzerdzicki-37116225
