Intelligent video surveillance can not only record what happens in front of the camera lens, but it can also count people and recognize individuals. In connection with other digital services at an airport, this can consequently help to avoid the formation of large groups of people. Smart algorithms are the basis for this process – and they are currently being trained.
A person with a backpack. A person without a backpack. A person with a red jacket. A person with a green jacket. When Robert Borchers’ team trains algorithms, it does not always seem like Science Fiction. Mostly, it rather resembles the explaining of a painting book to a toddler. “The task we give the algorithm is: recognize regularities and recurring patterns,” explains the team leader for Connected Mobility and Computer Vision Systems at Bosch. In the end, the software should have learned enough to be able to judge independently whether the object it sees is a person. It should not only be able to distinguish individuals from each other, but also from rolling suitcases and vehicles. This training of algorithms is a time-consuming matter. Nevertheless, Borchers believes that it is possible to manage crowds of waiting people at airports more efficiently or entirely avoid them with the algorithms help.
“In the past, video-surveillance was mainly used in order to serve as evidence in court in the case of emergency,” says Borchers. Real-time analysis, however, was difficult. Even though the human eye can grasp sudden changes very well in an otherwise static arrangement, it does have difficulties to focus on pictures or sequences that contain a constant restlessness. “In a mass of surveillance-pictures, it is almost impossible for human beings to keep an eye on all recognizable persons at the same time,” explains Borchers. However, a software which once learned to separate single persons in pictures can theoretically recognize how many people reside in one supervised space within seconds.
Individual analyses thanks to multi-camera-tracking
The possibility to let an algorithm count the passengers at an airport on a real-time basis is, in itself, already quite exciting. Nevertheless, the true potential only reveals itself when the algorithm is combined with other components of the more complex airport assistance system. For instance, with the help of so-called multi-camera-tracking, all surveillance cameras of a terminal or an airport building can be connected – in small airports, these are a few hundred single devices, in big airports at least around 1,000. “An algorithm which has such hardware at its disposal, can assign a well-defined identity figure to every passenger and offer him customized services throughout the whole premises,” describes Borchers the possible fields of application. Thus, an almost infinite number of combined calculations could arise from the mass of passengers: What percentage of passengers who have entered the airport at 10 o’clock made their way to the security check by 10:30 am? What is the average movement speed of all passengers in the terminal? For a passenger that has recently arrived, when is the prospective waiting period at the baggage check-in the shortest? “The data of all movement profiles collected at the airport permit predictions for every individual,” underlines Borchers. “Intelligently used, processes can therefore be massively accelerated.”
Customized notifications for every passenger
Thereby, the expert thinks beyond the standard video analytics. The comprehensive visions which Borchers develops, as well as other connected-mobility-experts from Silicon Valley, deal more with integrated processes. The focus lies on the smartphone. At the same moment when a passenger enters the airport, registers themselves at the system, he or she also scans a virtual Boarding-Pass via a smartphone. From this moment on, the smart algorithm knows from which gate the passenger will be departing from, and, thus, it can calculate the right time when the passenger has to check-in at which place. At the same time, the intelligent video analytics knows the exact position of all other passengers of the same flight within the terminal. “Ideally, when the right time for a passenger approaches to go to the security check, the system can send every passenger an individual message to his or her smartphone,” says Borchers. In addition, the video system will recognize queues at the baggage check-in as more people than usual will be found in the area of the corresponding camera. In such situations, passengers of the concerning flights can be informed via push notifications and, for instance, be given the option to drink another cup of coffee.
The future starts now
“We are currently working on three challenges so that monitoring systems are capable of such performances,” says Borchers. “Software, computing power, and terminal devices.” The software, consisting of algorithms, must still learn to interpret the behavior of a person. Borchers finds it very convenient that Bosch does not have to develop the airport application from scratch, but that they can benefit from proven algorithms, for instance, from the automotive industry. They already know what a person is, and only have to learn the characteristics of airport passengers. Nevertheless, it is computationally intensive, and at this point, Borchers mentions the second current challenge. “The more data that is analyzed at the same time, the more computing power I need,” explains the expert. Not all processors which are required for the evaluation must be installed on site at the airport. “Nowadays, much computing power is sourced out to Clouds,” admits Borchers, “but the rule of thumb is valid: the longer the distance the data needs to cover, the lengthier the analysis.” Therefore, modern camera systems often already have a processor on board which directly carries out the first evaluations. With regard to the hardware, a monitoring system is very dependent on the quality of the connected cameras. Of course, all single devices must be able to connect to each other and form a huge net, a function that is usually a standard feature nowadays. However, also features like high resolution, a smooth day-night-rhythm, and the provided recording solutions can influence how efficient the single devices work within the overall system and how they can be controlled by the algorithm.
Step by step to the intelligent video analytics
Airport managers who would like to implement such systems can soon start the planning. “We make considerable progress in Deep Learning, because our algorithms are continuously learning and, at the same time, the available hardware becomes exponentially faster,” says Borchers. First software-based analysis programs might already be launched in 2018 or 2019. However, according to experts, solutions for airports that function comprehensively and allow multi-camera-tracking, a prediction of waiting times, and direct interfaces to other sensors, probably need about ten more years until a smooth serial production is possible. Indeed: “Such intelligent monitoring systems can be implemented step by step,” admits Borchers. “First of all, an airport manager can test the functionality at one terminal, or he introduces the multi-camera-tracking on the complete airport premises and, a few years later, he subsequently adds the individual smartphone notification system.”
