Closed circuit televisions (CCTVs) are practically everywhere – keeping an eye on surroundings for safety and security purposes. Each CCTV captures lots of video and data that are mainly used for incident reporting due to stringent privacy regulations.
What if the massive data captured could be used for better purposes without compromising on privacy? What if it could be analysed to help cities run more efficiently and make them more liveable for residents?
The Digital Living Lab (DLL) of University of Wollongong’s SMART Infrastructure Facility aims to do just that – open and unlock data so that the owner of the CCTV network can access a new source of privacy-compliant and real-time data to make better, informed decisions.
High impact projects are delivered through the research and development of the next generation of smart edge-computing devices and sensors embedding the latest development in artificial intelligence (AI) and interoperability with existing infrastructures. By converting data into knowledge and actions, its projects help cities, communities and organisations to address key social, economic, health, safety, and environmental issues.
The state-of-the art deep learning algorithms developed at DLL are part of its Versatile Intelligent Video Analytics (VIVA) platform. These algorithms are re-trained using transfer learning to adapt to use cases, which are then optimised via TensorRT and deployed in the NVIDIA Jetson edge computers.
“We designed VIVA to process the video feed as close as possible to the source, which is the camera. The video feed is processed by an NVIDIA Jetson. Once a frame has been analysed using a deep neural network, the outcome is transmitted and the current frame is discarded,” said Dr Johan Barthelemy, SMART Research Fellow, University of Wollongong.
Disposing of the frame tackles the privacy issue as no image transmitted and overcomes bandwidth limitation.
VIVA has been adapted for applications such as identifying and tracking wildlife and detecting culvert blockage for stormwater management and flash flood early warnings; firefighters and people during a building evacuation; and tracking of people using thermal cameras to understand people’s mobility behaviour during heatwaves.
Smart pedestrian project in Liverpool
One such project is with the City of Liverpool, 27 kilometres southwest of Sydney in New South Wales. The city’s Central Business District (CBD) was growing fast with a new University of Wollongong campus and new airport being built.
The research collaboration between SMART, Liverpool City Council and industry partners was aimed at improving the efficiency and effectiveness of and accessibility to a range of government services and facilities by significantly adding value to the new Liverpool Civic Place project at the southern end of Liverpool’s CBD.
As more than 30,000 people are expected to commute to the CBD daily with the new facilities, the city needed to know the possible impact to traffic flow and movement of pedestrians, cyclists and vehicles.
Without existing data, the team needed to find a way to gather such information for urban design and traffic management without compromising on the privacy of the people of Liverpool.
For pedestrians, it’s about knowing where they are going, their preferred routes and which areas are congested. Routes that cyclists use and ways to improve bicycle usage are also important. Similarly, vehicle movement and where they stop and park matter in assessing impact.
“To understand mobility within a city, we typically need a fleet of costly and fixed sensors to count only one specific type of traffic, whether it is the number of cars, cyclists or pedestrians. If we need data on different types of traffic and the interactions between those types of traffic, we must do it manually,” said Barthelemy.
Machine learning application built using NVIDIA technologies
SMART has developed for the City of Liverpool a machine learning application based on its VIVA platform to count people, cyclists and cars at key locations using Internet of Things (IoT) technologies.
For training, it relies on four workstations powered by NVIDIA Titan V, NVIDIA Titan RTX and NVIDIA Titan Xp GPUs, as well as six workstations equipped with NVIDIA RTX2060 GPUs to generate synthetic data and run experiments.
Inferencing is done with NVIDIA Jetson Nano, NVIDIA Jetson TX2 and NVIDIA Jetson Xavier, depending on the use case and processing required. Based on a deep learning framework consisting of Pytorch, Tensorflow and Keras, the application is built using NVIDIA TensorRT and NVIDIA DIGITS, which is designed for rapidly training highly accurate deep neural network for intelligence video analytics (IVA) such as image classification, segmentation and object/people detection tasks.
“In addition to using open databases such as OpenImage, COCO and Pascal VOC for training, we also created synthetic data via an in-house application based on the Unity Engine. This synthetic data generation allowed us to generate 35,000 plus images per scenario of interest under different weather/time of day/lighting conditions. The synthetic data generation uses ray tracing to improve the realism of the generated images,” said Barthelemy.
Making IVA at the edge possible
By using computer vision with the NVIDIA Jetson TX2 at the edge, SMART was able to count the different types of traffic and capture their trajectory and speed. Data was captured using the city’s existing CCTV network, eliminating the need to invest in additional sensors.
Patterns of movements and points of congestion were identified and predicted to help in improved street and footpath layout connectivity, traffic management and guided pathways.
The data has been invaluable in helping Liverpool plan for the urban design and traffic management of its CBD.
According to Wendy Waller, Mayor of Liverpool, the city will be using the data to plan future pedestrian and vehicle movements throughout Liverpool to ease congestion, provide better transport options and improve health and safety.
“We traditionally relied on human labour to analyse video and perform surveys, as well as data coming from sensors dedicated to one particular task. The process to get relevant data was time consuming and expensive. Also due to privacy regulations, it was impossible to access data recorded on the CCTV network,” said Barthelemy.
“The NVIDIA Jetson platform is what makes IVA on the edge possible. It is at the core of our VIVA platform for which we received invaluable support from NVIDIA during its development. The ‘code once, deploy anywhere’ was also a key factor to use NVIDIA hardware and software technologies,” he added.
Speeding rescues, saving lives
Another application built on VIVA and developed using the NVIDIA hardware and AI helps speed up the rescue process during an emergency evacuation.
The software can identify firefighters from victims and sends information to first respondents to speed up the rescue process. It can tell the difference between firefighters searching the building from other building occupants, and identify those who need help to evacuate, due to a disabled person.
Better use of rich data
Vast amounts of data are collected by CCTVs everywhere but such data is only used for incident investigation, resulting in more than 95 percent of data being discarded.
Barthelemy believes that VIVA can help local governments, business and communities unlock and open the massive amounts of rich data without infringing privacy.
