A New System for License Plate Recognition in Traffic Violation Scenarios in Vietnam

License plate recognition in traffic violation scenarios is a key issue for any traffic management system. In this work, a new system that can detect traffic violations and output the license plate numbers of violating vehicles and evidence was introduced. The system was developed based on image processing techniques combined with deep neural networks. A dataset, which contains thousands of images of traffic vehicles from public cameras in Hue City, was collected. Based on this dataset, three deep learning networks for different tasks in the overall pipeline process were developed. A pretrained-model of the YOLOv5s6 neural network was used to create a model to detect four classes of vehicles, including cars, buses, trucks, and motorbikes with riders. The so-called “Deep SORT” model was exploited for vehicle tracking and traffic violation detection tasks. For license plate detection, a model based on the RetinaFace with backbone MobileNet was trained. Rectification techniques for the detected license plates were applied. Finally, an OCR model for license plate number recognition was finetuned from a pre-trained model with CRNN architecture. Experimental evaluations show that all developed models are very lightweight and have high accuracy. The YOLOv5s6 achieves an mAP of 87.8%; the RetinaFace has a maximal mean square error of only 1.6, and the OCR model attains a very high accuracy of 99.72%. The results of the implementation of the pipeline on embedded hardware, namely the Jetson Xavier Development Kit, show that our system is very computationally efficient. The total computing time is only 41 ms and the use of RAM is less than 3 GB. These results show high potential for practical applications.