Khóa luận tốt nghiệp Hệ thống thông tin: A stolen vehicle locator system via dash cam

Multi-Stage License Plate Recognition Systems The multi-stage method is the most used, which consists of three main steps.. To the best of our knowledge, each attempt makes use of a sing

VIETNAM NATIONAL UNIVERSITY - HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY

FACULTY OF INFORMATION SYSTEMS

LE TRAN BAO NAM DINH XUAN HUNG

VIETNAM NATIONAL UNIVERSITY - HO CHI MINH CITY UNIVERSITY OF INFORMATION TECHNOLOGY

FACULTY OF INFORMATION SYSTEMS

LE TRAN BAO NAM - 18521123 DINH XUAN HUNG - 18520791

INFORMATION ON THE THESIS GRADUATION

GRADING COUNCIL

Graduation thesis grading council, established under Decision No

dated of the Principle of the University of Information Technology.

Finally, I would like to express my sincerest thanks for my Ph.D Nguyen Thanh Binh enthusiastically guided and supported us throughout the study and research process to complete the thesis In addition to teaching and commenting on academic knowledge,

presentation skills, research, and reporting, he also cares about students’ health and

psychology, as well as always listens, shares, and inspires students and motivated us to complete the thesis The knowledge and skills imparted by him will definitely be valuable luggage for our future growth.

Next, we would like to thank the Information Systems Laboratory for facilitating and supporting us during the course of the graduation thesis Besides, we would also like to thank the teachers in the Faculty of Information Systems in particular, and the teachers

in the University of Information Technology - Vietnam National University, Ho Chi

Minh City in general, for teaching us knowledge and skills for us during the past four years of school.

Once again, I would like to express my gratitude to Ph.D Nguyen Thanh Binh and the

teachers who have always accompanied and supported me during my university studies.

Le Tran Bao Nam

Dinh Xuan Hung

TABLE OF CONTENTS

Chapter 1 INTRODUCTION 0x2 1 LoD Motivation he 1

1.2 Objectives

Chapter 2 BACKGROUND KNOWLEDGE AND CURRENT WORKS

2.1 Automatic License Plate Recognition Approaches - - ¿+ +++cec+x+xe+ 5 2.2 License Plate Detection

2.2.1 Overvicw MAA ốất À LÀN ccccỀT LH 8 2.2.2 Edge-Based Methods

3.1 General Processing Flow

3.2 Annotation Tool and Dataset cccceescsseecsessesesesssessesesesescsnesesesesseneseacseseeee 28 3.2.1 Roboflow Dataset

3.1.2 Our DataSe( 22t t2 2H21 re 35 3.3 Solution for License Plate RecognitiON - ¿+ + +E‡+EkEekekekerkrkekerrree 38

3.3.1 License Plate Recognition Processing Flow ‹ - 5555-52 38 3.3.2 Model SeÏ€CtiOT + + kh HT TH HH TH HH 39

4.2.1 Client Dashboard - ¿5:52 22t S* 121 1213 1212101130111 11111 crree 70

4.2.2 Admin Dashboard - - - ¿25+ 2% S*SE2E£E#EEEk2kEkrkrkrkerrrrrkrrerrree 73

CS 77

4.2.4 Database ch re 78 4.2.5 Dashcam + tt Ỳ HH He 84

Chapter 5 RESULTS AND CONCLUSIONS - 55-55 Sccccscccxercrreree 85

5.3 Limitations.

5.4 Future WOFK 5-5 + Set t1 E1 1712101211101 1 T210 H00 T001 0110170 re 88

TABLE OF FIGURES

Figure | Main stages in a multi-stage license plate recognition system [6] 6

Figure 2 Classification of related license plate detection techniques [6] - 9 Figure 3 License plate recognition pipeline with associated techniques [6] 19

Figure 4 Flowchart diagram of the Automation License Plate Recognition process [2]

— 27

Figure 5 General processing ÍÏOW ¿-¿- + - + 2+2 E12 2212181 11111113 kg 28 Figure 6 Roboflow license plate detection dataset OV€TVICW c-ccxxcsereerey 29 Figure 7 Roboflow license plate detection training dataset - - + +©5+<es2 30 Figure 8 Roboflow license plate detection validation dataset -. - <2 31 Figure 9 Roboflow license plate detection testing dafaSet -. :- + + +55+<+5+ 32

Figure 10 Roboflow license plate recognition dataset OVeTVIeW -+- 33

Figure 11 Roboflow license plate recognition training dataset oo eee 33 Figure 12 Roboflow license plate recognition validation daf(aset -‹- 34 Figure 13 Roboflow license plate recognition testing dataset -‹-‹-« ‹ OD

Figure 14 Our license plate detection dataset cccccceseseseseeteseseseseeseseseseeneneeess 36

Figure 15 Labellmg 'TOOlL ¿+ S5 S*9E£E£E£EEEEEEEkEEEEEEkEEEkEEEETH tr, 37 Figure 16 Label file after labeled -¿-¿-+- + + +52 +*+*£E£+#£v£zkzEEkerrkrkrkerrrrrerrree 38

Figure 17 License Plate Recognition Processing FÏOW -cc«ccxsxsserrrexex 38

Figure 18 Object detection using YOLO [56] - (SE 40 Figure 19 YOLOvS5 Network architecture [61] Al Figure 20 A comparative plot of the performance of the YOLOvS family [57]

Figure 21 Network architecture of VGG16-SSD [63].

Figure 25 Create configuration file 46

Figure 26 Train model 47

Figure 27 Convert model to TF Lite and quantization it 47 Figure 28 Result from the quantization model - 2+ 2+5 +£+z++++£+se++zsz++ 48 Figure 29 Images.zip folders S†TUCfUTG ¿+ +55 +2£ 22+ 2*+E2*£*t£t£zxrxerrrersrsree 49 Figure 30 Install Tensorflow object detection dependencies - - ¿5-5 «e5+ 50 Figure 31 Create Labelmap and TFReCOrd ¿-¿- + 25+ ++++2£+£££z£tzxzkseerersrerke 51 Figure 32 Pipeline configuration of SSD MobileNetW2 5< + sxsxssrexsxex 52

Figure 33 Train model c.ccccccssssesesecsessseseseeseseseseseesescseseeseneseseseeeeseseseseeneesssseeeeeneaess 53

Figure 34 Convert model to Tensorflow Imodel - - + +6 2 + +*££x+xsEvxerxrxexe 54 Figure 35 Convert Tensorflow model to Tensorflow Lite model -‹- 54

Figure 36 Get list of all images in train fỌder - ¿+52 < 55s £+s+x+eesezszscx+ 55 Figure 37 Representative images - ¿E112 131 1 121 1 1111 11210101010 vi 56

Figure 38 Convert to Tensorflow Lite model with representative images 56 Figure 39 Result of YOLOv5 quantization model - - + scs+++s+c+<e+szsz++ 57

Figure 40 Dataset evaluating of license plate detection and license plate recognition.57

Figure 41 Frame1000 License Plate Detection Evaluating with SSD MobileNet V2 61 Figure 42 Frame1000 License Plate Detection Evaluating with YOLOVS 61

Figure 43 Frame382 License Plate Recognition with SSD MobileNet V2 62

Figure 44 Frame382 License Plate Recognition with YOLOV5 -5- 5-52 63 Figure 45 Frame382 License Plate Recognition with SSD MobileNet V2 for both

license plate detection and license plate recOgnitiON ¿ - 5-52 255+++s+c+ss+sz<cs+ 64

Figure 46 Frame382 License Plate Recognition with SSD MobileNet V2 for license

plate detection and YOLOVS for license plate recognition cece 64

Figure 48 Frame382 License Plate Recognition with YOLOvS for both license plate

65

detection and license plate recognition

Figure 49 System Design Ó8 Figure 57 Client Login Page - + - 5+1 t2 2221212121212 011111 xe 70

Figure 58 Client Profile Page 55+ 2221 t2 222111 1121212101110 1 1c 70 Figure 59 Client Home Page - ¿525222322 2E239121 1212121111211 71

Figure 60 Client List Camera Page -¿ - - + 5452k k2 v2 E12 211211 xe 71 Figure 61 Client Create New CaIm€Ta - - + th vn nh ve 72

Figure 62 Client Lost Vehicle List Page -. ¿ - 5c 252 2x+xtstsrrekerrrerrrree 72

Figure 63 Client Request Lost Vehicle Page c.cccccessessssseseseesesesesesesseaesesneneneseaees 73 Figure 65 Admin Login Page c.ccccccscsesessesesesescssesesessesseseseseseeseseseerseeneacseessseeaeaees 73

Figure 66 Admin Profile PPage - - + 2522222212121 13 1212121011111 11 re 74 Figure 67 Users Management Page ¿5:52 S22 2t2tttkrreerirrrrre 74

Figure 68 Lost Vehicles Management Page - ¿+6 2 S+£*£zxztseererxrkrke 75 Figure 69 Lost Vehicle Request Page -. - ¿+2 25222 St2t2Etersrrkrterrrrrrree 75

Figure 70 Cameras Management Page - - ¿2-55 22+ 2+2x+xtxxerrekerrrrxrrree 76

Figure 71 Admin Create New Camera Page ¿-¿- 5c + Stt‡kéEkEEEkekererrkekrke 76 Figure 74 API DoCUIN€HL - - 6 S522 E#EVEEEEEEEEEEEEEEEEEEEEEEEEERrkrkrkrkrke T7

Figure 75 ERD diagram of Stolen lost vehicle locator system -¿-« + <<+ 78 Figure 77 Dashcam haT(ÌWATC - - S1 EE121 1E 131 1012111 111 H0 HH HH ii 84

Figure 78 User login into system oo ccc escesesesesssseescseseeesesesessesessseersesneaeseensessseaees 85 Figure 79 Home page after user logged into system - - - 252 55+++c+c+cssezece+ 85 Figure 80 User create a new lost vehicle request c.c.ccccesseseseeeeseeseseseeeseseseeneneeess 86 Figure 81 User's list of lost vehicles - - ¿+55 SS+‡EEEEk+kEEEEEEkrkekerrkrkerke 86

LIST TABLES

Table 1 Comparison speed inference between models on Raspberry Pi for license plateCeteCtiON ĐT 4134 Ý 58Table 2 Comparison speed inference between models on Raspberry Pi for license platedetection and license plate T€CÒTIẨIOTI - - 5 s1 931991193 91 9119 1 2v vn rệt 59Table 3 Mean average precision comparison of license plate detection - 60Table 4 Mean average precision comparison of license plate recònition 62Table 5 Recall COTTDATISOH - 6 5 2E 1625191018511 11 911 11 91 210 H1 hi HH nh nh nh rg 66Table 6 Table 0: 0 Ố.ồ ố 79Table 7 Table of TỌ€ 26 c1 2319911231 1131191 931 1 vn HH TH HH HH rc 80Table 8 Table uáv 0n 80Table 9 Table Camera Detected ResulÏ( - - - 2 + 31 +39 viec 81Table 10 Table of lost vehicle reQU€SES 2< 652 S2 E222 E£2EEEEEEEErrrrirree 82

10 cases of property theft, there will be 4 cases of motorbike theft with the rate of 40%[1] and the rate of investigation to find a stolen vehicle is very low, less than 30% [2].This type of crime tends to increase in part due to the complicated and prolongedepidemic situation, which leads many businesses to suspend operations, many people tolose their jobs, and fall into economic difficulties.

However, the epidemic is not the only cause of the increase in this type of crime Manysubjects are lazy to work, have good conditions, have the ability to find jobs, work ontheir own to generate income to serve themselves and their families, but with a sense ofbeing lazy to work, preferring to enjoy the results other people's labor but commit acts

of stealthily appropriating other people's property Considering the consequences caused

by property theft, it can be seen that: all acts of infringing upon the property of othersneed to be detected and dealt with strictly to limit the impact on social order and security

as well as the anxiety of the people

site installation by workers) Moreover, currently, in Ho Chi Minh City's districts,robbery hunting teams have also been established to assist people in finding stolenvehicles, but most of these teams are based on vehicle search operations on the locatormounted on the vehicle, but for the vehicles without the locator, the search is almost

impossible

€.

e Report stolen vehicle at police stations

o Advantages: Official solution of the Ministry of Public Security of

Vietnam with the highest reliability in the solutions

o Disadvantages: The process of reporting and finding stolen vehicles is

complicated and time-consuming

e Attaching a locator to a vehicle

o Advantages: Real-time vehicle tracking

o Disadvantages: High cost, battery of motorbike are easily damaged

e Report stolen vehicle to robbery hunting teams

o Advantages: The process of reporting and finding stolen vehicles is done

of Vietnam and the Department of Transport, by the end of December 31, 2021, thepercentage of active transport business vehicles with cameras installed reached morethan 81% (103,000) vehicles out of a total of 126,000 vehicles) In which, passenger

cars with 9 seats or more reached 100%, fixed route passenger cars 91%, contract cars

69%, container trucks 82% and tractors 78% [3]

Car dashcam market research report on Shopee, Tiki, and Lazada e-commerce platformsfor sellers from January 2022 to December 2022, conducted by Metric.vn, shows therevenue of dashcams transactions on the e-commerce platform reached VND 3 billion in

12 months and increased by more than 89.8% compared to the last quarter [4]

Through the above research data, to solve the problem of motorcycle theft, our teamproposes the solution "A Stolen Vehicle Locator System Via Dashcam"

e Advantages:

o Real-time notification of stolen vehicle location after being detected by

dash cam

o Easy to use system.

o Can be easily integrated with existing modules and solutions

o The process of reporting and finding stolen vehicles is done quickly

e Disadvantages:

o Dashcam owners privacy issues

o The success rate of finding a stolen vehicle depends on the number of

cameras in the network

o Trade-off between precision and speed in license plate detection

o Existing dashcam can not be combined with our module

1.2 Objectives

e Build a complete system that responds to the described process.

e The stolen vehicle image and location will be stored on the cloud and users can

access the information on the Web platform

e Fast speed of license plate extraction, high accuracy, intuitive Web interface, easy

for users to get used to and use

® Deploy network architecture YOLOv5 and SSD MobileNetV2 on embedded

computer.

e Testing, evaluating, and comparing two different detection models are SSD

MobileNetV2 and YOLOVS to choose the right model

Chapter 2 BACKGROUND KNOWLEDGE AND CURRENT WORKS

2.1 Automatic License Plate Recognition Approaches

According to the solution mentioned in Chapter 1, the identification of a stolen vehicledepends on the dashcam recognizing the license plate of the vehicle moving on the roadand then sending the information back to the system Therefore, license plate recognition

is an important key to the solution Currently, there are many methods for license platerecognition, but in general, there are only 2 main approaches:

e Multi-stage License Plate Recognition

e Single-stage License Plate Recognition

a Multi-Stage License Plate Recognition Systems

The multi-stage method is the most used, which consists of three main steps In the firststage, the system detects or extracts the license plate by using traditional computer visiontechniques and deep learning methods with object detection to locate the license plate in

an image Traditional computer vision techniques are mainly based on the characteristics

of license plates such as shape, color, symmetry, texture, etc

The second stage is the license plate segmentation, and the characters are extracted usingsome techniques such as mathematical morphology, connected components, relaxationlabeling, and vertical and horizontal projection However, this stage is not necessary inevery multi-stage Automatic License Plate Recognition system because there are somesegmentation-free algorithms in which this stage is omitted

In the final stage, the characters were recognized using pattern matching techniques orclassifiers like neural networks and fuzzy classifiers However, the main defect ofseparating detection from recognition is its impact on the accuracy and efficiency of theoverall recognition process The recognition process is affected by the detection processsuch as flaws in the bounding box prediction For example, [5] if the license plate is

missing a part in the detection process, it will affect to decrease the overall accuracy of

the recognition process For that reason, in the multi-stage method, achieving satisfying

results in each stage is very important Figure 1 shows the main processing stages in amulti-stage plate recognition system

Stage 1 Stage 2

License Plate Detection

Figure 1 Main stages in a multi-stage license plate recognition system [6]

Through the strengths of multi stages in license plate recognition system, our teamdecided to use this approach to solve the problem in A Stolen Lost Vehicle LocatorSystem via Dashcam

b Single-Stage License Plate Recognition Systems

There have been successful attempts lately at single-stage processes, while most of [7]

the current research on license plate identification has been concentrated on multi-stagemethods To the best of our knowledge, each attempt makes use of a single deep neural

network that has been trained for end-to-end localization, detection, and recognition ofthe license plate in a single forward pass Recognition of license plates can be regarded

as a particular case in point of object detection These models can utilize the fact that

license plate detection and recognition are to a great extent linked, just like single-stageobject detectors [5] As a result, models can share parameters and require fewerparameters than a standard two-stage model They may therefore be quicker and moreeffective than a comparable two-stage technique [8], Li et al made the first known effort

at this [9] As a feature extractor, their method makes use of the convolutional neuralnetwork model VGG16 [9] The license plate covers a lesser area in a normal image,therefore they adjusted VGG16 to only use two pooling layers as opposed to five.

Following that, a Region Proposal Network (RPN) is supplied with the output from the

feature extractors [10] They have made alterations, such as switching from the standard

3 x 3 filters to two rectangular convolution filters This is done to more effectively takeuse of the higher aspect ratio of license plates and the superior performance of

rectangular over square filters These filters then combined the locally collected features

to maintain local and contextual information that aids in the classification stage of licenseplate data The categorization of a license plate as present or absent and box regressionare then performed on these concatenated feature maps using different sets ofconvolutional layers Then, this RPN received end-to-end training Only the RPNgenerated 256 anchors were randomly picked from the enormous number of suggestions,and the losses were determined The proposals were non-maximally suppressed duringtest time to choose only 100 alternatives with higher confidence According to that idea,regions are different sizes and subject to ROI pooling [11] After RoI pooling, the resultsare sent to two other sub-networks, the other license plate recognition network and thelicense plate detection network A network with two outputs and full connectivity is thelicense plate detecting network One for the likelihood that the Rol is a license plate andanother for the coordinates of the bounding box The characters on the license plate arerecognized by the license plate recognition network They have modeled this as asequence labeling problem using bidirectional RNNs to avoid character segmentation

Xu et al [5] offered a similar strategy They have utilized a streamlined ConvolutionalNeural Network (CNN) with 10 layers for feature extraction rather than directlyemploying an existing network like VGG16 [9] A CNN sub-network has been trained

to directly predict bounding boxes RoI pooling [11] was used to provide the output ofthe feature extractor's layers 1, 3, and 5 to several classifiers Because the receptive field

sizes of each layer vary, they have used outputs from numerous layers [12] This hasmade it possible to detect license plates at various camera distances They took advantage

of the fact that license plates have a set number of characters instead of using a singleBidirectional Recurrent Neural Network (BRNN) as in [8], which employed simplerclassifiers for each character in the license plate They were able to develop a model that

is substantially shallower than other single and multi-stage techniques because of the use

of individual classifiers and less complex feature extractions To the best of ourknowledge, their model is relatively straightforward, yet despite this, it is currently themost accurate model for detecting license plates

2.2 License Plate Detection

2.2.1 Overview

A license plate is generally defined as "a metal or plastic plate attached to avehicle that helps to identify them uniquely" However, a machine cannotunderstand this definition A definition that a machine can comprehend isnecessary in order to detect a license plate A license plate can be stated as "arectangular area of a vehicle with a high density of horizontal and vertical edges"[7] in light of its characteristics Numerous algorithms have been proposed toaddress the license plate detection problem based on these features, some of

which are based on deep learning and others on traditional computer visionmethods Figure 2 shows a classification of the license plate detection techniques

that are used in existing methods Table 1 compares those localization methods

along with the benefits and the limitations of each method

| License plate detection |

| Traditional computer vision techniques |

| Edge-based | | Color-based | Texture-based | Character-based | | Statistical | Deep-learning

Vertical Edge : Region-based Support Vector

Detection Genetic Algorithms Scan-line Techniques Approach Machine (SVM) CNN

(eg: using Neural

Networks

Detection Intersection “` Adaptive Mean (eg: YOLO

Scale-space Analysis shift (CAMShift YOLO9000) Mean Shift Sliding Concentric eslalaL ———————

Algorithms Windows Discrete-time CNN

Figure 2 Classification of related license plate detection techniques [6]

2.2.2 Edge-Based MethodsMost research has relied on edge-based techniques for license plate detectionbecause every license plate is rectangular and has a predetermined aspect ratio.Since the license plate color is distinct to the color of the vehicle body, theboundary of the license plate appears as an edge in the image There are two kinds

of edges in an image as horizontal and vertical edges When two horizontal edges

are concerned it is known as horizontal edge detection and for two vertical edges,

it is called vertical edge detection Several studies have used Sobel filter for edgedetection [13].The Sobel filter has two 3 x 3 convolutional matrices and one isdedicated for vertical and the other for horizontal edge detection Ease of use isone of the key advantages of this method One of the major drawbacks of thisapproach is its responsiveness to noise

Only the vertical edges were extracted using a Sobel filter in [14], by Luo et al.and Sarfraz et al However, relying just on the horizontal lines can produce

inaccurate results because the vehicle's bumper can be mistaken for the licenseplate's horizontal border A technique to create a few candidate rectangles in theimage using vertical edge detection was put forth by Sarfraz et al [14] Thelicense plate was then identified as the candidate rectangle with the matchingaspect ratio Even under various lighting setups, this approach has a stated successrate of 96.2% In [15] a faster vertical edge detection technique than the Sobelfilter was suggested According to the authors, the new algorithm extracts license

plate information in the later stages of processing and is 7-9 times faster than theSobel filter

A new technique for extracting license plates has been created by Heo et al [16]

by combining an edge-density mapping algorithm with a line-grouping algorithm.The edge-density map recognizes the area that is highly dense with lines as thecandidate plate, while the line grouping method extracts the line segments and

groups them at the license plate boundary They have demonstrated a 99.45%success rate for the new algorithm [17] discusses a block-based approach for

extracting license plates By taking into account the areas with a high edgemagnitude and a variance, they have located the location of the license plate.Their strategy works well with moving objects and photos with hazy plateboundaries

The conventional method uses the Hough Transformation to locate the straightlines in the images This method has the advantage of recognizing lines up to a

30-degree inclination However, the technique uses up more time and memory

Additionally, boundary deformations have a significant impact on Houghtransformation Because of this, it is not applied to license plates with any unclear

boundaries In order to get quicker results and greater accuracy of 98.8%, Duan

et al [18] suggested a new method that combines the Hough transformation with

a contouring algorithm Due to their simplicity and speed, edge-based techniques

for license plate detection have been employed in several studies Thesetechniques can't be employed with complicated or fuzzy photos due to their greatsensitivity to undesirable edges

2.2.3 Color-Based Methods

Color-based techniques focus on the distinction between a license plate's color

and the color of the vehicle's background Additionally, outside of the plate zone,the image does not contain the color scheme of the plate and its text You can use

the Hue, Lightness, and Saturation (HLS) color model to categorize the pixels in

an input image in relation to various lighting conditions The HLS model, in

contrast to the Red, Green, and Blue (RGB) model, has categorized pixels into 13

groups and was tested on Chinese license plates The HLS model is neverthelesssusceptible to noise The Genetic Algorithm hasn't been extensively studied as asearch heuristic for determining the color of license plates For instance, [19] is a

preliminary work that uses Genetic algorithm for license plate detection and

requires deeper analysis as future extensions

For the classification of license plates, Zhang et al [19] have developed a

brand-new technique called Gaussian Weighted Histogram Intersection (GWHI) By

comparing the color histograms of two colors, histogram intersection is used tomatch two colors The main issue with traditional histogram intersectiontechniques is their sensitivity to lighting As a result, they modified the traditional

histogram intersection by including a Gaussian function The use of mean shift

segmentation as a reliable method for license plate localization is explored in [20].Using the Mean Shift technique, which is based on the colors in the image, they

have divided the vehicle photographs into regions of interest To identify license

plates in these areas, a Mahalanobis classifier was used Similar to [20], another

study [21] used a region-based strategy to localize the license plates

Wang et al have suggested a method to recognize the color of the license plateusing fuzzy mathematics to compete with the uncertainties in real-timeapplications such as lighting changes The color properties of the image areextracted using the Hue, Saturation, Value (HSV) model Utilizing variousmembership functions, the three elements of the HSV model—hue, saturation,

and value—are mapped to fuzzy sets

The color-based techniques might find bent or crooked license plates However,because they are sensitive to variations in illumination, they are rarely utilized bythemselves for plate detection Additionally, they are based on the features of thecamera that takes the pictures Additionally, they produce inaccurate results if theimage includes additional regions that have the same color as the license plate

As a result, to get precise findings, these procedures are frequently paired with

another methodology

2.2.4 Texture-Based Methods

The foundation for license plate detection in texture-based approaches is thepresence of characters on the plate It causes a frequent color change on the licenseplate because of the significant color difference between the plate and itscharacters The characters and the plate background therefore stand out clearly in

a greyscale image As a result, the plate region's surrounding pixels have a special

distribution of intensity Additionally, the color transition creates a high edgedensity in the plate region

For license plate detection, they employed scan-line techniques in [22] Thesetechniques are based on the observation that a grey-scale image lacks all evidence

border details Using Vector Quantization, Zunino et al [7] have introduced a

unique localization technique (VQ) While several research took into account

candidate characteristics such edges and contrast, VQ approaches took intoaccount the actual content of the license plate The authors tested in a real-timeindustrial application and reported a detection accuracy of 98% Frequently, thetexture of the supplied image is inconsistent due to the license plate, revealing its

presence in the picture In light of this, Anagnostopoulos et al [23] haveintroduced a novel segmentation method for detecting license plates calledSliding Concentric Windows (SCW) By making use of the abnormalities in thetexture of the image, it was used for the quicker and more precise detection of theplate regions They reported a license plate detection accuracy of 96.5% SlidingConcentric Windows and the histogram method are combined in [24] to give asimilar methodology However, by improving total detection accuracy, thisapproach produced better outcomes

The Gabor filter is frequently utilized in texture analysis The capacity of a Gabor

filter to examine texture in infinite dimensions and scales is a significant benefit.[25] provides a description of a technique for license plate localization utilizingthe Gabor filter For large-scale picture analysis, this method is time-consumingand less effective A method for detecting license plates that uses the WaveletTransform was presented by Hsieh et al [26] The Wavelet Transform's four sub-bands (or sub-images), which are denoted by the letters LL, HL, LH, and HH Hand L stand for high and low frequencies, respectively The original image ispresent in the LL sub-band after being put through a low-pass filter, while theremaining information is gone While LH sub-band has the qualities in thehorizontal direction, HL sub-band has those in the vertical direction Theprocedure in [26] was broken down into three steps A binary image is subjected

to the Wavelet transform using a Haarscaling function in the initial stage The LH

sub-band is then used to locate a reference line with the greatest horizontal

variance The candidate regions are then extracted using the reference line, and

ultimately, the extracted candidates are used to precisely find the license plate.The reported accuracy of detection was 92.4%

Another Wavelet transform-based approach is given in [27] However, theyextracted features using the HL subband and then confirmed the features usingthe LH subband by checking to see if a horizontal line surrounded the feature.They claim that the localization procedure was 97.33% accurate

It is a major benefit to use texture-based approaches that they are all resistant tolicense plate deformation However, these techniques need complicatedcalculations and perform badly when dealing with complex backdrops andvarious lighting situations

2.2.5 Character-Based Methods

Another method for detecting license plates involves scanning an image to see ifany characters are present and then locating them These techniques fall under thecategory of character-based methods and look at the character-filled region as apotential plate region An approach that extracts every region in an image thatresembles a character has been suggested by Matas and Zimmermann [28] Then,those extreme locations are classified using a neural network classifier, and if anylinear spatial arrangement is discovered, it is assumed to be a potential locationfor the license plate This approach is said to be resistant to various lighting setupsand viewing angles, and the reported detection accuracy is 95%

Draghici has horizontally scanned the image in [29] to look for any repeatedcontrast changes that are at least 15 pixels long This method is based on threemajor premises: (1) that the backdrop and characters have a suitable amount of

(3) that each character has a minimum vertical dimension of 15 pixels The

minimum vertical size measurement, on the other hand, is dependent on the

camera's specifications and needs to be calibrated for every hardwaremodification In outdoor settings, the reported detection rate for this method is99% A method to identify the character region has been put forth by Cho et al.[30] using the character width and the contrast between the backdrop of thelicense plate and the characters They claimed a detection rate of 99.5% and used

the inter-character distance to extract the precise plate region Character-basedapproaches also have the advantage of being robust to license plate rotations Ifthere are additional texts in the input image, these methods can be time-consuming and error-prone

2.2.6 Statistical Classifiers

Several researches have trained cascade classifiers for license plate detectionusing Haar-like features with Adaptive Boosting (AdaBoost) [31], proposes a

cascaded classifier with AdaBoost training that is decision tree based Since

statistical features make the process simpler, they combined statistical and like features for training A detection rate of 94.5% was attained using thisimproved algorithm in a variety of lighting situations and viewing angles Astrategy for detecting license plates that is comparable to [31], has been given in[32] Additionally, they have chosen local Haar-like features and employedstatistical characteristics for simplicity to train the cascade classifier usingAdaBoost learning They reported a 93.5% detection rate

Haar-A new technique for object detection based on color texture has been suggested

by Kim et al [33] and shown with a system for locating license plates By using

a Support Vector Machine (SVM)-based method for identifying plate regions,

they have expanded the prior studies [34] on texture categorization The SVM

uses a region's color and texture to determine whether it is a license plate or not.

When compared to earlier methods for texture classification, the SVM-based

method was substantially more reliable and effective Following the classificationstage, each pixel provides a probability or score for whether or not it belongs tothe plate region The continuously adaptive mean-shift technique is then applied

to these scores to predict the bounding box (CAMShift) This SVM and

CAMShift combination offers a high detection rate with effective processing

2.2.7 Deep-Learning Techniques

Deep learning neural networks have largely supplanted most statistical methods

in computer vision systems in recent years due to their excellent object detection

accuracy Numerous studies on the detection of license plates have employed

various forms of neural networks in recognition of this feature Selmi et al

presented a localization technique in [35] that makes use of a Convolutional

Neural Network (CNN) Two key processes in the license plate detection stagehave been followed in their study To reduce the noise and recover the finer parts

or details from the input image, preprocessing techniques were used in the firststage The following step involved extracting potential bounding boxes for theplate region and classifying them with a CNN classifier as either a license plate

or not The experiment was conducted using the Caltech data set, and the resultsshow that the recall and f-score accuracy are, respectively, 93.8% and 91.3% Zou

et al [36] offered yet another CNN-based investigation They have employed twoCNNs—the shallow CNN and deep CNN—that have been completely trained forlicense plate recognition In order to lower the computational expense, shallowCNN was utilized to remove the majority of the background regions from theimage The license plate from the remaining locations was then detected using themore potent deep CNN Finally, they located the precise plate region using non-

maximum suppression (NMS) The experimental outcomes demonstrated

above-average accuracy with a lower computing cost

Modern, real-time object detectors with great success, such as once (YOLO) [37], served as an inspiration for the license plate identificationmethod in several recent ALPR investigations The cutting-edge YOLO objectdetector has been utilized in several experiments to find license plates [38], and

You-only-look-[15] In [38], Laroca et al developed an effective and reliable localization systemusing the YOLO (version 2) [39] object detector For vehicle detection and licenseplate detection, they deployed two different CNNs With respect to both theSegPlate (SSIG) [40] dataset and the Federal University of Parana (UFPRALPR)dataset, this work has produced encouraging findings when compared to previous

approaches

Numerous experiments on the detection of license plates in confined spaces haveproduced noteworthy findings For instance, Gee-Sern et al [38] have suggested

a novel technique for localizing license plates in the wild utilizing YOLO and

YOLO-9000 (YOLO-2) Due to the shifting weather and lightning, finding a

license plate in the wild might be difficult The direct use of YOLO detectors

exhibits very poor performance in license plate detection despite their outstandingperformance in other object detection tasks such as face recognition As a result,

they have altered the grid sizes and bounding box parameters of the YOLO andYOLO-2 models In a variety of environmental settings, including daytime,nighttime, and moist environments, this approach has demonstrated positiveoutcomes.

A methodology comparable to the work done in [38] has been published by Xie

et al in [41] The original YOLO framework has been enhanced to handle

multi-directions; hence, the term MDYOLO framework The only details provided by

the original YOLO framework are the object's height, breadth, and center

coordinates However, the MDYOLO model also offers details regarding the

angle of rotation for any specific license plate

Through the strengths of deep learning in license plate detection, our teamdecided to use this method to solve the problem in A Stolen Lost Vehicle LocatorSystem via Dashcam

2.3 License Plate Recognition

2.3.1 Overview

This stage, which is the second in a multi-step automated pipeline for readinglicense plates, is in charge of doing so after the detecting stage has located theplate This is a particular instance of optical character recognition that takes intoaccount specific elements of the license plate For instance, the font and color ofthe license plate are subject to tight regulations in many nations, and they aretypically chosen to be legible The license plates do have a few peculiar problems,but [42] For instance, because the photograph was taken outside, the systemdesigners had to take the effects of the weather, uneven brightness and fluctuatingambient light into account They may still be rotated or damaged even thoughthey have a regular license plate The recognition pipeline of a typical ALPRsystem is shown in Figure 3, along with potential strategies that could be used ateach stage Nevertheless, based on the chosen approaches, some of the choresmay be skipped

Correcting License Plate

Rotations

Bi-linear Transformation h ' Local Thresholding '

Line Fitting Methods

Gray Level Transformation

Character Segmentation Character Recognition

Using Pixel Connectivity Using Raw Pixel Data

Using Prior Knowledge

Using Deep Neural Networks

Using Deep Neural Networks

Figure 3 License plate recognition pipeline with associated techniques [6]

2.3.2 Pre-Processing Techniques

Before character segmentation and recognition to meet specific issues in licenseplate recognition, a number of pre-processing operations are carried out.Examples of rotation methods that have been applied in related studies includeline fitting methods, least square-based methods, and bilinear transformations[43] The image is binarized before segmentation in several traditional machinevision-based approaches for character recognition Compared to images ingreyscale or color, the method makes it simpler to distinguish the pixels thatbelong to the characters in the image To avoid the characters mixing or mergingwith the license plate frame in the binary image, which makes it impossible tosegment [44], the threshold for this binarization must be accurately established

Image enhancement methods including noise removal, histogram equalization,

contrast enhancement, and grey level can be used to define the threshold value

2.3.3 Character Segmentation

a Overview

In many optical character recognition methods, segmenting the characters comesbefore classifying them Techniques for segmenting the characters on a licenseplate take into account the fact that the backdrop and characters should bedifferent colors This distinction is made simpler by binarization of the image,which gives the foreground (character) and background pixels opposing "colors."Table 2 contrasts the segmentation techniques currently in use while taking intoaccount both their benefits and drawbacks

b Character Segmentation Using Pixel Connectivity

A straightforward method for character segmentation is pixel connectivity [45].Here, the connected pixels are identified by labels, and if several labels are presentfor an object with a specific size or aspect ratio, the pixels are retrieved ascharacters Pixel connectivity-based techniques have the drawback of failing withdamaged characters or when characters are linked as a result of binarizationthreshold selection However, techniques based on pixel connection arereasonably easy to develop and resilient against rotated license plates Thepipeline for license plate recognition is further made simpler by the lack of a needfor pre-processing to account for license plate rotation

c Character Segmentation Using Projection Profiles

The fact that the character and background pixels in the license plate image afterimage binarization have the opposite colors is used by projection profilesmethods Typically, horizontal projects are utilized to extract the character after

vertical projections are used to determine the character's beginning and endingplaces [44] Project-based approaches, however, are sensitive to image noise and

quality As a result, the pre-processing phase of the recognition pipeline must

contain a denoising stage The projection-based approaches are nonetheless

robust to rotations and independent of character placements, despite the fact thatthese methods yield lower robust values than pixel connectivity-based methods

for rotations

d Character Segmentation Using Prior Knowledge

For character segmentation, prior information about the license plate is employed,such as the aspect ratio of the characters and the ratio of different colored pixels

in the image To locate the starting position of the character and the opposite forthe ending point of the character, Busch et al [46] scanned the binary imagehorizontally to find the spot where the pixel ratio of background pixels to

character pixels exceeds a predefined threshold Paliy et al [47].'s straightforward

method involved scaling the retrieved license plate to a predefined size with the

characters' positions already predetermined Prior knowledge-based approaches

are frequently straightforward to implement, but they are typically restricted to

the areas in which they were intended to function and do not generalize in other

situations

e Character Segmentation Using Deep Learning

CNN is used for the computer vision job in neural network charactersegmentation, which is a more recent method [38] The CNN is given a localizedlicense plate as input, and as an output, it produces each character's boundingboxes The CNN execution, however, uses more time and resources than theconventional computer vision-based algorithms, depending on the dataset

pipelines, implicit character segmentation has been used in place of explicit

character segmentation, which reduces the number of parameters and lowers thecomputational cost [48]

2.3.4 Character Recognition

a Overview

Fixed-size inputs to the learning model are necessary for many classificationtechniques As a result of the variable output size from the segmentation stage,the input segments are rescaled before classification Every segment is identified

as belonging to one of the possible values because the number of characters, theirrelative positions, and their potential values are typically known There are threeways to approach this: (1) directly compare all the raw image data's pixel values

to predefined templates; (2) extract features using various image processing andmachine learning techniques before classifying the segments; (3) classify

segments using deep learning techniques

b Template And Pattern Matching Techniques

Given that characters on license plates typically have a known font and charactersize, one popular method of classifying characters is to use template matchingtechniques Binarized images are frequently used with template matching Eachpotential character has a predetermined template, and each segment is compared

to each template to determine which one is the most similar Here, metrics likethe Mahalobian distance, Jaccard value, Hausdorff distance, Hamming distance,and normalized cross-correlation are used to gauge similarity Although thesetechniques are easy to use, their nature prevents them from being broadlyapplicable to various types of license plates Furthermore, this method ischallenging to apply when there are numerous typographically viable templates

Additional templates must be stored to handle character rotations, which adds tothe computation time and processing memory requirements.

c Character Recognition Using Feature Extractors

In general, not every pixel is necessary to identify a character By lowering thecomputational costs, feature extractors are used to separate simpler features fromthe images Some feature extraction methods can extract features that are resilient

to image noise and rotations [49] These techniques create a feature vector fromeach segment using a transformation, which is then classified using a machinelearning model Eigenvector transformation [48], Gabor filter [50], and Kirshedge detection are a few of the feature extraction methods In order to categorizethe extracted features, machine learning models like SVM [51] and HiddenMarkov Models (HMM) [52] are used

d Character Recognition Using Deep Learning

The benefit of using neural networks is that they can function independently asfeature extractors and classifiers when given the raw pixel data This task has beenaccomplished using a variety of neural network architectures, including discrete-time cellular networks, probabilistic neural networks (PNN), and basic multi-layer perceptrons [23] But many recent studies have employed CNN, which hasdemonstrated great promise in a variety of computer vision tasks Directlyutilizing object detection-based techniques [48] like YOLO is another modernstrategy Deep learning-based approaches offer better accuracy overall, despitebeing computationally more expensive than alternatives like template matchingand statistical feature extractors

e Constraint-Based License Plate Recognition Models

Chang et al [53] have proposed a constraint-based license plate recognition

system for Taiwanese license plates Figure 4 illustrates how the model, which

consists of two sub-modules for license plate localization and numberidentification, is based on multi-stage plate recognition disciplines For traffic lawenforcement, the system is intended to work in tandem with an event detectionsystem Since Taiwanese license plates come in four colors white, black, red, and

green, they developed a localization method that uses a color edge detector that

is only sensitive to the edges of black-white, red-white, and green-white An RGBimage is provided as input to the plate-locating module, and color edge detection

is used to produce an edge map The RGB image is then converted into HSI (Hue,Saturation, Intensity) space because the HSI model is independent of lighting,

shading, viewing angles, and surface orientations Fuzzy techniques were used to

extract the license plate region, and the corresponding edge, hue, saturation, andintensity fuzzy maps were produced The degree to which a cell is occupied by alicense plate is shown in the cell entry of a fuzzy map The plate region is thenlocated using the integrated map, which has been combined using a two-stagefuzzy aggregator

The pre-processing and recognition modules are the two main components of thesubsequent phase, which is number identification They have employedsequential noise removal, connected components, and binarization with variablethresholding pre-processing techniques A character segmentation stage comesafter the three main steps of optical character recognition Characters arecategorized as either numerical or alphabetical, and then topological sorting andself-organizing (SO) recognition based on Kohonen's SO network are used toidentify them They have, however, noted some difficulties in separatingcharacters, such as 8 with B and 0 with D As a result, they created an ambiguitycharacter set, and if the system discovers any character in it, it will perform an

additional comparison Although the suggested algorithm is only applicable toone country, it could be expanded to include other types The system's detectionand recognition rates were both 97.9% and 95.6%, respectively, with a reportedoverall success rate of 93.7%.

[18] suggests a different strategy for developing an automatic license platerecognition system that would read Vietnamese license plates at toll booths For

pre-processing, plate detection, character segmentation, and recognition, thesystem consists of four modules The pre-processing stage improved the edgefeatures of the input image because boundary features for license plate detection

were taken into consideration Algorithms like grey-scaling, normalization, andhistogram equalization are used during the preprocessing stage It converts theinput image to greyscale and applies the Sobel filter to produce an edge map Theimage is then binarized using a local adaptive thresholding algorithm before being

fed to the detection module Here, license plate recognition is accomplished using

a boundary-based strategy Although Hough Transform is a widely usedalgorithm for edge detection, its application in practical situations is limited byits lengthy execution time and intricate computation To achieve quicker and moreprecise results, they combined the Hough Transform and Contour algorithm.However, a verification phase is used to determine whether the candidate is alicense plate or not in order to prevent false detection Vertical and horizontalprojection were used in the character segmentation module, and HMMs were used

in the character recognition module License plate detection, segmentation, andrecognition modules have reported success rates of 98.76%, 97.61%, and 97.52%,respectively Thus, a 92.85% overall success rate is reported

[54] discusses a preliminary study to identify Saudi Arabian license plates The

input images are initially converted by the system into greyscale images before

being fed to the plate extraction module Vertical edge matching, filtering, anddetection of vertical edges are the three steps in the plate extraction process TheSobel and Prewitt edge detectors were used to perform vertical edge detection

because most vehicles have more horizontal than vertical lines, preventing thealgorithm from becoming unnecessarily complex After that, the seed-filling

algorithm was used to filter out the unwanted objects The potential plate regions

are then extracted by comparing them to the Saudi Arabian license plate's typicalheight to width ratio In order to eliminate any distractions from logos and bolts,the upper portion of the license plate has also been removed during thesegmentation stage The image is then vertically projected, binarized, andsegmented They have normalized images to a size of 40 40 and recognized thecharacters using template matching techniques after segmenting the characters.The detection, segmentation, and recognition accuracies for this method werereported to be 96.22%, 94.04%, and 95.24%, respectively

Wen et al [55] suggest another novel algorithm to identify license plates byremoving the shadow from the plates in the presence of asymmetric lighting Thecharacter recognition and pre-processing algorithms that already exist have beenimproved by this study As a result, the system works well in challengingsituations like rotation, plate variations, and illumination variance They haveregarded the pre-processing stage as their primary contribution They haveutilized adaptive local binary techniques for binarization rather than globalthresholding techniques Thus, for shadow removal, two local binary methods—Otsu and an improved local Bernsen algorithm are used They have employed apixel-based technique called Connected Component Analysis for the license platedetection stage (CCA) The detected license plates are, however, resized to 100

200 pixels and processed for tilt correction in both the vertical and horizontaldirections before segmentation The characters are then all resized to the same

size after segmentation is completed using projection techniques Last but notleast, they employed an SVM classifier for character recognition that used inputfeatures like contour features and stroke direction features like Global DirectionContributivity Density (G-DCD) and Local Direction Contributivity Density (L-DCD) Additionally, this approach is less constrained and more reliable incomplex environments The reported success rates for character segmentation andrecognition, as well as license plate detection, are 97.16%, 98.34%, and 97.8%,

respectively A 93.54% overall success rate is reported

Input RGB image

Transform to HS! model

License plate detection

Color Edge Detection

Candidate license plate

pre-processing

Fuzzy Aggregation

Character Recognition a

Binarization Character categorization

Topological sorting

Self-organizing (SO)

recognition

License plate number

Figure 4 Flowchart diagram of the Automation License Plate Recognition process [2]

Connected component

Noise removal

Chapter 3 OUR PROPOSED LICENSE PLATE RECOGNITION

APPROACH

3.1 General Processing Flow

The ALPR problem has many approaches, algorithms, and processes In this thesis, the

data to be processed is real-time video from dashcam A comprehensive picture of theanalysis process with two main processing parts: License Plate Detection and License

Plate Recognition is shown in Figure 5, while the detailed approach to each problem isexplained below

—— Ỷ

Unlabeled license — _—|

plate detection Annotate image Labeled license Pre Processing Selection Newel — * ae _ wee Evaluation

dataset plate detection _ | Network Model etection Model

| Evaluation |

ee

Selection Neural Build License Plate Apply License Plate

TÊN ”| Network Model | Recognition Model | Recognition Model

Ỷ

Characters were extracted from license

plate

Labeled license

plate recognition

dataset Annotate image

Figure 5 General processing flow

3.2 Annotation Tool and Dataset

For license plate detection we use dataset was mixed from Roboflow dataset and ourdataset

3.2.1 Roboflow Dataset

License plate dataset for license plate detection and license plate recognition we

are using in this project is provided by Roboflow which is a startup company that

provides solutions for Developers in building Computer Vision Applications such

as providing datasets, training and deploying computer vision applications, etc

a License plate detection dataset:

The license plate dataset of Roboflow is collected from different sources

Therefore, it supports a variety of license plates from many different countries, inwhich Vietnamese license plates account for a relatively large number In

particular, this dataset provides 21.000 images for training, 2.000 images forvalidation and 1.000 images for testing

Training Set Validation Set Testing Set

Figure 6 Roboflow license plate detection dataset overview