Designing the smart locking door by using image processing

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT AUTOMATION AND CONTROL ENGINEERING DESIGNING THE SMART LOCKING DOOR BY USING IMAGE PROCESSING ADVISOR: DR TRAN VU HOANG MINH STUDENT:PHAM TUAN QUANG HUY NGUYỄN NGOC BAO Ho Chi Minh city, Au SKL009730 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY OF HIGH QUALITY TRAINING GRADUATION PROJECT DESIGNING THE SMART LOCKING DOOR BY USING IMAGE PROCESSING Student name: Pham Tuan Quang Huy 18151 Nguyen Ngoc Bao 18151 School year: 2018 – 2022 Major: Automation and Control Engineering Instructor: Dr Tran Vu Hoang Ho Chi Minh city, August 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY OF HIGH QUALITY TRAINING GRADUATION PROJECT DESIGNING THE SMART LOCKING DOOR BY USING IMAGE PROCESSING Student name: Pham Tuan Quang Huy 18151 Nguyen Ngoc Bao 18151 School year: 2018 – 2022 Major: Automation and Control Engineering Instructor: Dr Tran Vu Hoang Ho Chi Minh city, August 2022 ACKNOWLEDGEMENT Primarily, I would like to show appreciation to all the people that supported our team to complete this project The success and final outcome of this project required a lot of guidance from members in the laboratory and I was extremely fortunate to have got this all along the completion of our project work Thanks to all the support and assistance of our seniors and advisors, our team had more confidence to overcome all the challenges and difficulties during the project I also want to show our appreciation to our advisor Dr Tran Vu Hoang – A lecturer of Automation and Control Engineering, who was always supportive and willing to instruct us when facing challenges In addition, I would like to thank to all tutors and advisors in Ho Chi Minh University of Technology and Education general and all tutors and advisors in Department of Automation and Control Engineering in particular, who taught us basic knowledge and major knowledge about the studying field, so that we had enough principal knowledge and experience to apply to this project Moreover, all tutors always gave us a hand when our team needed any advice for the project Despite a careful research and design process, the project may still have certain limitations We hope to receive the feedback from all tutors, further improving limitations From there, we have a stronger basis to put our project into practice In conclusion, we would like to show our appreciation to the members of 18151CLA who supported and gave our team a lot of advice iv ENGAGEMENT All the achievements of this project are not a copy of any documents or research papers All the references of this project are put in the reference session Member of project (Signature) v CONTENTS LIST OF FIGURES LIST OF TABLES ABSTRACT Chapter 1: OVERVIEW 1.1 Introduction 1.2 Objective of the research 1.3 Limitations 1.4 Research methods 1.5 Structure of the project Chapter 2: PRINCIPAL THEORIES 2.1 Face detection model 2.1.1 Pytorch library 2.1.2 Ultra light fast generic face detector 2.1.2.1 Object detection 2.1.2.2 SSD - Single Shot Multibox Detector [3] 2.1.2.3 RFB – Receptive Field Block [5] 2.1.2.4 Ultra light face generic face detector architecture 11 2.1.3 2.2 Tensorflow Lite 12 Face recognition 13 2.2.1 FaceNet [14] 13 2.2.1.1 Triplet loss [14] 14 2.2.1.2 ReLU – Rectified Linear Unit [25] 16 2.2.1.3 Inception architecture [13] 16 2.2.2 VGGFace2 training dataset [15] 19 2.3 Dlib facial landmarks [25] 21 2.4 Tkinter [27] 22 Chapter 3: SYSTEM DESIGN 24 3.1 Design requirements 24 3.1.1 System block diagram 24 3.1.2 Block design on requirements 25 3.1.2.1 Image receiving block and recognition block 25 3.1.2.2 Data block and management block 28 3.2 System design 31 vi 3.2.1 Hardware 31 3.2.1.1 Choosing embedded hardware 31 3.2.1.2 Choosing camera 36 3.2.1.3 Choosing Arduino board 38 3.2.1.4 Relay module 40 3.2.1.5 Choosing LCD screen 41 3.2.1.6 Hardware block and wiring diagram 43 3.2.2 Methods survey 45 3.2.2.1 Survey face detection methods 45 3.2.2.2 Surveying human facial feature extraction methods 47 3.2.3 Operation process 49 3.2.3.1 Face registration 49 3.2.3.2 Face recognition 50 3.2.3.3 Liveness detection 51 Chapter 4: EXPERIMENT RESULT 53 4.1 Environment dataset 53 4.2 Evaluation methods 53 4.3 Performance of the system 53 4.4 System operation and hardware result 54 4.5 System validation 56 4.4.1 Environment and dataset 56 Chapter 5: CONCLUSION 62 5.1 Conclusion 62 5.2 Improvement 62 REFERENCE 63 vii viii LIST OF FIGURES Figure 2.1 Workflow of Pytorch Figure 2.2 Architecture of SSD [3] Figure 2.3 Construction of RFB module by combining multiple branches with different kernels and dilated convolution layers [5] 10 Figure 2.4 The architecture of RFB and RFB-s RFB-s is employed to mimic smaller pRFs in shallow human retinotopic maps, using more branches with smaller kernels [5]10 Figure 2.5 The pipeline of RFB-Net300 The Conv4_3 feature map is tailed by RFB-s which has smaller RFs and an RFB module with stride is produced by operating 2stride multi kernel conv-layer in the original RFB [5] 11 Figure 2.6 Architecture of Tensorflow Lite 13 Figure 2.7 Illustration of FaceNet recognition 14 Figure 2.8 Triplet loss 15 Figure 2.9 Regions of embedding space of negatives 15 Figure 2.10 ReLU activation function [25] 16 Figure 2.11 Inception network architecture [13] 17 Figure 2.12 VGGFace2 pose and age statistic [15] 19 Figure 2.13 VGGFace2 template examples (a) pose templates from three different viewpoints, (b) age templates for two subjects for young and mature ages 21 Figure 2.14 68-facial landmarks [25] 22 Figure 2.15 Fundamental structure of Tkinter program 23 Figure 3.1 System block diagram 24 Figure 3.2 System process 25 Figure 3.3 Input image 25 Figure 3.4 Input image of face detection model 26 Figure 3.5 Output of Ultra light fast detector 26 Figure 3.6 Input of face recognition model 27 Figure 3.7 Output of liveness detection 27 Figure 3.9 First window 28 Figure 3.10 Function window 29 Figure 3.11 Login window 29 Figure 3.12 Add data window 30 Figure 3.13 History window 30 Figure 3.14 Management window 31 Figure 3.15 Jetson Nano B01 Developer kit 31 Figure 3.16 connection ports of Jetson Nano B01 Developer kit 32 Figure 3.17 Pin diagram of Jetson Nano B01 33 Figure 3.18 Pin diagram of Raspberry Pi 34 Figure 3.19 Raspberry Pi 34 Figure 3.20 Camera Logitech C920 36 Figure 3.21 Camera Logitech C310 36 Figure 3.22 Camera Logitech C270 37 Figure 3.23 Arduino Uno R3 38 Figure 3.24 Pin diagram of Arduino Uno R3 39 Figure 3.25 Arduino Mega 2560 39 Figure 3.26 Relay module 41 Figure 3.27 HDMI LCD inch 42 Figure 3.28 HDMI LCD 10.1 inch 42 Figure 3.29 Hardware block diagram 43 Figure 3.30 Wiring diagram 44 Figure 3.31 Hardware SolidWorks design 45 Figure 3.32 Performance of models without mask 46 Figure 3.33 Performance of models with mask 46 Figure 3.34 Face registration process 50 Figure 3.35 Face recognition process 51 Figure 3.36 Liveness detection process 52 Figure 4.1 Performance of the system 54 Figure 4.2 Face recognition result 55 Figure 4.3 Face recognition + Liveness detection 55 Figure 4.4 Hardware design 56 Figure 4.5 Straight face result in good brightness condition 56 Figure 4.6 Result on one side of the face 57 Figure 4.7 Result with mask cover completely 57 Figure 4.8 Result with classes cover 58 Figure 4.9 Result with mask cover incompletely 58 Figure 4.10 Performance in backlit and low brightness condition 59 Figure 4.11 Performance in backlit and low brightness condition with mask 59 Figure 4.12 Result in 2m to 3m distance 60 Figure 4.13 Performance of face recognition and liveness detection 60