Design and implementation of classfication and delivery based on computer vision

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT COMPUTER ENGINGEERING TECHNOLOGY DESIGN AND IMPLEMENTATION OF CLASSFICATION AND DELIVERY BASED ON COMPUTER VISION ADVISOR: TRƯƠNG QUANG PHÚC, M.Eng STUDENT: PHẠM MINH QUÂN NGUYỄN HOÀI PHƯƠNG UYÊN SKL 0 9 Ho Chi Minh City, December, 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF CLASSFICATION AND DELIVERY BASED ON COMPUTER VISION PHẠM MINH QUÂN Student ID: 18161031 NGUYỄN HOÀI PHƯƠNG UYÊN Student ID: 18119053 Major: COMPUTER ENGINEERING TECHNOLOGY Advisor: TRƯƠNG QUANG PHÚC, M.Eng Ho Chi Minh City, December 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF CLASSIFICATION AND DELIVERY BASED ON COMPUTER VISION PHẠM MINH QUÂN Student ID: 18161031 NGUYỄN HOÀI PHƯƠNG UYÊN Student ID: 18119053 Major: COMPUTER ENGINEERING TECHNOLOGY Advisor: TRƯƠNG QUANG PHÚC, M.Eng Ho Chi Minh City, December 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness Ho Chi Minh City, December 25, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Phạm Minh Quân Student ID: 18161031 Student name: Nguyễn Hoài Phương Uyên Student ID: 18119053 Major: Computer Engineering Technology Class: 18119CLA Advisor: Trương Quang Phúc, MEng Phone number: _ Date of assignment: Date of submission: _ _ Project title: Design and Implementation of classification and delivery based on Computer Vision Initial materials provided by the advisor: _ Content of the project: _ Final product: CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) Trương Quang Phúc THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, December 25, 2022 ADVISOR’S EVALUATION SHEET Student name: Phạm Minh Quân Student name: Nguyễn Hoài Phương Uyên Student ID: 18161031 Student ID: 18119053 Major: Computer Engineering Technology Project title: Design and Implementation of classification and delivery based on Computer Vision Advisor: Trương Quang Phúc, MEng EVALUATION Content of the project: Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Approved Overall evaluation: (Excellent, Good, Fair, Poor) Good Mark: 9.0 (in words: .) Ho Chi Minh City, December 25, 2022 ADVISOR (Sign with full name) Trương Quang Phúc HO CHI MINH CITY UNIVERSITY TECHNOLOGY AND EDUCATIO FACULTY FOR HIGH QUALITY TRAINING THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom - Happiness Ho Chi Minh City, January 13, 2023 MODIFYING EXPLANATION OF THE GRADUATION PROJECT MAJOR: COMPUTER ENGINEERING TT Project title: Design and Implementation of classification and delivery based on Computer Vision Student name: Phạm Minh Quân Student ID: 18161031 Student name: Nguyễn Hoài Phương Uyên Student ID: 18161031 Advisor: Trương Quang Phúc, Meng Defending Council: Council 2, Room: A3-404, 3rd January 2023 Modifying explanation of the graduation project: Council comments Editing results Many figures in chapter are reused from other Many figures in chapter are provided sources without providing the related references the related references Visual quality of many figures in chapter is Many figures in chapter are modified to very low and hard to follow improve their visual quality In conclusion: Author should clearly point out which objective have accomplished instead of a general summarization The flowchart of figure 3.6 must have a “Begin” point and a “End” Point in a terminator shape Head of Department (Sign with full name) The conclusion section is clearly pointed out which objective have accomplished The flowchart of figure 3.6 is modified Advisor (Sign with full name) Trương Quang Phúc Students (Sign with full name) Note THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, December 25, 2022 PRE-DEFENSE EVALUATION SHEET Student name: Phạm Minh Quân Student ID: 18161031 Student name: Nguyễn Hoài Phương Uyên Student ID: 18119053 Major: Computer Engineering Technology Project title: Name of Reviewer: EVALUATION Content and workload of the project Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: .) Ho Chi Minh City, December 25, 2022 REVIEWER (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Phạm Minh Quân Student ID: 18161031 Student name: Nguyễn Hoài Phương Uyên Student ID: 18119053 Major: Computer Engineering Technology Project title: Name of Defense Committee Member: EVALUATION Content and workload of the project Strengths: Weaknesses: Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, December 25, 2022 COMMITTEE MEMBER (Sign with full name SUPERVISOR APPROVAL i ACKNOWLEDGEMENTS Over the course of undertaking the project, our group received a plenty of valuable support that incentivize us to overcome all the problems and challenges and end up this quite hard and meaningful project Firstly, we would like to thanks to the School Board of the Ho Chi Minh City University of Technology and Education and Faculty for High Quality Training creating wonderful conditions for me to take my project Secondly, sincerely thank to Mr.Trương Quang Phúc, our advisor who gave us useful guidance and instruction that help us to finish our project successfully From these advices we can improve our project contents and correct the mistakes as well Thirdly, we are grateful to all of the nice classmates of class 18119CLA who contributed to give advice and warm guidance whenever we need the support Last but not least, Due to limited knowledge and implementation time, we cannot avoid errors We look forward to receiving your comments and suggestions to improve this topic In short, we really thank to all people are a part of our achievement Ho Chi Minh city, Friday, December 23, 2022 Student performance Pham Minh Quan Nguyen Hoai Phuong Uyen ii Figure 4.11: Result of labelling each image The pink rectangle is draw around the barcode, class “right” is labelled for it 4.4.3 Annotations for data Additional annotated data will be better when it is used in the training process since, in many circumstances, more input data will aid in the model's learning 84 Figure 4.12: Result of add annotation process We added some annotation methods such as: brightness, blur, noise This some above images are results of annotation process 4.4.4 Training process We are training the Yolov7 model using the data that was generated above Additionally, we employ Google Colab to support the training process because its GPU has a very fast velocity 85 Figure 4.13: Yolov7 dataset training results were successful After successfully training, file “best.pt” and “last.pt” which is result of training process is weight for detecting 4.4.5 The text detecting process by Tesseract OCR model We will utilize the Tesseract OCR model to perform character recognition below the barcode after going through the object recognition process utilizing the Yolov7 model Figure 4.14: The model's performance when tested with the input image 86 Figure 4.15: Accuracy of input image after testing process These images return result of accuracy is 0.94 Moreover, Tesseract OCR model detect all of characters in image according to input image 4.4.6 Interface of delivery app Based on PyQt5’s support for creating an app, we chose it to generate interface of delivery app which display information of the number of parcels, the place, location of AGV vehicle is designed by Python and Qt Designer Software The following is an image of delivery app’s interface As mentioned, we generate an app has three pages are stacked Besides, app also has three buttons to link them At the left column, app has two buttons, one is linked to main page, another is linked to three above buttons 87 Figure 4.16: The interface of delivery app 4.5 Evaluation and comparison 4.5.1 Comparison Yolo with others CNN In this part, we will evaluate the various YOLO object identification algorithms' performance comparison which authors completed on many different devices such as: GPU, CPU, TESLA P100, … Because versions 5,6,7 are the best models of yolo to date, both in terms of accuracy and performance So, author decided to them to experiment 88 Figure 4.17: Comparison of mAP and FPS between Yolov7 with Yolov5, Yolov6 on CPU [9] The above graph compares the mAP and FPS of several yolo models used on model CPU Model yolov7 and yolov5n are recalculated on 640 resolution images Look at the figures on image, we can evaluate that model yolov7t gains the highest Map, yolov5n gains the lowest Mean Average Precision For FPS, in constract to the above, yolov7t has the lowest Mean Average Precision, less than 20 frames Meanwhile, yolov5n has the highest FPS compared others model, greater than 30 frames Figure 4.18: Comparison of mAP and FPS between Yolov7 with Yolov5, Yolov6 on GPU [9] This graph shows comparison the mAP and FPS in a few models belong to families of models of yolo However, this experiment was conducted using a GPU (RTX 4090), and the aspect ratio is still 640 resolution image Looking at the graph, it is clear that the two models, yolov7 and yolov7x, which achieved the highest mAP, more than 50, are associated with Yolo version Yolov5s and Yolov5n, on the other hand, have the lowest mAP, less than 40 Yolov6l also is a model gains almost as high as mAP with yolov7x 89 However, about FPS, yolov7x and yolov7 have the lowest FPS, less than 50 frames, on the other hand, yolov5n and yolov5s gain the highest FPS, more than 225 Furthermore, there is only difference between models of belongs to yolov5 that yolov5x has the smallest FPS in families of versions of yolov5, less than 75 frames Meanwhile, yolov5s and yolov5n are two models gain the tallest FPS, greater than 225 frames Oppositely, yolov5x has a fairly high mAP ratio, close to yolov7 Yolov5s and yolov5n just only greater than 35 Figure 4.19: Comparison of mAP and FPS between Yolov7 with Yolov5, Yolov6 on TELSA P100 [9] Following that, this graph compares the mAP and FPS on the TESLA P100, with the image quality remaining at 640 resolution image The three models with the greatest mAP, greater than 50, are still yolov7 and yolov7x, yolov6l, yolov5x Yolov5s and Yolov5n are both a little over 35 Oppositely, yolov5s and yolov5n have the highest FPS compared others model, more than 140 frames Yolov7x, yolov7, yolov5x, yolov6l are these model have the smallest FPS, less than 60 frames 90 Table 4.1: Comparison of mAP and FPS between Yolov7 with others version of Yolo mAP is high FPS is high ( > 40) ( > 90 ) Yolov7x ✓  Yolov7 ✓  Yolov6l ✓  Yolov6m ✓  Yolov6s ✓  Yolov6t ✓  Yolov6n   Yolov5x ✓  Yolov5l ✓  Yolov5m ✓  Yolov5s  ✓ Yolov5n  ✓ We can infer from the evaluation above that yolov7 achieves high mAP but has a relatively low FPS Yolov5 features a low mAP yet a very high FPS, in contrast More specifically, yolov7 can return the high accuracy but FPS need to be improved Yolov5 gains the FPS very perfect, accuracy of yolov5 is also relatively acceptable Comparison of AP and inference time between Yolov7 with others version of Yolo 91 Figure 4.20: Comparison of AP and inference time between Yolov7 models with others Yolo’s different version models [4] From the above graph, an experiment is conducted on GPU V100, we can see that Yolov7 models gains the smallest inference time, about 5ms Meanwhile, others are higher than inference time, means that they run the result of object detection is late compared with Yolov7 models About AP, Yolov7 models also gain the highest AP Others just gain the AP is lower than Yolov7 models 4.5.2 Comparison between Yolo and others CNN After researching some papers or documents which relevant to Yolov5, Faster RCNN model illustrate pros and cons of both models In order to appropriately assess, the author has employed two models that are operating in real-time on the same environment Table 4.2: Comparison between Yolo and others CNN Inference Detection No Speed of small or overlapping Objects boxes 92 Missed Detection of crowded objects far away objects Yolov5 ✓ ✓ ✓ ✓ Faster  ✓  ✓ RCNN The final comparison of the two versions reveals that the running speed of the YOLO v5 is clearly superior The compact YOLO v5 model operates around 2.5 times more quickly and performs better when identifying smaller things Results that have few or no overlapping boxes are likewise easier to understand With their open source YOLO v5 model, which is simple to train and use for inferences Through few above evaluations, we can infer that Yolo is the best model for detecting object in real time Moreover, it also gains relatively fast inference time, and returns more precision than current object recognition models 93 CHAPTER 5: CONCLUSION AND FUTURE WORK 5.1 Conclusion We have presented a method for the categorization and conveyance of goods based on past research This system has accomplished the objectives set by our team The original objectives are as follows: • Develop an AI system capable of identifying 128 barcodes on parcel packages in order to categorize shipments in accordance with their respective warehouse • Construct an AGV system managed by the AI system through a Wi-Fi link for package delivery to each fixed compartment • Additionally, the system is equipped with tracking software that indicates the position of autonomous cars, the location of warehouses, and the amount of packages on each warehouse Following this subject, team members will learn more about neural models that enhance this model's recognition capabilities In addition, we will learn more about hardware and mechanical design in order to construct a system that can be applied in the real world 5.2 Future Work Based on the results obtained while implementing this project, we hope to develop and upgrade the project in the future to be able to apply delivery in the warehouses of manufacturing plants that use automation technology We will expand the model by making the vehicle larger in order to transport large, heavy packages In addition, we add self-sorting functionality to the warehouse floors to reduce labor costs and production time 94 APPENDIX 95 REFERENCE [1] R N Keiron O’Shea, "An Introduction to Convolutional Neural Networks," CoRR, p 11, 2015 [2] IndoML, "Convolutional Neural Networks (CNN) Introduction," March 2018 [Online] Available: https://indoml.com/2018/03/07/student-notes-convolutional-neuralnetworks-cnn-introduction/?fbclid=IwAR2Clu7-ZWgDygWwpS16XheKZ1kLhiT8eLglBQ-85QKjaEtmH9OOWysb6A [3] J Redmon, "You Only Look Once: Unified, Real-Time Object Detection," CVPR, p 10, 2016 [4] C.-Y Wang, "YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors," CVPR, p 15, 2022 [5] V T Hang, "AUTOMATIC PARCEL SORTING SYSTEM WITH AI TECHNOLOGY," TNU Journal of Science and Technology, pp 127-134, 2022 [6] M pakdaman, "A Line Follower Robot from design to Implementation: Technical issues and problems," ICCAE, pp 5-9, 2010 [7] A Pathak, "Line Follower Robot for Industrial Manufacturing Process," International Journal of Engineering Inventions, pp 10-17, 2017 [8] R Smith, "An Overview of the Tesseract OCR Engine," Ninth International Conference on Document Analysis and Recognition (ICDAR 2007), pp 1-5, 2007 [9] S Rath, "Performance Comparison of YOLO Object Detection Models – An Intensive Study," [Online] Available: https://learnopencv.com/performance-comparison-of-yolomodels/?fbclid=IwAR3YUKptiMxwZocJmtI70m-qVEgPf3UvpMLF2KGPV4kEp6kECuzne9mUDE [10] "SPI communication protocol," [Online] Available: https://aticleworld.com/spicommunication-protocol/ 96 97 S K L 0