(Đồ án hcmute) research, design and construct an autonomous golf cart using multisensor fusion

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT COMPUTER ENGINEERING TECHNOLOGY RESEARCH, DESIGN AND CONSTRUCT AN AUTONOMOUS GOLF CART USING MULTISENSOR FUSION ADVISOR: DR LE MI HA STUDENT: PHAN THANH DANH NGUYEN TAN THIEN NIEN SKL009830 Ho Chi Minh city, January 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT RESEARCH, DESIGN AND CONSTRUCT AN AUTONOMOUS GOLF CART USING MULTISENSOR FUSION PHAN THANH DANH - 18119214 NGUYỄN TẤN THIÊN NIÊN - 18119033 MAJOR: COMPUTER ENGINEERING TECHNOLOGY ADVISOR: LÊ MỸ HÀ, Assoc.Prof Ho Chi Minh City, January 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT RESEARCH, DESIGN AND CONSTRUCT AN AUTONOMOUS GOLF CART USING MULTISENSOR FUSION PHAN THANH DANH - 18119214 NGUYỄN TẤN THIÊN NIÊN - 18119033 MAJOR: COMPUTER ENGINEERING TECHNOLOGY ADVISOR: LÊ MỸ HÀ, Assoc.Prof Ho Chi Minh City, January 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, July 27, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Phan Thanh Danh Student ID: 18119214 Student name: Nguyễn Tấn Thiên Niên Student ID: 18119033 Major: Computer Engineering Technology Class: 18119CLA1 Advisor: Assoc Prof Lê Mỹ Hà Phone number: 0938811201 Date of assignment: Date of submission: Project title: Research, design, and construct an autonomous golf cart using multisensor fusion Initial materials provided by the advisor: - Image processing and machine learning documents such as papers and books: - The related thesis of previous students - The hardware specifications and its review Content of the project: - Refer to documents, survey, read and summarize to determine the project directions - Calculate parameters and design block diagram for steering system using DC servo - Try and handle errors in the wheeling system (mechanical and electrical) - Collect and visualize data of sensors - Choose models and algorithms for the car’s perception - Write programs for microcontrollers and processors - Test and evaluate the completing system - Write a report - Prepare slides for presenting Final product: The golf cart model uses a multisensor combination that has two modes: Automatic and Manual The golf cart can operate well on HCMUTE campus with not many complex scenarios CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) i THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, July 27, 2022 ADVISOR’S EVALUATION SHEET Student name: Phan Thanh Danh Student ID: 18119214 Student name: Nguyễn Tấn Thiên Niên Student ID: 18119033 Major: Computer Engineering Technology Project title: Research, design, and construct an autonomous golf cart using multisensor fusion Advisor: Assoc Prof Lê Mỹ Hà EVALUATION Content of the project: The thesis has a total of six chapters with 73 pages The construction and design of an autonomous golf cart that can run on the HCMUTE campus with two modes: Manual and Automatic The system works with a combination of different sensors and their according algorithms The real system is successfully completed following the objectives in the proposal The thesis forms a basic foundation for the next generation of HCMUTE students in the field of the practical autonomous car Strengths: The system can support the UTE students and lectures in moving around campus The golf car is designed with image processing and machine learning algorithms combined with the control technique and mechanism design The whole sensors of this project are low-cost The execution time for the automatic mode is suitable for practical application The accuracy and the safety of the system are guaranteed Weaknesses: The system cannot operate fully autonomous without an operator, because several things need to be upgraded and improved All the sensors and hardware are low-cost Therefore, the accuracy and the effectiveness of the system are just acceptable in real environments Approval for oral defense? (Approved or denied) Approved Overall evaluation: (Excellent, Good, Fair, Poor) Excellent Mark: 10 (In words: Ten) Ho Chi Minh City, July 27, 2022 ADVISOR (Sign with full name) ii APPENDIX 5: (Pre-Defense Evaluation sheet) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, January 20, 2020 PRE-DEFENSE EVALUATION SHEET Student name: Phan Thanh Danh Student ID: 18119214 Student name: Nguyễn Tấn Thiên Niên Student ID: 18119033 Major: Computer Engineering Technology Project title: Research, design, and construct an autonomous golf cart using multisensor fusion Name of Reviewer: EVALUATION Content and workload of the project Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark: ……………… (In words: ) Ho Chi Minh City, month day, year REVIEWER (Sign with full name) iii APPENDIX 6: (Evaluation sheet of Defense Committee Member) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Phan Thanh Danh Student ID: 18119214 Student name: Nguyễn Tấn Thiên Niên Student ID: 18119033 Major: Computer Engineering Technology Project title: Research, design, and construct an autonomous golf cart using multisensor fusion Name of Defense Committee Member: EVALUATION Content and workload of the project Strengths: Weaknesses: Overall evaluation: (Excellent, Good, Fair, Poor) Mark: ……………… (In words: ) Ho Chi Minh City, month day, year COMMITTEE MEMBER (Sign with full name) iv ACKNOWLEDGEMENT We would like to sincerely thank Professor Le My Ha for his thorough instruction, which helped us to have the necessary information to use for completing the thesis During the whole progress, even if we did our best to complete everything completely, mistakes are still inevitable We anticipate having my advisor's focused assistance and direction to help us gain more experience and successfully complete the topic project On the other hand, we would like to express our sincere thanks to the Faculty of Hight Quality Training and Faculty of Electrical and Electronics Engineering where we obtained basic knowledge and experience Especially, we received the golf cart from FEEE honorably Moreover, we also would like to thank ISLab members who helps us in detailing the works of this project They shared valuable experience and knowledge with us Ultimately, we would like to express our gratitude to our families for their support of our team throughout the implementation of this thesis Sincere thanks for everything! v A GUARANTEE This thesis is the result of our study and implementation, which we hereby formally proclaim We did not plagiarize from a published article without author acceptance We will take full responsibility for any violations that may have occurred Authors Phan Thanh Danh Nguyễn Tấn Thiên Niên vi ABSTRACT Autonomous cars will be able to make an analysis and manage themselves on the ongoing path, depending on scene understanding and surrounding observation Particularly, the automobile does need to clarify the whole information surrounding it Inspired by that ideas, in this paper, we proposed a multi-sensor fusion method for autonomous cars operating on the HCMUTE campus The fusion method comprises three types of sensors, which are a Camera, GPS, and 2D LiDAR To begin with, we utilized and enhanced two deep learning models, which are lane-line detection and semantic segmentation Two of these models are pre-trained and fine-tuned on our self-labeled dataset As for the GP signal, we used Kalman Filter to reduce the noises from the environment and then check the continuous destination by a circular equation Additionally, we took advantage of 2D LiDAR as the safety term during the car avoidance process Last but not least, we combined all these programs by using threading and distributed system, which communicates with each other by User Datagram Protocol The system uses our laptop with NVIDIA GTX 1650 graphic card as the primary processing unit To reduce the execution burden of our laptop, we subjoined a Jetson TX2 for processing GPS and LiDAR sensors In terms of the controlling system, the main microcontroller is two Arduino boards (Mega and Nano); one is for steering control, and another is for speed control Furthermore, we designed a simple interface using PyQT to display the on-road information Experimental results reveal that the whole system works well on several campus roads Furthermore, the lowest frame per second of the system is 20, which satisfies realtime practical applications vii APPENDIX In this part, the authors present several extra works which are being developed to push the project further Besides, there is a limitation of mechanism capability, so the wheeling structure has been tried and errored to get better versions Through this section, the contribution as well as a load of works, is proved The following extra works are listed below:  Low-light image at night Lacking visual information at night is one of the problems for the self-driving car, whilst the extra light does not supply enough images to get the feature of the whole image The algorithm utilized the inversion of an image to enhance the quality of visual information Furthermore, the mathematical model is designed to de-foggy as well as solve low-light problems Figure shows the testing result with the random data collected on the Internet Figure The testing results with images collected on the internet The algorithm is tested with HCMUTE campus data that gives good results in different cases shown in Figure The algorithms will be applied in the soon future to save the car in dark scenarios 66 Figure The testing results with images collected on campus  Extend Kalman Filter with multiple sensor fusion The other better version of the Kalman filter is the Extended Kalman filter with the advantage of using the differential function for the transition model and observation model This signature can be suitable for nonlinear data like the Lidar, Radar, and some other data types 𝑥𝑡 = 𝑓 (𝑥𝑡−1 , 𝑢𝑡 ) + 𝑤𝑡 𝑧𝑡 = ℎ(𝑥𝑡 ) + 𝑣𝑡 (1) (2) The main difference between the Kalman filter and the Extended Kalman filter (EKF) is the transition model (𝑓) and observation model (ℎ) of EKF does not need to be linear functions but may instead be differentiable functions and both are performed by the Jacobian matrix With that difference, the EKF is suitable for nonlinear data like Lidar, Radar, or some other data types 67 Table The Difference between the Kalman filter and Extended Kalman filter mathematically Kalman filter Extended Kalman filter 𝐹= 𝑥̅ = 𝐹𝑥 𝜕𝑓 (𝑥𝑡 , 𝑢𝑡 ) | 𝜕𝑥 𝑥 𝑡 ,𝑢𝑡 𝑃̅ = 𝐹𝑃𝐹 𝑇 𝑥̅ = 𝑓(𝑥, 𝑢) 𝑃̅ = 𝐹𝑃𝐹 𝑇 𝐾 = 𝑃̅𝐻 𝐻𝑃̅ 𝐻 + 𝑅 𝑇( 𝑇 𝐻= )−1 𝜕ℎ(𝑥̅𝑡 ) | 𝜕𝑥̅ 𝑥𝑡 𝑥 = 𝑥̅ + 𝐾(𝑧 − 𝐻𝑥̅ ) 𝑃 = (𝐼 − 𝐾𝐻 )𝑃̅ 𝐾 = 𝑃̅𝐻 𝑇 (𝐻𝑃̅𝐻 𝑇 + 𝑅)−1 𝑥 = 𝑥̅ + 𝐾(𝑧 − ℎ(𝑥̅ ) 𝑃 = (𝐼 − 𝐾𝐻 )𝑃̅ The extended Kalman filter for the position data First, we need to build a state matrix for the filter, this state matrix consists of components like x, y represents Latitude, Longitude 𝜓 represents heading and finally, 𝑣 is the symbol of velocity 𝑃𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝑋 𝑥∗ 𝑦∗ 𝑃𝑜𝑠𝑖𝑡𝑖𝑜𝑛 𝑌 𝑥𝑘 = [ 𝜓 ] = [ 𝐻𝑒𝑎𝑑𝑖𝑛𝑔 ] 𝑣 𝑉𝑒𝑙𝑜𝑐𝑖𝑡𝑦 (3) The practice has proven that all sensors if operating in a natural environment, will have a certain amount of noise, and so are the sensors we use, when visualizing data, it can be seen (figure 3) that the data of the variables has a very high noise amplitude 68 Figure The Noise Of Data After applying the EKF to the autonomous GPS navigation, the signal has been significantly filtered (figure 4), which directly affects the performance of an autonomous vehicle in general and driverless Robot systems by GPS in particular 69 Velocity Figure Four Signal of the state after filtering The application of the GPS that has been processed through the EKF filter to the University of Technology and Education campus (Figure 5) and the result is a low-noise signal, which contributes significantly to building the accuracy algorithms for our autonomous vehicle system 70 Figure EKF and UTE map  Traffic sign detection As for the traffic detection, we used a model called YOLO V5 which is trained with TT100K dataset and our self-label dataset The results of the object detection model are shown in Figure Figure Traffic detection results 71  PyQT5 app This is an interface for people who can interact with the car without terminal command In the app, we have pages: login page, main page, introduction, and car manual Now we have just developed some features of them Primarily, the main page is used for the user to select the expected destination Figure Login interface Figure Main page 72 Figure Car monitoring page  Multi-camera In this part, we research the method to expand the camera angle This investigation gives an autonomous car more visual information If using a mono frontal camera leads to a lack of right and left information One possible approach is the panorama camera, but it will be distorted heavily and the cost is really expensive Therefore, we managed to use the stitching image technique with mono cameras to create the larger view shown in Figure 10 73 Figure 10: The stiching image  Publication In this thesis, we have related publications on the different International Conferences: ICSSE2021, GTSD 2022, IWIS 2022 Figure 11: ICSSE 2021 Acceptance Letter (Paper 10) 74 Figure 12: GTSD 2022 Acceptance Letter (Paper 308) Figure 13: Best paper award of paper 308 75 Figure 14: GTSD 2022 Acceptance Letter (Paper 23) Figure 15: IWIS 2022 Acceptance Letter (Paper 8721) 76  Plagiarism check Figure 11: Plagiarism check using turnitin 77 78 79 S K L 0