(Đồ án tốt nghiệp) research, design and construct an autonomous golf cart using multisensor fusion

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT COMPUTER ENGINEERING TECHNOLOGY RESEARCH, DESIGN AND CONSTRUCT AN AUTONOMOUS GOLF CART USING MULTISENSOR FUSION H ADVISOR: DR LE MI HA STUDENT: PHAN THANH DANH NGUYEN TAN THIEN NIEN SKL009830 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT RESEARCH, DESIGN AND CONSTRUCT AN AUTONOMOUS GOLF CART USING MULTISENSOR FUSION PHAN THANH DANH - 18119214 NGUYỄN TẤN THIÊN NIÊN - 18119033 MAJOR: COMPUTER ENGINEERING TECHNOLOGY ADVISOR: LÊ MỸ HÀ, Assoc.Prof Ho Chi Minh City, January 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT RESEARCH, DESIGN AND CONSTRUCT AN AUTONOMOUS GOLF CART USING MULTISENSOR FUSION PHAN THANH DANH - 18119214 NGUYỄN TẤN THIÊN NIÊN - 18119033 MAJOR: COMPUTER ENGINEERING TECHNOLOGY ADVISOR: LÊ MỸ HÀ, Assoc.Prof Ho Chi Minh City, January 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, July 27, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Phan Thanh Danh Student ID: 18119214 Student name: Nguyễn Tấn Thiên Niên Student ID: 18119033 Major: Computer Engineering Technology Class: 18119CLA1 Advisor: Assoc Prof Lê Mỹ Hà Phone number: 0938811201 Date of assignment: Date of submission: Project title: Research, design, and construct an autonomous golf cart using multisensor fusion Initial materials provided by the advisor: Image processing and machine learning documents such as papers and books: The related thesis of previous students The hardware specifications and its review Content of the project: Refer to documents, survey, read and summarize to determine the project directions Calculate parameters and design block diagram for steering system using DC servo Try and handle errors in the wheeling system (mechanical and electrical) Collect and visualize data of sensors Choose models and algorithms for the car’s perception Write programs for microcontrollers and processors Test and evaluate the completing system Write a report Prepare slides for presenting Final product: The golf cart model uses a multisensor combination that has two modes: Automatic and Manual The golf cart can operate well on HCMUTE campus with not many complex scenarios CHAIR OF THE PROGRAM (Sign with full name) ADVISOR (Sign with full name) i THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, July 27, 2022 ADVISOR’S EVALUATION SHEET Student name: Phan Thanh Danh Student ID: 18119214 Student name: Nguyễn Tấn Thiên Niên Student ID: 18119033 Major: Computer Engineering Technology Project title: Research, design, and construct an autonomous golf cart using multisensor fusion Advisor: Assoc Prof Lê Mỹ Hà EVALUATION Content of the project: The thesis has a total of six chapters with 73 pages The construction and design of an autonomous golf cart that can run on the HCMUTE campus with two modes: Manual and Automatic The system works with a combination of different sensors and their according algorithms The real system is successfully completed following the objectives in the proposal The thesis forms a basic foundation for the next generation of HCMUTE students in the field of the practical autonomous car Strengths: The system can support the UTE students and lectures in moving around campus The golf car is designed with image processing and machine learning algorithms combined with the control technique and mechanism design The whole sensors of this project are low-cost The execution time for the automatic mode is suitable for practical application The accuracy and the safety of the system are guaranteed Weaknesses: The system cannot operate fully autonomous without an operator, because several things need to be upgraded and improved All the sensors and hardware are low-cost Therefore, the accuracy and the effectiveness of the system are just acceptable in real environments Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Excellent Mark: 10 (In words: Ten) Ho Chi Minh City, July 27, 2022 ADVISOR (Sign with full name) ii APPENDIX 5: (Pre-Defense Evaluation sheet) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, January 20, 2020 PRE-DEFENSE EVALUATION SHEET Student name: Phan Thanh Danh Student ID: 18119214 Student name: Nguyễn Tấn Thiên Niên Student ID: 18119033 Major: Computer Engineering Technology Project title: Research, design, and construct an autonomous golf cart using multisensor fusion Name of Reviewer: EVALUATION Content and workload of the project Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark: ……………… (In words: ) Ho Chi Minh City, month day, year REVIEWER (Sign with full name) iii APPENDIX 6: (Evaluation sheet of Defense Committee Member) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Phan Thanh Danh Student ID: 18119214 Student name: Nguyễn Tấn Thiên Niên Student ID: 18119033 Major: Computer Engineering Technology Project title: Research, design, and construct an autonomous golf cart using multisensor fusion Name of Defense Committee Member: EVALUATION Content and workload of the project Strengths: Weaknesses: Overall evaluation: (Excellent, Good, Fair, Poor) Mark: ……………… (In words: ) Ho Chi Minh City, month day, year COMMITTEE MEMBER (Sign with full name) iv ACKNOWLEDGEMENT We would like to sincerely thank Professor Le My Ha for his thorough instruction, which helped us to have the necessary information to use for completing the thesis During the whole progress, even if we did our best to complete everything completely, mistakes are still inevitable We anticipate having my advisor's focused assistance and direction to help us gain more experience and successfully complete the topic project On the other hand, we would like to express our sincere thanks to the Faculty of Hight Quality Training and Faculty of Electrical and Electronics Engineering where we obtained basic knowledge and experience Especially, we received the golf cart from FEEE honorably Moreover, we also would like to thank ISLab members who helps us in detailing the works of this project They shared valuable experience and knowledge with us Ultimately, we would like to express our gratitude to our families for their support of our team throughout the implementation of this thesis Sincere thanks for everything! v A GUARANTEE This thesis is the result of our study and implementation, which we hereby formally proclaim We did not plagiarize from a published article without author acceptance We will take full responsibility for any violations that may have occurred Authors Phan Thanh Danh Nguyễn Tấn Thiên Niên vi ABSTRACT Autonomous cars will be able to make an analysis and manage themselves on the ongoing path, depending on scene understanding and surrounding observation Particularly, the automobile does need to clarify the whole information surrounding it Inspired by that ideas, in this paper, we proposed a multi-sensor fusion method for autonomous cars operating on the HCMUTE campus The fusion method comprises three types of sensors, which are a Camera, GPS, and 2D LiDAR To begin with, we utilized and enhanced two deep learning models, which are lane-line detection and semantic segmentation Two of these models are pre-trained and fine-tuned on our self-labeled dataset As for the GP signal, we used Kalman Filter to reduce the noises from the environment and then check the continuous destination by a circular equation Additionally, we took advantage of 2D LiDAR as the safety term during the car avoidance process Last but not least, we combined all these programs by using threading and distributed system, which communicates with each other by User Datagram Protocol The system uses our laptop with NVIDIA GTX 1650 graphic card as the primary processing unit To reduce the execution burden of our laptop, we subjoined a Jetson TX2 for processing GPS and LiDAR sensors In terms of the controlling system, the main microcontroller is two Arduino boards (Mega and Nano); one is for steering control, and another is for speed control Furthermore, we designed a simple interface using PyQT to display the on-road information Experimental results reveal that the whole system works well on several campus roads Furthermore, the lowest frame per second of the system is 20, which satisfies realtime practical applications vii