(Đồ án tốt nghiệp) design and implementation of a lane keeping and navigation system for self driving vehicle based on the fusion of camera, gps, and imu

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT AUTOMATION AND CONTROL ENGINEERING DESIGN AND IMPLEMENTATION OF A LANEKEEPING AN NAVIGATION SYSTEM FOR SELFDRIVING VEHICLE BASED ON THE FUSION OF CAMERA, GPS AND IMU LECTURER: LÊ MỸ HÀ, Assoc Prof STUDENT: NGUYỄN HOÀNG HẢI NAM PHẠM DUY HƯNG SKL009344 Ho Chi Minh City, August, 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF A LANE KEEPING AND NAVIGATION SYSTEM FOR SELFDRIVING VEHICLE BASED ON THE FUSION OF CAMERA, GPS AND IMU NGUYỄN HOÀNG HẢI NAM - 18151021 PHẠM DUY HƯNG - 18151188 Major: AUTOMATION AND CONTROL ENGINEERING Advisor: LÊ MỸ HÀ, Assoc Prof Ho Chi Minh City, August 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF A LANE KEEPING AND NAVIGATION SYSTEM FOR SELFDRIVING VEHICLE BASED ON THE FUSION OF CAMERA, GPS AND IMU NGUYỄN HOÀNG HẢI NAM - 18151021 PHẠM DUY HƯNG - 18151188 Major: AUTOMATION AND CONTROL ENGINEERING Advisor: Assoc Prof LÊ MỸ HÀ Ho Chi Minh City, August 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 5, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Nguyễn Hoàng Hải Nam Student ID: 18151021 Student name: Phạm Duy Hưng Student ID: 18151188 Major: Automation and Control Engineering Class: 18151CLA2 Advisor: Assoc Prof Lê Mỹ Hà Phone number: 0938811201 Date of assignment: Date of submission: Project title: Design and implementation of a lane keeping and navigation system for selfdriving vehicle based on the fusion of camera, GPS and IMU - Initial materials provided by the advisor: Documents and articles related to machine learning and image processing The related thesis of previous students The hardware specifications and its review Content of the project: Read, perform surveys, summarize to determine the scope of the project Research the theoretical array math related to self-driving cars to give directions and handle problems during model building Read and process sensors signal Choose model and algorithm for lane keeping of vehicle Write program to control microcontroller Research fusion algorithm for sensors Write project report Prepare slide for presenting Final product: The car model can operate inside HCMUTE school campus based on a combination of camera, GPS and IMU in Auto and Manual mode under not too complex conditions CHAIR OF THE PROGRAM (Sign with full name) ADVISOR (Sign with full name) i THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 5, 2022 ADVISOR’S EVALUATION SHEET Student name: Nguyễn Hoàng Hải Nam Student ID: 18151021 Student name: Phạm Duy Hưng Student ID: 18151188 Major: Automation and Control Engineering Project title: Design And Implementation Of A Lane Keeping And Navigation System For SelfDriving Vehicle Based On The Fusion Of Camera, GPS And IMU Advisor: Assoc Prof Lê Mỹ Hà EVALUATION Content of the project: The content of this report is 64 pages The design and construction of car model that can operate HCMUTE school campus in Auto and Manual mode The system runs based on a series of different sensors and algorithms The final product meets the requirements in the proposal Strengths: The car model creates a pipeline for future implementation on life size car Project was designed with machine learning-based image processing imbued with controller technique All the sensors used for this project are low-cost Process time of program is suitable for real-time applications Weaknesses: The system cannot reach full automation level instead requires operator presence due to there are some improvements to be made The accuracy and stability of system stops at acceptable level in outdoor environment due to all sensors are low-cost Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:…………… (in words: ) Ho Chi Minh City, August 5, 2022 ADVISOR (Sign with full name) ii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 5, 2022 PRE-DEFENSE EVALUATION SHEET Student name: Student ID: Student name: Student ID: Major: Project title: Name of Reviewer: EVALUATION Content and workload of the project Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, month day, year REVIEWER (Sign with full name) iii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Student ID: Student name: Student ID: Major: Project title: Name of Defense Committee Member: EVALUATION Content and workload of the project Strengths: Weaknesses: Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, month day, year COMMITTEE MEMBER (Sign with full name) iv ACKNOWLEDGEMENTS We would like to express our utmost thanks to Professor Le My Ha for his thorough instructions which provided us with necessary information to complete this thesis It is expected despite the period this project requires and our full effort mistakes will still exist With help from our advisor’s especially his inputs and advice we hope to gain more experience and achieve success in this project topic We would also like to extend to thank Faculty of Hight Quality Training and Faculty of Electrical and Electronics Engineering where we obtained basic knowledge and experience Moreover, we would like to thank the members of ISLab member for helping us gaining the full perspective of this project They shared valuable knowledge and experience with us We would like to express our gratitude to our families for their support of our team throughout the working of this thesis Sincere thanks for everything! v A GUARANTEE This thesis is the result of our study and implementation, which we hereby formally proclaim We did not plagiarize from a published article without author acceptance We will take full responsibility for any violations that may have occurred Authors Nguyễn Hoàng Hải Nam Phạm Duy Hưng vi ABSTRACT This thesis proposes a self-driving system for Autonomous Ground Vehicle in urban area based on the fusion between Camera-based Image Segmentation method and a GPS-and-IMU-based lateral controller For the task of Image Segmentation, we use a Segmentation Model called BiseNet to segment images coming from the camera and derive a heading position for the vehicle using the Edge-detected segmented images and geometric operations To navigate the vehicle using GPS signals, we design a waypointbased lateral controller to steer the vehicle with respect to a designated GPS track Our GPS measurements are collected, corrected, and updated with the help of the Extended Kalman Filter The fusion between the two controllers is flag-based and used to derive the final steering control of the vehicle in different road scenarios The system includes a NVIDIA Jetson TX2 development board stacked on the rear top of a 1/10 scale RC car along with a 130 POV camera, a smart phone to stream sensor signals through Wi-Fi All experiments were carried out on a pre-defined road track along HCMCUTE campus where the system was able to run autonomously in real-time at 11-13 fps vii