Thông tin tài liệu
MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT AUTOMATION AND CONTROL ENGINEERING h DESIGN AND IMPLEMENTATION OF A LANE KEEPING AN NAVIGATION SYSTEM FOR SELFDRIVING VEHICLE BASED ON THE FUSION OF CAMERA, GPS AND IMU LECTURER: LÊ MỸ HÀ, Assoc Prof STUDENT: NGUYỄN HOÀNG HẢI NAM PHẠM DUY HƯNG SKL009344 Ho Chi Minh City, August, 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT h DESIGN AND IMPLEMENTATION OF A LANE KEEPING AND NAVIGATION SYSTEM FOR SELFDRIVING VEHICLE BASED ON THE FUSION OF CAMERA, GPS AND IMU NGUYỄN HOÀNG HẢI NAM - 18151021 PHẠM DUY HƯNG - 18151188 Major: AUTOMATION AND CONTROL ENGINEERING Advisor: LÊ MỸ HÀ, Assoc Prof Ho Chi Minh City, August 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF A LANE h KEEPING AND NAVIGATION SYSTEM FOR SELFDRIVING VEHICLE BASED ON THE FUSION OF CAMERA, GPS AND IMU NGUYỄN HOÀNG HẢI NAM - 18151021 PHẠM DUY HƯNG - 18151188 Major: AUTOMATION AND CONTROL ENGINEERING Advisor: Assoc Prof LÊ MỸ HÀ Ho Chi Minh City, August 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 5, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Nguyễn Hoàng Hải Nam Student ID: 18151021 Student name: Phạm Duy Hưng Student ID: 18151188 Major: Automation and Control Engineering Class: 18151CLA2 Advisor: Assoc Prof Lê Mỹ Hà Phone number: 0938811201 Date of assignment: Date of submission: Project title: Design and implementation of a lane keeping and navigation system for selfdriving vehicle based on the fusion of camera, GPS and IMU h Initial materials provided by the advisor: - Documents and articles related to machine learning and image processing - The related thesis of previous students - The hardware specifications and its review Content of the project: - Read, perform surveys, summarize to determine the scope of the project - Research the theoretical array math related to self-driving cars to give directions and handle problems during model building - Read and process sensors signal - Choose model and algorithm for lane keeping of vehicle - Write program to control microcontroller - Research fusion algorithm for sensors - Write project report - Prepare slide for presenting Final product: The car model can operate inside HCMUTE school campus based on a combination of camera, GPS and IMU in Auto and Manual mode under not too complex conditions CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) i THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 5, 2022 ADVISOR’S EVALUATION SHEET Student name: Nguyễn Hoàng Hải Nam Student ID: 18151021 Student name: Phạm Duy Hưng Student ID: 18151188 Major: Automation and Control Engineering Project title: Design And Implementation Of A Lane Keeping And Navigation System For SelfDriving Vehicle Based On The Fusion Of Camera, GPS And IMU Advisor: Assoc Prof Lê Mỹ Hà EVALUATION h Content of the project: The content of this report is 64 pages The design and construction of car model that can operate HCMUTE school campus in Auto and Manual mode The system runs based on a series of different sensors and algorithms The final product meets the requirements in the proposal Strengths: The car model creates a pipeline for future implementation on life size car Project was designed with machine learning-based image processing imbued with controller technique All the sensors used for this project are low-cost Process time of program is suitable for real-time applications Weaknesses: The system cannot reach full automation level instead requires operator presence due to there are some improvements to be made The accuracy and stability of system stops at acceptable level in outdoor environment due to all sensors are low-cost Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:…………… (in words: ) Ho Chi Minh City, August 5, 2022 ADVISOR (Sign with full name) ii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, August 5, 2022 PRE-DEFENSE EVALUATION SHEET Student name: Student ID: Student name: Student ID: Major: Project title: Name of Reviewer: EVALUATION Content and workload of the project h Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, month day, year REVIEWER (Sign with full name) iii THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Student ID: Student name: Student ID: Major: Project title: Name of Defense Committee Member: EVALUATION Content and workload of the project h Strengths: Weaknesses: Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, month day, year COMMITTEE MEMBER (Sign with full name) iv ACKNOWLEDGEMENTS We would like to express our utmost thanks to Professor Le My Ha for his thorough instructions which provided us with necessary information to complete this thesis It is expected despite the period this project requires and our full effort mistakes will still exist With help from our advisor’s especially his inputs and advice we hope to gain more experience and achieve success in this project topic We would also like to extend to thank Faculty of Hight Quality Training and Faculty of Electrical and Electronics Engineering where we obtained basic knowledge and experience Moreover, we would like to thank the members of ISLab member for helping us gaining the full perspective of this project They shared valuable knowledge and experience with us We would like to express our gratitude to our families for their support of our team throughout the working of this thesis Sincere thanks for everything! h v A GUARANTEE This thesis is the result of our study and implementation, which we hereby formally proclaim We did not plagiarize from a published article without author acceptance We will take full responsibility for any violations that may have occurred Authors Nguyễn Hoàng Hải Nam Phạm Duy Hưng h vi ABSTRACT This thesis proposes a self-driving system for Autonomous Ground Vehicle in urban area based on the fusion between Camera-based Image Segmentation method and a GPSand-IMU-based lateral controller For the task of Image Segmentation, we use a Segmentation Model called BiseNet to segment images coming from the camera and derive a heading position for the vehicle using the Edge-detected segmented images and geometric operations To navigate the vehicle using GPS signals, we design a waypointbased lateral controller to steer the vehicle with respect to a designated GPS track Our GPS measurements are collected, corrected, and updated with the help of the Extended Kalman Filter The fusion between the two controllers is flag-based and used to derive the final steering control of the vehicle in different road scenarios The system includes a NVIDIA Jetson TX2 development board stacked on the rear top of a 1/10 scale RC car along with a 130 POV camera, a smart phone to stream sensor signals through Wi-Fi All experiments were carried out on a pre-defined road track along HCMCUTE campus where the system was able to run autonomously in real-time at 11-13 fps h vii Figure 4.6 UHD Webcam 4.2.6 PCA9685 Servo Driver User can only control one singular servo with normal means when steering car angel But PCA 9685 Servo driver can control up to servos much better and more accurate The servo is controlled by Jetson NVIDIA TX2 through I2C protocol Below is the pinout diagram for this driver h Figure 4.7 PCA 9685 Servo Driver 4.2.5 Jetson NVIDIA TX2 AI is now becoming a technology development trend, so there are many products made to serve this huge chain of problems NVIDIA TX2 is a typical example, although with its simple size, it can handle large problems such as running many artificial networks in parallel to speed up algorithm processing, object recognition, math problems, image segmentation or speech recognition 55 GPU 256-core Pascal™ GPU CPU Dual-core Denver 64-bit CPU + Quad-core ARM® Cortex®-A57 MPCore RAM 8GB LPDDR4 Memory 32GB eMMC 5.1 - Encode : 4K x 2K 60 Hz (HEVC) Video - Decode: 4K x 2K 60 Hz (12-bit support) - Ethernet: negotiation 10/100/1000BASE-T auto- - Wireless: 802.11ac WIFI + Bluetooth - Camera: 12 lanes MIPI CSI-2, D-PHY 1.2 (30 Gbps) Interface - Display: HDMI 2.0 / eDP 1.4 / 2x DSI / 2x DP 1.2 h - USB: USB 3.0 + USB 2.0 (Micro USB) - Others: GPIO, I2C, I2S, SPI, UART Power DC Jack (19V) - Core module: 87mm × 50mm Dimensions - Mother board: 170.2mm x170.2mm x15.6mm 7.5W/15W Power Networking 10/100/1000 BASE-T Ethernet Table 4.1 Specifications of Jetson TX2 Below is the NVIDIA TX2 board image 56 Figure 4.8 NVIDIA JETSON TX2 board 4.2.6 4G Dongle h Figure 4.9 4G dongle OLAX U80 OBC OLAX U80 4g USB Dongle device with Wi-Fi transmitter operating in high speed 4G network up to 150Mbps, allowing Wi-Fi broadcasting for 10 devices at the same time is ideal for personal usage, working groups, problem solving, decorating, or use in small automobiles 4.2.7 RX601 RX601 is used to receive signal from DEVO7 to control steering and throttle of RC car 57 Specifications: - Frequency: 2.4GHz - Number of channels: - Sensitivity: -105dbm - Power source: 4.8-6V - Weight: 5g - Dimensions: 33*20*13mm - Compatible with controller (Tx): DEVO 6S, DEVO 7, DEVO 8S, DEVO 10, DEVO 12S Figure 4.10 RX601 module 4.3 Self-driving Car Model The self-driving RC car model was built by student in figure 4.11 The self-driving car model in the project includes a servo attached to the steering wheels, the servo turns in a limited angle range to keep the vehicle running towards the center of the road A stationary camera attached to the rod affixed at the roof of the car collects image from the road for processing A phone holder is also equipped for mobile phone to stream GPS and IMU signal h Figure 4.11 The Autonomous self-driving RC Car model 58 h Figure 4.12 Hardware connection of the project 59 CHAPTER 5: EXPERIMENTAL RESULTS 5.1 Image Segmentation Network result The author team conducts BiseNet training on custom dataset (1300 images) on the backbone ResNet18 The selected hyper parameters are epochs = 250, learning rate = 0.0001, batch size = 8, decay rate = 0.8 The coefficients are selected according to the recommendation of author Yu et al [26] and some other authors At the end of training, the loss parameter for the training set is 1.67, the mIOU score on the Validation set is 0.78 mIoU The parameters are shown through Tensorboard as follows: h Figure 5.1 mIOU score record tracked over a 200-epoch training session a) b) Figure 5.2 Train loss record tracked over 200 epochs; a) Training loss; b) Validation loss The results of the assessment segmentation via the BiseNet network on the selflabeled data set on the actual data Inference results were assessed on i7 10750 and GTX1660ti systems through factors: FPS (frames per second), mIOU (mean Intersection Over Union), and Per-class IOU (road, pedestrian, and car): 60 The FPS of the model is calculated by taking inverse of the total processing time shown in the following formula: 𝑓𝑝𝑠 = ~Per-class IOU Model BiseNet (Resnet-18) BiseNet (Xception-39) LiteSeg (Mobilenet) 𝑇2 − 𝑇1 (5.1) mIOU* Road Human Car 0.97 0.52 0.95 0.89 0.43 0.92 0.39 FPS fp32** fp16** 0.78 10 21 0.83 0.71 14 24 0.88 0.73 17 28 Table 5.1 Base model with backbone CNN performance comparison on (360×640) image h Note that the above mIOU score may vary among different implementations, Pytorch version, CPU and GPU Fp32 is the FPS of the Pytorch implementation, fp16 is measured by converting the model to TensorRT static engine on the Jetson TX2, which may vary depending on different OS version The segmentation results are shown below: Figure 5.3 Prediction results of model BiseNet-ResNet18 on test data set 61 5.2 Real-time Navigation Results of Self-driving Car The autonomous car model was tested real-time on the road around HCMCUTE campus We record all measurements including the GPS tracks, EKF update, yaw heading and steering angle record during a test run from the road near block C to block A and finally stopped at block F h a) b) Figure 5.4 a) The map of HCMCUTE prepared using JavaScript OpenStreetMap tool, b) The route track used for the experiment Figure 5.5 Testing the self-driving car model on HCMUTE campus on a route track from block C to block F 62 h Figure 5.6 Result screen during testing Top left: The vehicle’s camera; Top right: GPS tracking simulation; Bottom right: Third-person camera Figure 5.7 Yaw rate of the autonomous car from ground truth (blue) and EKF estimation update over 2260 timesteps Figure 5.8 shows the recorded GPS record tracked over the predefined route Blue points are the ground truth GPS measurements, green cross represent EKF measurements 63 correction over the original signals The maximum speed of vehicle on straight road is measured to be approximately km/h Figure 5.8 GPS record over a test run Green: Original GPS measurements; Green: Extended Kalman Filter update over original values h The figure below demonstrates the result of the PID controller for camera steering based on error average point of image segmentation Figure 5.9 Steering angle recorded during the test run Green signal is the original steering signals, blue signals is the PID filtered signals 64 CHAPTER 6: CONCLUSION AND FUTURE WORK 6.1 Conclusion In conclusion, our group manages to build a system for Autonomous Ground vehicle based on Camera Image Segmentation and GPS and IMU lateral controller which run on HCMUTE school campus The requirements of the car include lane navigation, people, car, and pedestrian segmentation, rudimentary car avoidance and position tracking For the task of straight lane navigation, we use a Segmentation Model called BiseNet to segment images coming from the camera and derive a heading position for the vehicle using the Edge-detected segmented images and geometric operations As for position tracking and navigation, we design a waypoint-based lateral controller to steer the vehicle with respect to a designated GPS track Our GPS measurements are collected, corrected, and updated with the help of the Extended Kalman Filter Finally, we fuse the two controllers with flagbased method and used it to derive the final steering control of the vehicle in different road scenarios The final product is a NVIDIA Jetson TX2 development board stacked on the rear top of a 1/10 scale RC car along with a 130 POV camera, a smart phone to stream GPS, IMU signals through Wi-Fi 6.2 Work in the future h In this article, the authors used a convolutional neural network to perform image segmentation from Camera data After separating the border from the segmented image, we extract the boundary coordinates based on trigonometry, thereby deduce the steering angle to control the vehicle To make the car move smoothly on the road, the team proposed GPS and IMU sensors to help with positioning on the map However, there are still roadblocks in this project The following works are expected: - Improved segmentation results, improved pedestrian segmentation results - Integrated network to recognize traffic signs - Recommend LiDAR sensor to increase the ability to locate and avoid obstacles - Use Kalman filter for sensor fusing between camera, GPS, and LiDAR 65 REFERENCE [1] C Häne et al., “3D visual perception for self-driving cars using a multi-camera system: Calibration, mapping, localization, and obstacle detection,” Image Vis Comput., vol 68, pp 14–27, Dec 2017, doi: 10.1016/J.IMAVIS.2017.07.003 [2] T Kim and T H Park, “Extended kalman filter (Ekf) design for vehicle position tracking using reliability function of radar and lidar,” Sensors (Switzerland), vol 20, no 15, pp 1–18, Aug 2020, doi: 10.3390/S20154126 [3] A A Housein, G Xingyu, W Li, and Y Huang, “Extended Kalman Filter Sensor Fusion in Practice for Mobile Robot Localization,” IJACSA) Int J Adv Comput Sci Appl., vol 13, no 2, 2022, Accessed: Aug 04, 2022 [Online] Available: www.ijacsa.thesai.org [4] E Bostanci, B Bostanci, N Kanwal, and A F Clark, “Sensor fusion of camera, GPS and IMU using fuzzy adaptive multiple motion models,” Soft Comput., vol 22, no 8, pp 2619–2632, Apr 2018, doi: 10.1007/S00500-017-2516-8 [5] S Verghese, “Solid State Technology and Devices Seminar Self-Driving Cars and Lidar” [6] C J Hegarty and E Chatre, “Evolution of the Global Navigation SatelliteSystem (GNSS),” Proc IEEE, vol 96, no 12, pp 1902–1917, 2008, doi: 10.1109/JPROC.2008.2006090 h [7] “(15) (PDF) Design and Implementation of Web-Based GPS-GPRS Vehicle Tracking System.” https://www.researchgate.net/publication/310267800_Design_and_Implementation_of_ Web-Based_GPS-GPRS_Vehicle_Tracking_System (accessed Aug 04, 2022) [8] J Redmon, S Divvala, R Girshick, and A Farhadi, “You only look once: Unified, real-time object detection,” Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit., vol 2016-December, pp 779–788, Dec 2016, doi: 10.1109/CVPR.2016.91 [9] S Ren, K He, R Girshick, and J Sun, “Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks,” IEEE Trans Pattern Anal Mach Intell., vol 39, no 6, pp 1137–1149, Jun 2015, doi: 10.48550/arxiv.1506.01497 [10] “Convolutional neural network Wikipedia.” https://en.wikipedia.org/wiki/Convolutional_neural_network (accessed Aug 04, 2022) [11] D A Roberts, S Yaida, and B Hanin, “The Principles of Deep Learning Theory,” Princ Deep Learn Theory, Jun 2021, doi: 10.1017/9781009023405 [12] “(15) (PDF) AN AUTOMATED IMAGE ANALYSIS SYSTEM FOR ON LINE STRUCTURAL CHARACTERIZATION OF THE ACTIVATED SLUDGE FLOCS.” https://www.researchgate.net/publication/242410957_AN_AUTOMATED_IMAGE_AN ALYSIS_SYSTEM_FOR_ON_LINE_STRUCTURAL_CHARACTERIZATION_OF_T HE_ACTIVATED_SLUDGE_FLOCS/figures?lo=1 (accessed Aug 04, 2022) [13] D Bradley and G Roth, “Adaptive Thresholding using the Integral Image,” J Graph Tools, vol 12, no 2, pp 13–21, Jan 2007, doi: 66 10.1080/2151237X.2007.10129236 [14] “Canny edge detector https://people.eecs.berkeley.edu/~sastry/ee20/cademo.html (accessed Aug 04, 2022) [15] A A Novikov, D Lenis, D Major, J Hladuvka, M Wimmer, and K Buhler, “Fully Convolutional Architectures for Multi-Class Segmentation in Chest Radiographs,” IEEE Trans Med Imaging, vol 37, no 8, pp 1865–1876, Jan 2017, doi: 10.48550/arxiv.1701.08816 [16] Y Sugawara, S Shiota, and H Kiya, “Checkerboard artifacts free convolutional neural networks,” APSIPA Trans Signal Inf Process., vol 8, pp 1–9, 2019, doi: 10.1017/ATSIP.2019.2 [17] K He, X Zhang, S Ren, and J Sun, “Deep residual learning for image recognition,” Proc IEEE Comput Soc Conf Comput Vis Pattern Recognit., vol 2016-December, pp 770–778, Dec 2016, doi: 10.1109/CVPR.2016.90 [18] F Chollet, “Xception: Deep learning with depthwise separable convolutions,” Proc - 30th IEEE Conf Comput Vis Pattern Recognition, CVPR 2017, vol 2017-January, pp 1800–1807, Nov 2017, doi: 10.1109/CVPR.2017.195 [19] M Tan and Q V Le, “EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks” h [20] E Shelhamer, J Long, and T Darrell, “Fully Convolutional Networks for Semantic Segmentation,” IEEE Trans Pattern Anal Mach Intell., vol 39, no 4, pp 640–651, Nov 2014, doi: 10.48550/arxiv.1411.4038 [21] O Ronneberger, P Fischer, and T Brox, “U-net: Convolutional networks for biomedical image segmentation,” Lect Notes Comput Sci (including Subser Lect Notes Artif Intell Lect Notes Bioinformatics), vol 9351, pp 234–241, 2015, doi: 10.1007/9783-319-24574-4_28 [22] A Paszke, A Chaurasia, S Kim, and E Culurciello, “ENet: A Deep Neural Network Architecture for Real-Time Semantic Segmentation” [23] “CamVid (Cambridge-Driving Labeled Video Database) | Kaggle.” https://www.kaggle.com/datasets/carlolepelaars/camvid (accessed Aug 04, 2022) [24] “Cityscapes Dataset – Semantic Understanding of Urban Street Scenes.” https://www.cityscapes-dataset.com/ (accessed Aug 04, 2022) [25] “A single application for all your vision AI needs - Hasty.ai.” https://hasty.ai/ (accessed Aug 04, 2022) [26] C Yu, “BiSeNet: Bilateral Segmentation Network for Real-time Semantic Segmentation” [27] J Kong, M Pfeiffer, G Schildbach, and F Borrelli, “Kinematic and dynamic vehicle models for autonomous driving control design,” IEEE Intell Veh Symp Proc., vol 2015-August, pp 1094–1099, Aug 2015, doi: 10.1109/IVS.2015.7225830 [28] “(15) (PDF) Stanley: The robot that won the DARPA Grand Challenge.” 67 https://www.researchgate.net/publication/220648006_Stanley_The_robot_that_won_the_ DARPA_Grand_Challenge (accessed Aug 04, 2022) [29] “OpenStreetMap.” (accessed Aug 04, 2022) https://www.openstreetmap.org/#map=14/53.8531/27.6113 [30] M I Ribeiro, “Kalman and Extended Kalman Filters: Concept, Derivation and Properties,” 2004 h 68 h S K L 0
Ngày đăng: 06/11/2023, 09:25
Xem thêm:
