Research, design and construct a quadcopter for searching accidents in the outdoor environment

HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY OF ELECTRICAL AND ELECTRONICS ENGINEERING DEPARTMENT OF AUTOMATIC CONTROL ***** GRADUATION THESIS Research, Design and Construct a Quadcopter for Searching Accidents in the Outdoor Environment Students: Nguyen Thanh Trung Tran Ngoc Khanh Advisor: My-Ha Le, Ph.D – 15151236 – 15151163 Ho Chi Minh City, July 2019 HCMC UNIVERSITY OF TECHNOLOGY AND EDUCATION SOCIALIST REPUBLIC OF VIETNAM FACULTY OF ELECTRICAL AND ELECTRONICS Independence – Freedom – Happiness ENGINEERING o0o -DEPARTMENT OF AUTOMATIC CONTROL Ho Chi Minh City, Jul 2nd, 2019 THESIS ASSIGNMENTS Student 1: Nguyễn Thành Trung – Student ID: 15151236 Student 2: Trần Ngọc Khanh – Student ID:15151163 Major: Automation and Control Engineering Technology Training system: Formal Training System Academic year: 2015 – Class: 151511A I PROJECT: Research, Design and Construct a Quadcopter for Searching Accidents in the Outdoor Environment Advisor: My-Ha Le, Ph.D II ASSIGNMENTS: Collected Data: - Condition environment: indoor and outdoor - Flight time: > 10mins - Control limitation: > 100m Implementation Content: - Build model of quadcopter with camera - Build a flight control algorithm based on the PID controller and tune PID - Build landing algorithm with image processing - Collect data - Write thesis III ASSIGNED DATE: March 22nd, 2019 IV COMPLETE DATE: June 30th, 2019 V ADVISOR: My-Ha Le, Ph.D ADVISOR DEPARTMENT OF AUTOMATIC CONTROL My-Ha Le, Ph.D i HCMC UNIVERSITY OF TECHNOLOGY AND EDUCATION SOCIALIST REPUBLIC OF VIETNAM FACULTY OF ELECTRICAL AND ELECTRONICS Independence – Freedom – Happiness ENGINEERING o0o -DEPARTMENT OF AUTOMATIC CONTROL Ho Chi Minh City, Jul 2nd, 2019 PROJECT IMPLEMENTATION SCHEDULE Student 1: Nguyen Thanh Trung – Student ID: 15151236 Student 2: Tran Ngoc Khanh – Student ID:15151163 Major: Automation and Control Engineering Technology Academic year: 2015 – Class: 151511A Thesis: Research, Design and Construct a Quadcopter for Searching Accidents in the Outdoor Environment Advisor: My-Ha Le, Ph.D Week/Date Content Mar 22nd – 24th, 2019 - Survey and selection topic Mar 25th – 31st, 2019 - Build model of quadcopter for flight Apr 1st – 7th, 2019 - Test and fix model of quadcopter for flight Apr 8th – 14th, 2019 - Build and program for controller Apr 15th – 21st, 2019 - Test flight with controller transmitter and tune PID Apr 22nd – 30th, 2019 - Upgrade with ultrasonic sensor HC-SR04 May 1st – 5th, 2019 - Test and debug code with new flight code for keep the height May 6th – 12th, 2019 - Upgrade with GPS NEO-6M and transmit GPS signal May 13th – 19th, 2019 - Test camera with Raspberry Pi (how to connect, how to work) ii Advisor confirm May 20th – 31st, 2019 - Research and study about Machine learning, AI, image processing; - Comparing and choose a suitable model with our task and hardware; - Generation data for training Jun 1st – 9th, 2019 - Implement training the data; - Debug code detection and recognize Jun 10th – 16th, 2019 - Connection Arduino Uno and Raspberry Pi Build the code Jun 17th – 23th, 2019 - Combination and correction of the program - Final test and recording video, image Jun 24th – 30th, 2019 - Complete graduate thesis ADVISOR My-Ha Le, Ph.D iii HCMC UNIVERSITY OF TECHNOLOGY AND EDUCATION SOCIALIST REPUBLIC OF VIETNAM FACULTY OF ELECTRICAL AND ELECTRONICS Independence – Freedom – Happiness ENGINEERING o0o -DEPARTMENT OF AUTOMATIC CONTROL Ho Chi Minh City, Jul 2nd, 2019 ADVISOR’S COMMENT SHEET Student 1: Nguyen Thanh Trung – Student ID: 15151236 Student 2: Tran Ngoc Khanh – Student ID:15151163 Major: Automation and Control Engineering Technology Academic year: 2015 – Class: 151511A Thesis: Research, Design and Construct a Quadcopter for Searching Accidents in the Outdoor Environment Advisor: My-Ha Le, Ph.D COMMENT About the thesis’s contents: - Students have completed the requirements of graduation thesis Advantage: - The system works stably in real time - The system detects accident with high accuracy Disadvantage: - Need to experiment on more diverse data Propose defending thesis? - Yes Rating: - Excellent Mark: 9.8/10 (In writing: nine point ten) Ho Chi Minh City, July 4th 2019 ADVISOR My-Ha Le, Ph.D iv v vi HCMC UNIVERSITY OF TECHNOLOGY AND EDUCATION SOCIALIST REPUBLIC OF VIETNAM FACULTY OF ELECTRICAL AND ELECTRONICS Independence – Freedom – Happiness ENGINEERING o0o -DEPARTMENT OF AUTOMATIC CONTROL Ho Chi Minh City, Jul 2nd, 2019 COMMITMENT We commit the contents and research data presented on this graduate thesis is executed by ourselves The excerpts and image used in the thesis are attached their references, which are public in the correct rule The thesis is our knowledge about this project STUDENT Nguyen Thanh Trung STUDENT Tran Ngoc Khanh vii ACKNOWLEDGMENT We finally completed my final project the end of my education in Ho Chi Minh City University of Technology and Education To be honest, it is not perfect but we have tried whole us term to doing this project better day by day Up to now, we place on record our sincere thankful to Faculty of Electrical and Electronics Engineering and Department of Automatic Control supporting us the thing we need and allowing us to execute this project We wish to express our sincere thankful to My-Ha Le, Ph.D giving, instructing and supporting us then research this project We are grateful to Do Truong Dong, Duong Minh Thien, Tran Le Anh, Le Tien Sy, Nguyen Trung Hieu, Ngo Tran Khanh Dang, Vo Minh Cong, Dang Quoc Vu, Nguyen Duy Thong, for providing us the necessary facilities for the research In this term of us, we have friends, who are Le Manh Cuong, Dao Duy Phuong, Phan Vo Thanh Lam, Vo Anh Quoc, Tran Van Son, Duong Thuy Huynh We the final project together in the laboratory They have supported too much such as telling us about this/that is not good or good, on processing helped me collecting data and ideas which help us better and developing us project day by day Specially, we also thank our parents, family and friends for the encouragement, support and attention viii ABSTRACT In this report, we explained the theory of quadcopter dynamics, image processing theory, flight control theory and principles of operation of the circuit board used Furthermore, we have focused on machine learning, artificial intelligence It is a comprehensive overview of using deep learning-based object detection methods for aerial image via Quadcopter The result of our project is target detection and positioning system and aerial image collection are developed and integrated into Quadcopter Base on the results obtained from reality, we assess and propose future directions of development ix Figure 5.12: Classification loss value Figure 5.13: Localization loss value Figure 5.14: The result of the traning 108 Final of step traing, we have evaluated our model by code python, we took about 150 images in test_image to evaluate As the result of evaluation, we have mAP (mean Average Precision) 0.9999915 5.6 Compare MobileNet-SSD Model After training and receiving results, we have made some comparisons about the effectiveness of the MobileNet-SSD model used with some other existing methods [32] With traditional models that apply to identification, the model we use has higher accuracy, and response time is also appropriate When compared to the HMM-only model, both response time and accuracy are higher As for MobileNet model only, the model used also shows the superiority We show the comparison in the Figure 5.15 and 5.16 Figure 5.15: Compare with HMM and SVM-HMM Figure 5.16: Compare with MobileNet only 109 5.7 Flight and Detection Results 5.7.1 Flight Results 5.7.1.1 Indoor Results b) Figure 5.17: Indoor flight a) c) a) In the room; b) In the hallway; c) In the lobby 5.7.1.2 Outdoor Results Figure 5.18: Manual flight control a) c) b) Figure 5.19: Daylight flight results a) Turn right; b) Flight on the grass; c) Flight on the asphalt road 110 b) a) Figure 5.20: Evening light flight results a) Flight on the grass; b) Flight on the asphalt road 5.7.2 Detection Results 5.7.2.1 Indoor Results Before testing the outdoor environment, we tested it in laboratory conditions In this environment, the objects are all very well detected with no obstruction and good lighting We can detect single object, or multiple objects easily As shown in the figure below, the quadcopter can detect three objects at the same time The speed of quadcopter is 3-4km/h while detecting Accuracy up to 90% In Figure 5.21, we show some typical identification results a) b) c) Figure 5.21: Typical Indoor Detection a) Two objects detected; b) Two objects detected, with a lying and a standing; c) Three objects detected, with two objects lying and one standing For this environment, objects can still be identified when not obscured by strange objects In case, the obscured object is larger than 50% of the shape of the object, the model does not have the correct identity Figure 5.22 and 5.23 shows the ability to identify objects that are obscured 111 b) a) c) Figure 5.22: Detectable the object obscured less than 50% a) Only object obscured by black object; b) Object obscured by black object and normal object; c) Object obscured by green object and normal object Figure 5.23: Undetectable the object obscured more than 50% 5.7.2.2 Outdoor Results 5.7.2.2.1 Daylight Environment After testing in indoor environment We tested model in outdoor environment The first is in daylight conditions The quadcopter that meet good detection requirements are slightly better at home when objects under relatively high light intensity are still successfully detection Some environmental parameters in experiment: temperature of about 30-35 degrees C, no wind, light intensity moderate, on the grass and on the road Figure 5.24 and 5.25 shows typical result of outdoor detection 112 a) b) c) d) e) f) Figure 5.24: Typical daylight detection results on the road a) Lying object in hardlight; b) Three objects detected; c) Two objects detected; d) Standing and lying objects; e) Object in pavement; f) Standing object a) b) c) d) e) Figure 5.25: Typical daylight detection results on the grass a) Lying object in hard light; b) Three objects detected; c) Lying Object; d) Two objects detected; e) Standing object Figure 5.26: The Sore confidence around 97% 113 5.7.2.2.2 Evening Light Environment To test more of the capabilities of the model, we have experimented in an evening environment The light in this environment achieves a similar brightness to daylight We show some results in Figure 5.27 a) b) c) d) Figure 5.27: Typical evening light detection results a), b) Object on the road; c), d) Object on the grass Figure 5.28: The Score confidence around 69% 114 5.7.2.2.3 The Object Obscured and Lack of Light Like the indoor experiments, with the object obscured, our model can still be identified when not obscured by strange objects In case, the obscured is larger than 50% of the shape of the object, the model does not have the correct identity a) b) Figure 5.29: Detectable the object obscured less than 50% a) On the road; b) On the grass a) b) Figure 5.30: Undetectable the object obscured more than 50% a) On the grass; b) On the road When we test in the evening, also had the same problem And in evening, the light is much less than the day, so if not have enough light, the object will undetectable, we show it in Figure 5.31 a) b) Figure 5.31: Undetectable the object c) a) Too dark; b) Not enough light; c) In shadow 115 5.8 GPS Location Results Our GPS data collection station connects to a computer If no object is detected, it will only receive normal information When a signal is detected, the Buzzer will alarm, the observer will use this special information to check the location on the map Figure 5.32: GPS Station a) b) Figure 5.31: Typical Map of Fight Route and Detected Location a) Map of Route 1; b) Map of Route 116 This is a map for us to track the flight area, the route it has taken and the location where detected the object needs rescue We use GPS signals to locate Once data is available, they will be transferred to the receiving and backup station in Excel To observe, we apply mapmakerapp.com web address for creating online maps After more than three months of research and development, we have designed and assembled a complete quadcopter model The experimental results have executed indoor many times, after that, outdoor in some different environment Experimental results indicated: - The quadcopter works very well indoors and outdoors - Operation with easy control, good implementation of flight route requirements - Program recognition works with high accuracy in object detection and ensures normal flight operation - Positioning via GPS is relatively stable 117 CHAPTER 6: CONCLUSIONS AND DEVELOPMENTS 6.1 Conclusions 6.1.1 Achieved Results After more than three months of research and implementation, the project “Research, Design and Construct a Quadcopter for Searching Accidents in the Outdoor Environment” has been completed with our efforts After many attempts and error, the system now operates with the good results We have achieved some following contents: - Learn about flight principles and quadcopter control principles - Solve dynamic equilibrium problems for models in x, y and z axes - Constructing own control algorithm model - Understood the flight principles of the design model - Understood the flight principles of the design model - Understood model neural network such as mobilenet-SSD - The training model has adapted with tasks of system quadcopter as indoor and outdoor environments 6.1.2 Advantaged - As the result of evaluation, we have mAP (mean Average Precision) 0.9999915 It can be seen that the precision of the models is generally high, indicating good bounding box prediction performance - The final detection time of our algorithm was merely 380 milliseconds per piece, which meets the real-time requirements - Improving and increasing about frame per second approximate – fps with our hardware (Raspberry Pi 3) 118 - Low project execution price, optimal size 6.1.2 Disadvantaged Besides the achieved results, the project has many limitations as follows: - Using the modules is only applicable, not have a deep understanding of their operation - Although solving the equilibrium problem allows the model to take off and fly in the air, but the balance is not really good, the level of fluctuation remains, the stability is not very high - Limited about the speed of quadcopter system to detect object tasks 6.2 Development Directions Due to the flexibility in control, UAV systems have a wide application in all civil and military fields, such as search and rescue operation, tracking object, delivery product Nowadays, experts are outlining positive methodologies to solve traffic problems; notably, tracking a crime or supporting victims of a traffic accident A potential method proposed in this thesis is to use the UAV as a supporting robot since its traveling path is less blocked from the air than that of the unmanned ground vehicles (UGVs) Currently, researches on the UAVs for SAR is still underway in the world An improvement for the system to be capable with higher accuracy will be proposed in the future 119 REFERENCES [1] Le Tien Sy, Nguyen Thanh Binh, and Huynh Ngoc Thuong, “Scientific Research Report: Design and Construction of Robot Quadrotor”, HCMC University of Technology and Education, 2015 [2] Tran Le Anh and Le Tuan Thong, “Design and Implement a Quadcopter Model Autonomously Landing on a Stationary Target”, HCMC University Technology and Education, 2018 [3] Ngo Tran Khanh Dang, “Simulate Quadrotor in Simulink with SimMechanics”, HCMC University of Technology, 2014 [4] Lam Ngoc Tam, “Design and Manufacture of Flight Model – Quadrocopter”, The University of Da Nang, 2012 [5] Bui Quang Minh, “Lagrange mechanics”, Vietnamese – Germany University, 2015 [6] Phung Thai Son, “Introducing how to use Ublox's NEO and NEO GPS modules”, Arduino Viet Nam, 2017 [7] Nguyen Manh Hung, “Using NRF24L01 Module - 2.4GHz radio transceiver with Arduino”, Arduino Viet Nam, 2015 [8] Nguyen Tan Phuc, “Dynamics for Robots”, Nong Lam University – HCMC, 2014 [9] Teppo Luukkonen “Modelling and control of quadcopter”, Aalto University, 2011 [10] Shaikh Altamash, Syed Adnan and Padwekar Aamir, “Kinematic, Dynamic Modeling and Simulation of Quadcopter”, University of Mumbai, 2016 [11] Joop Brokking, “Project YMFC-AL - The Arduino auto-level quadcopter”, 2017 [12] Andrew Gibiansky, “Quadcopter Dynamics, Simulation and Control”, 2012 120 [13] Charles Tytler, “Modeling and simulation of a quadcopter’s vehicle dynamics”, 2017 [14] Matej Andrejašiˇc, “MEMS Accelerometers”, University of Ljubljana, 2008 [15] Kenzo Nonami, Farid Kendoul, Satoshi Suzuki, Wei Wang and Daisuke Nakazawa, “Autonomous Flying Robots: Unmanned Aerial Vehicles and Micro Aerial Vehicles”, Springer, 2010 [16] Axel Reizenstein, “Position and Trajectory Control of a Quadcopter Using PID and LQ Controllers”, Linkưping University, 2017 [17] Carlos Murga, “Rigid-Body Dynamics”, Eindhoven University of Technology, 2016 [18] Haomiao Huang, Gabriel M Hoffmann, Steven L Waslander and Claire J Tomlin, “Aerodynamics and Control of Autonomous Quadrotor Helicopters in Aggressive Maneuvering”, 2009 IEEE International Conference on Robotics and Automation Kobe International Conference Center, Kobe, Japan, May 12-17, 2009 [19] I Can Dikmen, Aydemir Arısoy and Hakan Temeltaş, “Attitude Control of a Quadrotor”, IEEE, 2009 [20] Jay Esfandyari, Roberto De Nuccio, Gang Xu, “Introduction to MEMS gyroscopes”, STMicroelectronics, 2010 [21] Johann-Sebastian Pleban, Ricardo Band and Reiner Creutzburg, “Hacking and securing the AR.Drone 2.0 quadcopter - Investigations for improving the security of a toy”, Brandenburg University of Applied Sciences, 2014 [22] InvenSense Inc., “MPU-6000 and MPU-6050 Register Map and Descriptions Revision 4.2”, 2013 [23] MIT OpenCourseWare, “Design of Electromechanical Robotic Systems”, 2010 [24] García Carrilo, Dzul López, Lozano and Pégard, “Quad Rotorcraft Control”, Spinger, 2013 121 [25] Pau Seg Gascó, “Development of a Dual Axis Tilt Rotorcraft UAV: Modelling, Simulation and Control”, Cranfield University, 2012 [26] Implementation of Regional-CNN and SSD Machine Learning Object Detection Architectures for the Real Time Analysis of Blood Borne Pathogens in Dark Field Microscopy, July 2018 [27] Fast Object Detection for Quadcopter Drone using Deep Learning, April 2018 [28] Wei Liu1, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, Alexander C Berg, “SSD: Single Shot MultiBox Detector”, Dec 2016 [29] MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications, Apr 2017 [30] A deep learning based solution for construction equipment detection: from development to deployment, April 2019 [31] Convolutional neural networks (CNN) explanation and implementation, November 2018 [32] Yiting Li, Haisong Huang, Qingsheng Xie, Liguo Yao and Qipeng Chen, “Research on a Surface Defect Detection Algorithm Based on MobileNet-SSD”, Applied Sciences, September 2018 122 ... system: Formal Training System Academic year: 2015 – Class: 15151 1A I PROJECT: Research, Design and Construct a Quadcopter for Searching Accidents in the Outdoor Environment Advisor: My-Ha Le,... region, the UAV market has been segmented into North America, Europe, Asia Pacific, the Middle East, Latin America, and Africa North America is estimated to be the largest market for UAV in 2018 The. .. aerial imaging and notification for the emergency center Therefore, the team has delighted in researching and learning things above at the last of us term 1.2 Objectives of the thesis The thesis