Quadcopter bám theo vật thể dựa trên mô hình xử lý ảnh deep learning

ĐẠI HỌC QUỐC GIA TPHCM TRƯỜNG ĐẠI HỌC BÁCH KHOA NGUYỄN PHÚC KHÔI QUADCOPTER BÁM THEO VẬT THỂ DỰA TRÊN MƠ HÌNH XỬ LÝ ẢNH DEEP-LEARNING OBJECT TRACKING QUADCOPTER BASED ON DEEPLEARNING IMAGE PROCESSING MODEL Chuyên ngành: Kỹ thuật Điều khiển Tự động hóa Mã số: 60.52.02.16 LUẬN VĂN THẠC SĨ TP HỒ CHÍ MINH, tháng 09 năm 2020 CƠNG TRÌNH ĐƯỢC HỒN THÀNH TẠI TRƯỜNG ĐẠI HỌC BÁCH KHOA –ĐHQG -HCM Cán hướng dẫn khoa học : PGS TS Huỳnh Thái Hoàng Cán chấm nhận xét : TS Nguyễn Trọng Tài Cán chấm nhận xét : TS Ngô Thanh Quyền Luận văn thạc sĩ bảo vệ Trường Đại học Bách Khoa, ĐHQG Tp HCM ngày tháng năm 2020 Thành phần Hội đồng đánh giá luận văn thạc sĩ gồm: GS TS Hồ Phạm Huy Ánh TS Nguyễn Vĩnh Hảo TS Nguyễn Trọng Tài PGS TS Trương Đình Nhơn TS Ngơ Thanh Quyền CHỦ TỊCH HỘI ĐỒNG TRƯỞNG KHOA ĐẠI HỌC QUỐC GIA TP.HCM TRƯỜNG ĐẠI HỌC BÁCH KHOA CỘNG HÒA XÃ HỘI CHỦ NGHĨA VIỆT NAM Độc lập - Tự - Hạnh phúc NHIỆM VỤ LUẬN VĂN THẠC SĨ Họ tên học viên: NGUYỄN PHÚC KHÔI MSHV: 1770287 Ngày, tháng, năm sinh: 02/04/1994 Nơi sinh: TPHCM Chuyên ngành : Kỹ thuật Điều khiển Tự động hóa Mã số : 60520216 I TÊN ĐỀ TÀI : Quadcopter bám theo vật thể dựa mơ hình xử lý ảnh deep-learning II NHIỆM VỤ VÀ NỘI DUNG : - Lắp ráp mơ hình quadcopter với camera video transmitter Huấn luyện mơ hình CNN YOLOv3 để nhận dạng lớp đối tượng Xây dựng xử lý tín hiệu logic mờ để xuất tín hiệu PWM điều khiển quadcopter bám theo vật III NGÀY GIAO NHIỆM VỤ : 10/02/2020 IV NGÀY HOÀN THÀNH NHIỆM VỤ : 25/08/2020 V CÁN BỘ HƯỚNG DẪN : PGS TS Huỳnh Thái Hoàng Tp HCM, ngày… tháng… năm 2020 CÁN BỘ HƯỚNG DẪN CHỦ NHIỆM BỘ MÔN ĐÀO TẠO TRƯỞNG KHOA….……… Object tracking Quadcopter ACKNOWLEDGEMENT First of all, I am grateful to all lecturers of Control and Automation Department for their devotion, dedication and enthusiasm They have provided me extensive personal and professional guidance and taught me a great deal about both scientific research and life in general Learning all the best from them is my pleasure Secondly, I would like to express my deepest gratitude Assoc Prof Huynh Thai Hoang for sharing expertise, sincere and valuable guidance and encouragement extended to me, without his dedicated support, I could not complete my goal Finally, I must express my very profound gratitude to my parents and to teammates – Phạm Ngọc Trung, Trần Kha for providing me with unfailing support and continuous encouragement throughout my years of study and through the process of researching and writing this thesis This accomplishment would not have been possible without them Ho Chi Minh City, Monday, 25th August, 2020 Student Nguyễn Phúc Khôi i Object tracking Quadcopter ABSTRACT Over the last few years, remote control airborne vehicles have been incredibly popular for hobbyists, academic research groups, and for individuals who are planning for a start-up Unmanned Arial Vehicles (UAV) have made great advances in automation, ability to perform complex tasks and ability to make decisions on their own The research “Object tracking quadcopter bases on computer vision and fuzzy logic processor” has been made for the purpose of controlling a quadcopter which has the ability to detect and follow an identified object A Convolutional Neural Network (CNN) named YOLOv3 is used for objection recognition and a fuzzy processor is used for determining the PWM signals sent to the Ardupilot microcontroller The Quadcopter with Ardupilot microcontroller and PID firmware is used for this project The ground station is built with GPU GTX 1060 (VRAM 6GB), CPU AMD Ryzen Six-Core (~3.2GHz) to run YOLOv3 model to detect classes of object and the user can choose which object to follow The main object will be located with boungding box, center point of bounding box, x-distance and y-distance between center point of bounding box and center point of camera frame As the training process takes a lot of time and computer resources, the weights of the model is trained on Google Colab with Darknet – an open source neural network framework written in C and CUDA There are fuzzy processors also built on ground station for throttle, roll, pitch and yaw which take the x-distance and y-distance as input and output the PWM signals to quadcopter Instead of getting a person to play with the transmitter, the fuzzy processors act as a transmitter All the materials are displayed on a python graphic user interface The user can observe the object detection progress and other parameters in real time ii Object tracking Quadcopter TÓM TẮT LUẬN VĂN Trong vài năm qua, phương tiện bay điều khiển từ xa trở nên vô phổ biến người có sở thích, nhóm nghiên cứu học thuật cá nhân có kế hoạch khởi nghiệp Phương tiện vận tải không người lái (UAV) có bước tiến vượt bậc tự động hóa, khả thực nhiệm vụ phức tạp khả tự đưa định Nghiên cứu “Quadcopter theo dõi đối tượng dựa thị giác máy tính xử lý logic mờ” thực với mục đích điều khiển quadcopter có khả phát theo dõi đối tượng xác định Mạng thần kinh hợp pháp (CNN) có tên YOLOv3 sử dụng để nhận dạng phản đối xử lý mờ sử dụng để xác định tín hiệu PWM gửi đến vi điều khiển Ardupilot Quadcopter với vi điều khiển Ardupilot phần sụn PID sử dụng cho dự án Trạm mặt đất xây dựng với GPU GTX 1060 (VRAM 6GB), CPU AMD Ryzen Six-Core (~ 3.2GHz) để chạy mơ hình YOLOv3 để phát lớp đối tượng người dùng chọn đối tượng theo dõi Đối tượng định vị với hộp tung lên, điểm trung tâm hộp giới hạn, khoảng cách x khoảng cách y điểm trung tâm hộp giới hạn điểm trung tâm khung máy ảnh Vì trình đào tạo tốn nhiều thời gian tài ngun máy tính, trọng số mơ hình đào tạo Google Colab với Darknet - khung mạng thần kinh mã nguồn mở viết C CUDA Có xử lý mờ xây dựng trạm mặt đất cho bướm ga, cuộn, cao độ ngáp lấy khoảng cách x khoảng cách y làm đầu vào đầu tín hiệu PWM tới quadcopter Thay để người chơi với máy phát, xử lý mờ hoạt động máy phát Tất tài liệu hiển thị giao diện người dùng đồ họa python Người dùng quan sát tiến trình phát đối tượng thông số khác thời gian thực iii Object tracking Quadcopter PLEDGE I would like to assure that this is my own research project, with the approval of my instructor, Assoc Prof Huynh Thai Hoang Relevant documents have clear origins and published on official websites The results presented in the thesis are the achievement of practical Ho Chi Minh city, 25th August 2020 Author Nguyễn Phúc Khôi iv Object tracking Quadcopter CONTENTS ACKNOWLEDGEMENT i ABSTRACT ii TÓM TẮT LUẬN VĂN iii PLEDGE iv CONTENTS .v LIST OF FIGURES viii LIST OF TABLES xi LIST OF ABBREVIATIONS xii Chapter INTRODUCTION 1.1 Overview .1 History of Unmanned Aerial Vehicles UAV Classification and Applications Classification according to size Classification According to Range and Endurance Applications Related studies .9 1.2 Objective of Study .13 1.3 Outline of the thesis 13 Chapter THEORITICAL BASIS .14 2.1 Convolutional Neural Network (CNN) and YOLOv3 Algorithm 14 Convolutional Neural Network (CNN) .14 Convolutional layer .14 Pooling layer 18 v Object tracking Quadcopter Fully connected layer 19 YOLOv3 Algorithm .19 Network structure 19 Anchor Box Algorithm 22 Output processing 22 2.2 Fuzzy Logic .23 Membership function 24 Fuzzy Logic – Inference System (FIS) 24 Chapter HARDWARE DESIGN .26 3.1 Quadcopter system overview 26 3.2 Quadcopter configuration 29 Quadcopter frame 29 APM 2.8 Flight controller 29 FPV Camera 31 Video transmitter and receiver .31 Brushless DC Motors 32 Propellers .33 Electronic Speed Controller (ESC) 34 Battery 34 Transceiver 35 Chapter SOFTWARE DESIGN .37 4.1 Quadcopter control diagram 37 4.2 Object detection 39 Training model for object detection 39 vi Object tracking Quadcopter Collecting images data 40 Train model on Google Colab 42 Calculate object position 43 4.3 Fuzzy logic algorithm 44 Chapter RESULT AND FUTURE DEVELOPMENT 48 5.1 Result 48 Quadcopter model 48 Object recognition result 48 Fuzzy logic result 53 Evaluation 53 Advantages 53 Limitation 54 5.2 Future Development 54 REFERENCES 56 APPENDIX 59 CURICULUM VITAE .64 vii Object tracking Quadcopter Figure 5.4 Pikachu and Rabbit detected in the front direction The figures below are showing the value of Top-Left coordinates and BottomRight coordinates variation in 35 seconds From to 25s, the figures describe the first tested position and from 25 to 35s, the figures describe the second tested position As we can see, the values of the coordinates oscillate with amplitude less than 0.1 which means the position of the bounding box is stable 50 Object tracking Quadcopter Figure 5.5 Line Plot of Top-Left x value over time Figure 5.6 Line Plot of Top-Left y value over time 51 Object tracking Quadcopter Figure 5.7 Line Plot of Bottom-Right x value over time Figure 5.8 Line Plot of Bottom-Right y value over time 52 Object tracking Quadcopter Fuzzy logic result Using the values dx, dy, dS derived from the image processing, the Fuzzy logic process is able to calculate the output throttle, yaw angle and pitch angle These values then sent to flight controller via bluetooth The string format: Roll , Pitch , Throt , Yaw * Checksum \r\n Checksum function returns a BCD value: ord() function in python returns an integer representing the Unicode character The string then received in flight controller: Roll value is always ‘0’ as we not control roll angle Evaluation Advantages Within the thesis, student has completed the assembly of quadcopter Research and implementation issues include:  Learning about the composition, sensors, flight controller, motors, propelers, power distribution board, …as a basis for designing the automation algorithm  Learning about the process of developing a convolutional neural network, collect dataset, training the model, adjust the configuration parameters and running with GPU 53 Object tracking Quadcopter  Developing a real-time object detection application for FPV camera  Learning about fuzzy logic, membership function, fuzzy rules, fuzzification, defuzzification  Developing a fuzzy logic processor to output the PWM values to controll the motors  Combining the object detection and fuzzy logic process to build a ground station which controls the quadcopter to track the object automatically Limitation Quadcopter stabilization is not perfectly complete therefore the object detection algorithm is affected The FPS of the object detection output images is quite low, around 11 FPS, it is sufficient for slow tracking but should be around 20-30 FPS for other applications The application is able to dectect only classes of objects Due to low resolution of the camera, the objects can only be detected within 50 cm from the camera The small dataset (400 photos/class) has led to incorrect recognition in some cases The areas of bounding boxes are different between front, back and side direction This will affect the pitch angle output of fuzzy processor Bluetooth has a low range of communication and is affected by obstacles, needs an open large space for best performance 5.2 Future Development In the future, we would like to build a complete application on quadcopter with gimbal to stabilize the camera and obstacle avoidance feature Currently, the quadcopter does not use the roll angle, as following object in an open space without obstacle does not require roll movement The camera is permanently attached to the quadcopter frame, therefore it lacks of flexibility, the object can move out of the 54 Object tracking Quadcopter vision The gimbal is now playing an important role, it helps stabilize the images and has more flexibility to track and lock the object In addition, when we use the quadcopter in the environments which contain many obstacles, we definitely need an algorithm for avoidance, and roll angle is an indispensable factor Taking off and landing automatically are also important features needed to be considered Now for safety, we manually control the quadcopter to take off and land But in many situations, the quadcopter must run without human command Furthermore, object detection algorithm must be enhanced in terms of speed and accuracy 55 Object tracking Quadcopter REFERENCES [1] M.Navabi, H.Mirzaei (2017), Robust Optimal Adaptive Trajectory Tracking Control of Quadcopter Helicopter Latin American Journal of Solids and Structures 14, pp 4-16 Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 20-01-2020 [2] Ramiro Ibarra Pérez (2017), Attitude Control of a Quadcopter Using Adaptive Control Technique, UAM Reynosa Rodhe, Autonomous University of Tamaulipas, Tamaulipas, México, pp 4-11 Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 20-01-2020 [3] Axel Reizenstein (2017), Position and Trajectory Control of a Quadcopter Using PID and LQ Controllers, Master of Science Thesis in Electrical Engineering, Department of Electrical Engineering, Linköping University, pp 17-63 Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 20-01-2020 [4] Ali Rohan, Mohammed Rabah, Sung-ho Kim (2019), " Convolutional Neural Network-Based Real-Time Object Detection and Tracking for Parrot AR Drone 2", School of Electronics and Information Engineering, Kunsan National University, pp 2-8 Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 20-01-2020 [5] Katsuya Ogura, Yuma Yamada, Shugo Kajita, Hirozumi Yamaguchi, Teruo Higashino, Mineo Takai (2019), "Ground object recognition and segmentationfrom aerial image-based 3D point cloud", Graduate School of Information Science and Technology, Osaka University, Osaka, Japan, pp 3-9 Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 20-01-2020 56 Object tracking Quadcopter [6] Sabir Hossain and Deok-jin Lee (2019), "Deep-Learning Based RealTime Multiple-Object Detection and Tracking from Aerial Imagery via Flying Robot with GPU-based Embedded Devices", School of Mechanical & Convergence System Engineering, Kunsan National University, pp 1-20 Link: https://www.researchgate.net/, date access: 3001-2020 [7] Anjan Chakrabarty and Robert Morris (2016), "Autonomous Indoor Object Tracking with the Parrot AR.Drone", 2016 International Conference on Unmanned Aircraft Systems (ICUAS) Link: https://www.researchgate.net/, date access: 30-01-2020 [8] Peng Chen, Yuanjie Dang, Ronghua Liang, Wei Zhu and Xiaofei He (2018), "Real-Time Object Tracking on a Drone With Multi-Intertial Sensing Data", IEEE transactions on itelligent transportation systems, Vol.19 Link: https://www.researchgate.net/, date access: 30-01-2020 [9] Daniel Bolya, Chong Zhou, Fanyi Xiao, Yong Jae Lee (2019), “YOLACT Real-time Instance Segmentation”, arXiv:1904.02689v2, pp 1-9 Link: https://www.researchgate.net/, date access: 30-01-2020 [10] Cheng-Yang Fu, Mykhailo Shvets, and Alexander C Berg (2019), "Retinamask: Learning to predict masks improves state-of-the-art singleshot detection for free", arXiv:1901.03353, pp 1-9 Link: https://www.researchgate.net/, date access: 30-01-2020 [11] Hengshuang Zhao, Xiaojuan Qi, Xiaoyong Shen, Jianping Shi, and Jiaya Jia (2018), "Icnet for real-time semantic segmentation on high-resolution images", In ECCV Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 10-02-2020 57 Object tracking Quadcopter [12] Joseph Redmon, Santosh Divvala, Ross Girshick, Ali Farhadi (2015), You Only Look Once: Unified, Real-Time Object Detection, University of Washington, Allen Institute for AI, pp 1-8 Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 10-02-2020 [13] Joseph Redmon, Ali Farhadi (2015), YOLO9000: Better, Faster, Stronger, University of Washington, Allen Institute for AI, pp 1-8 Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 10-02-2020 [14] Kimon P.Valavanis (2007), “Advances in Unmanned Aerial Vehicles,” 1st Ed., Springer Ltd, pp 2-5 Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 10-02-2020 [15] Zdenek Kalal, Krystian Mikolajczyk, and Jiri Matas (2011), TrackingLearning-Detection, IEEE Transactions on pattern analysis and machine intelligence, Vol 6, No 1, pp 1-12 Link: https://ieeexplore.ieee.org/Xplore/home.jsp, date access: 10-02-2020 58 Object tracking Quadcopter APPENDIX HARDWARE SPECIFICATIONS Camera FPV Table Specifications of Predator mini V4 Parameters Value Image sensor CMOS Horizontal Resolution 1000TVL Aspect Ratio 16:9/4:3 Signal System NTSC/PAL Lens Size M12 Focal Length 1.8mm Minimum Latency 4ms Input Voltage 5-40V Dimensions 21.8x21.8mm Video transmitter and receiver Table Specifications of Foxeer ClearTX Parameters Value Weight 6.3g Power 25/200/500/800mW Input 2-6S 59 Object tracking Quadcopter Channel 48 Dimensions 22x32mm Table Specifications of Skydroid 5.8GHz receiver Parameters Value Frequency 5645MHz-5945MHz Working Current 190mA Latency ~100ms Dimensions 59x40x11mm Brushless motors Table Emax RS2205 2300KV specifications Parameters Value Kv (rpm/V) 2300 No-load current I0 1.8A at 10V Max performance 80% Current at max performance – 21.7 (A) Lipo battery – cells Max current 25 (A) Working current < 21A 60 Object tracking Quadcopter Weight 30g Size 28  32mm Shaft diameter 5mm Propellers 5045 or 6050 Table Thrust specifications of Emax RS2205 2300KV with 3S battery Prop(in) Volts (V) Amps (A) Thrust (G) Rpm/min Power (W) Efficiency (G/W) 5045 12 77 7010 12 6.42 5045 12 3.1 186 10820 37.20 5.00 5045 12 5.1 283 13010 61.20 4.62 5045 12 7.1 361 14720 85.20 4.24 5045 12 9.1 432 16140 109.20 3.96 5045 12 11.1 499 17360 133.20 3.75 5045 12 13.1 571 18450 157.20 3.63 5045 12 15.1 633 19370 181.20 3.49 5045 12 17.1 698 20290 205.20 3.40 5045 12 19.1 758 20940 229.20 3.31 5045 12 21.7 825 21750 260.40 3.17 61 Object tracking Quadcopter ESC Table Emax BLHeli 30A ESC Specifications Parameters Value Operating current 30A Max current 40A Lipo Battery – 3S BEC Yes BEC Output 5V/5A Weight 11g Dimension 25*12 mm Lipo battery Table 10 1500mAh 25C Lipo battery specifications Parameters Value Dimensions 70*35*26mm Capacity 1500mAh Voltage 11.1V (3S) Discharge current 25C Weight 180g 62 Object tracking Quadcopter Control transmitter and receiver Table 11 FlySky FS-i6 Transmitter specifications Parameters Value Channels RF Range 2.40 – 2.48 GHz Power 6V Bandwidth 500 KHz RF Power < 20 dbm Dimensions 174*89*190mm Antenna length 26mm*2 (dual antenna) Table 12 FlySky FS-i6 Receiver specifications Parameters Value Channels RF Range 2.40 – 2.48 GHz Nguồn cấp 5V Kích thước 40.4*21.1*7.35mm 63 Object tracking Quadcopter CURICULUM VITAE Full name: Nguyễn Phúc Khôi Date of birth: 02/04/1994 Place of birth: Ho Chi Minh City Address: 285/101/30 CMT8, Ward 12, District 10, HCM city EDUCATION 2012-2017: Engineering Degree at Bach Khoa University, Control and Automation Department 2017-2020: Master Engineering at Bach Khoa University, Control and Automation Department WORK 2016-2020: Robert Bosch Engineering and Business Solutions Viet Nam 64 ... I TÊN ĐỀ TÀI : Quadcopter bám theo vật thể dựa mơ hình xử lý ảnh deep- learning II NHIỆM VỤ VÀ NỘI DUNG : - Lắp ráp mơ hình quadcopter với camera video transmitter Huấn luyện mơ hình CNN YOLOv3... phức tạp khả tự đưa định Nghiên cứu ? ?Quadcopter theo dõi đối tượng dựa thị giác máy tính xử lý logic mờ” thực với mục đích điều khiển quadcopter có khả phát theo dõi đối tượng xác định Mạng thần... luyện mơ hình CNN YOLOv3 để nhận dạng lớp đối tượng Xây dựng xử lý tín hiệu logic mờ để xuất tín hiệu PWM điều khiển quadcopter bám theo vật III NGÀY GIAO NHIỆM VỤ : 10/02/2020 IV NGÀY HOÀN THÀNH