(Đồ án hcmute) design a conveyor system to sort products using qr code

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT ELECTRONICS AND TELECOMMUNICATION ENGINEERING TECHNOLOGY DESIGN A CONVEYOR SYSTEM TO SORT PRODUCT USING QR CODE LECTURER: Ph.D DO DUY TAN STUDENT: NGUYEN PHAM DUY THAI SKL 0 Ho Chi Minh City, December, 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION THESIS MAJOR: ELECTRONICS AND TELECOMMUNICATION ENGINEERING TECHNOLOGY DESIGN A CONVEYOR SYSTEM TO SORT PRODUCTS USING QR CODE ADVISOR: Ph.D DO DUY TAN STUDENT: NGUYEN PHAM DUY THAI 18161135 Ho Chi Minh, December, 2022 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION THESIS MAJOR: ELECTRONICS AND TELECOMMUNICATION ENGINEERING TECHNOLOGY DESIGN A CONVEYOR SYSTEM TO SORT PRODUCTS USING QR CODE ADVISOR: Ph.D DO DUY TAN STUDENT: NGUYEN PHAM DUY THAI 18161135 Ho Chi Minh, December, 2022 THE SOCIALIST REPUBLIC OF VIETNAM Independence - Freedom - Happiness -Ho Chi Minh City, July … 2022 GRADUATION PROJECT ASSIGNMENT Student name: Nguyen Pham Duy Thai Student ID: 18161035 Major: Electronics and Telecommunication Engineering Technology Class: 18161CLA Advisor: Ph.D Do Duy Tan Date of assignment: 01/10/2022 Date of submission: 24/12/2022 Thesis title: Design A Conveyor System To Sort Products Using Qr Code Initial materials provided by the advisor: Adrian-Vasile Duka (Dec 2014) “Neural network based inverse kinematics solution for trajectory tracking of a robotic arm” Presented in The 7th International Conference Interdisciplinarity in Engineering Available: www.sciencedirect.com W S Mada Sanjaya, Dyah Anggraeni, Madinatul Munawwaroh, M Yusuf S Nurasyidiek, Darmawan Setia Rahayu, Ahmad Samsudin, Ikhsan Purnama Santika, and Endah Kinarya Palupi (2018) “Colored Object Sorting using DoF Robot Arm based Artificial Neural Network (ANN) Method” Content of the thesis: Content 1: Learn QR code and set parameters Content 2: Write a program to recognize image processing using OpenCv Python Content 3: Build a database system to store data upon identification Content 4: Design GUI user interface Content 5: Hardware design of conveyor belt and robot arm Content 6: Write an Arduno program to control an automatic robot Content 7: Connect hardware and software, run model execution, evaluate results Content 8: Write an implementation report Content 9: Defend the graduation thesis CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence - Freedom - Happiness -Ho Chi Minh City, Dec 24 2022 ADVISOR’S EVALUATION SHEET Student name: Nguyen Pham Duy Thai Student ID: 18161035 Major: Electronics and Telecommunication Engineering Technology Thesis title: Design A Conveyor System To Sort Products Using Qr Code Advisor: Ph.D Do Duy Tan EVALUATION Content of the thesis: - Thesis has chapters with 51 pages - The real system is successfully completed following the objectives in the proposal Strengths: Basically finish the target within limited time Weaknesses: Content of some parts needs to be further improved e.g., chapter 4 Approval for oral defense? (Approved or denied) Approved Overall evaluation: (Excellent, Good, Fair, Poor) Fair Mark: 8.2 (in words: Eight point two) HCM City, month 12 day 24 year 2022 ADVISOR A GUARANTEE I am here with formally declare that this thesis is our research and implementation I am have not copied from any published articles without quoting the source If there is any violation, I amaccept full responsibility Student implement GRATITUDE First of all, I would like to express my sincere thanks to the teachers and lecturers at Ho Chi Minh City University of Technology and Education, especially the teachers in the electronics and telecommunications department who have imparted valuable knowledge so that my group can complete this thesis topic During the writing of the thesis, thanks to the help of Ph.D Do Duy Tan, he helped us to solve the difficulties and problems we encountered Therefore, we would like to express our sincere thanks to you, for your dedication and encouragement that you have given me In addition, my team would also like to express our deep gratitude to the teachers in the council for giving valuable suggestions so that I can summarize the experiences and lessons on my future study and career path Sincerely thank! ABSTRACT In various industries, robot manipulator systems combined with computer vision are increasingly being used Those systems usually have high accuracy, hence, they can replace humans in some specific complex applications and improve productivity effectively To understand more about these system, in this thesis, we want to research on robot manipulator that have ability to pick up moving objects on the conveyor and classify them to their right groups The main contents of this thesis consist of a program to control robot manipulator using Arduino, an image processing program for tracking and classifying moving objects, a communication program between robot manipulator and computer Besides, we implemented a graphic user interface to supervise the process of system The thesis has completed succesfully proposed requirements, despite of some specific limits Therefore, in the future, this thesis can be develop to reduce the image processing time, reduce noise, increase speed of gripping objects That will help improve productivity and get high accuracy for the system Contents CHAPTER 1: INTRODUCTION 1.1 OVERVIEW 1.2 OBJECTIVES 1.3 SCOPES 1.4 RESEARCH CONTENT 1.5 METHODOLOGY CHAPTER 2: BACKGROUND INFORMATION 2.1 Overview OF QR CODE 2.1.1 About QR CODE 2.1.2 The development history of QR codes 2.2 Structure and specifications of the QR code 2.2.1 QR code creation 2.2.2 The specifications of the QR code 2.3 Practical applications of QR codes 2.4 Background information of image processing 2.4.1 Overview of image processing 2.4.2 Open Source Computer Vision (OpenCV - Open Source Computer Vision) 2.4.3 Pretreatment methods 2.4.4 Image thresholding 10 2.5 Detect objects and scan QR codes with the camera 12 2.5.1 Detecting objects 12 2.5.2 Scan QR code 13 2.6 Robot control programming 13 2.6.1 Coordinate Axis System 13 2.7 Components Used To Design Hardware .15 2.7.1 Robot Arm 15 2.7.2 Stepper Motor 16 2.7.3 Arduino UNO R3 16 2.7.4 Cnc Shield V3 17 2.7.5 A4988 Driver 18 2.7.6 12V 5A Power voltage 19 2.7.7 12V geared moter LS220 20 2.7.8 Limit switch KW11-3Z 20 2.7.9 High trigger 5V relay module 21 2.7.10 Vacuum Pump 12VDC Pyp – 370 21 2.7.11 Mini Conveyor 22 to determine the contour of the object For the findContours algorithm to have good results, we need to limit the size of the object to avoid the noisy areas that are also returned Determine the center The center of a shape is the average of all the points in that shape For example, a figure consists of n distinct points x_1,x_2,…,x_n then the center is determined by the formula: 𝑛 𝑐 = ∑ 𝑥𝑖 𝑛 𝑖=1 In image processing and computer vision, each image is made up of pixels The center of the shape in the image plane is the weighted average of the pixels in that shape Image moments: Image moments are the weighted average of the intensity of pixels, by which we can determine some features of the image such as (center, area, radius, ) Formula for calculating moment of an image area: In the OpenCV library that can help us find the center of the shape that we have determined by the findContours algorithm is the function cv2.moments The result this function returns is a dictionary, in which the coordinates of the center of the object on the image plane are calculated according to the formula: 𝑐𝑥 = 𝑀["m10"] 𝑀["m01"] ; 𝑐𝑦 = 𝑀["m00"] 𝑀["m00"] With the coordinates of the center of the object found by the moments function, we can convert these coordinates to the coordinate system of the robot From there, send these coordinates to the microcontroller, instructing the robot to pick up the corresponding object 40 Figure 10 Example of defining the contour and center of an object 3.6.2 Create customer database using mysql workbench The database we use here is MySQL Workbenh We will create tables named info_customer and robots The info_customer table consists of columns: ID, Nguoi_Gui, Nguoi_Nhan, Noi_Gui, Noi_Nhan, Ngai_Gui, Ngan_Nhan, San_Pham and finally Phi_Code This table has the task of creating the ID of the QR code, in each ID code will include the information of the order Robot table consists of columns: ID, Ngai_Xuat, Nguoi_Gui, Noi_Gui, Ngai_Gui, Ten_SP, So_Tien This table is responsible for displaying information and time after being classified by the system Figure 11 Table Info_customer 41 Figure 12 Table Robot 3.6.3 User interface Figure 13 Manual mode control interface Connection : make connection with arduno Configuation : to configure the speed and acceleration for the stepper motor Basic control : Calib the robot angle to the original position, on off griper Joint mode : solve the forward kinematics problem World mode : solve inverse kinematics problem Current position : current position and orientation of the switch head 42 Set position : control the robot to the desired position Figure 14 Auto mode console Star cam : turn on the camera Stop cam : turn off the cam Run : run the progra Exit : exit the program 43 Figure 15 Information database interface CHAPTER 4: RESULTS AND ASSESSMENT 4.1 Result 4.1.1 Programming the control of the robot arm Understanding the characteristic parameters of the robot arm model Mk2, solving the problem of inverse kinematics to control the robot to the desired position Understand how to control a stepper motor, from which the joint can be controlled simultaneously I have measured times with a straightedge and collected the above data table, because there is no specialized measuring device, the measurement results will not be absolutely accurate 44 Figure 4.1 complete model of the system I use baskets respectively District 1, District 2, District Blue is district Red is district Green is district 1.Identify products going to District Figure Product go to district Identify products going to District 45 Figure 4.3 Product go to district 3.Identify products going to District Figure Product go to district The information of the order is displayed on the user's automatic mode after being recognized and classified 46 Figure 4 Order Information displayed Table 15 Position of end-effector desired to control and position of end-effector measured from reality STT 10 11 12 13 14 15 Position of the desired end-effector Py Px Pz (cm) (cm) (cm) 18 15 18 15 18 15 18 15 18 15 17 16 18 16 19 16 20 16 21 16 20 13 20 14 20 15 20 16 20 17 The position of the endeffector measured ' Px P'z P' y (cm) (cm) (cm) ePx ePy ePz (cm) (cm) (cm) 0.7 2.3 2.6 4.4 5.1 5.8 5.7 5.8 5.6 6.3 7.7 7.8 8.3 8.4 7.8 0.3 0.3 0.4 0.4 0.1 0.2 0.3 0.2 0.4 0.3 0.3 0.2 0.3 0.4 0.2 0.3 0.3 0.4 0.1 0.5 0.4 0.3 0.4 0.5 0.5 0.5 0.3 0.4 0.5 0.4 0.4 0.3 0.2 0.3 0.2 0.2 0.3 0.2 0.1 0.3 0.3 0.2 0.5 0.3 0.3 17.7 17.7 18.4 18.1 17.5 17.4 18.3 18.6 20.5 21.5 19.5 20.3 19.6 20.5 20.4 47 14.6 15.3 15.2 15.3 14.8 15.8 16.3 15.8 15.9 16.3 13.3 13.8 15.6 16.5 16.7 Error From the table I have: 0.3+0.3+0.4+0.4+0.1+0.2+0.3+0.2+0.4+0.3+0.3+0.2+0.3+0.4+0.2  = 0.2867(cm)  ePx = 15  0.3+0.3+0.4+0.1+0.5+0.4+0.4+0.4+0.5+0.5+0.5+0.3+0.4+0.5+0.4  = 0.3867(cm)  ePy = 15  0.4+0.3+0.2+0.3+0.2+0.2+0.3+0.2+0.1+0.3+0.3+0.2+0.5+0.3+0.3  = 0.2733(cm)  ePz = 15  4.1.2 Classification based on QR code Table 16 Classification results based on QR code using reverse kinematics method STT Time Number of things put on the conveyor Number of Correct correctly classification classified rate objects Morning 30 29 96.67% Wrong time due to not recognizing QR code Morning 30 29 96.67% Wrong time due to not recognizing QR code Morning 30 28 93.33% Wrong time due to not recognizing QR code Night 30 30 100% Night 30 30 100% Night 30 30 100% 48 Note Figure 4.5 Order information is displayed on the database After being classified, the order information will be displayed on the user interface in the data base section We can filter orders based on ID or product name 4.2 Comments and ratings 4.2.1 Algorithm to control the robot to pick up and classify objects 4.2.1.1 Robot control by traditional inverse kinematics method Due to the poor mechanical design and 3D printing quality, along with the error in transmission from the gear mounted on the stepper motor shaft to the gear mounted at the joints of the robot has led to the above error Control errors in x, y, z axes of approximately 0.3 (cm), 0.4 (cm), 0.3 (cm) are acceptable, respectively The robot's hit rate is quite high, depending on the coordinates of the object detected by the camera and the stability of the conveyor speed Realizing the control of joints of the robot simultaneously helps to increase product sorting time rather than controlling each joint 4.2.2 Object classification algorithm using computer vision 49 * Comment: Tracking objects, finding coordinates in pixels and converting them to the coordinates of the robot, calculates the rotation angle of the joints and sends it to the Arduino * Factors affecting the results: Due to rounding in the process of finding the coordinates of the object, plus the error in the calculation can lead to the cumulative error when sending the center coordinates to the robot Camera quality is not good, so when there is a significant change in the brightness of the environment, determining the position of the noisy object leads to the robot picking up the wrong position The frame rate is still slow, so the object tracking is not stable 50 CHAPTER 5: CONCLUSION 5.1 Results achieved - Build and understand the robotic system model and the equipment components required for a robotic arm model - Programming the robot arm control by controlling stepper motors with Arduino UNO R3, using driver circuits - The rate of picking up objects is almost 100% accurate, the recognition ability of the computer vision system is quite high, for QR code recognition is 98.05% - Calculate coordinates from the coordinate system on the picture frame to the coordinate system of the robot to perform accurate picking - Programming interface to connect conveyor system and camera with robot system, making operation and observation more intuitive 5.2 Futher work Learn about the G-code programming language to solve the problem of orbital planning for robots Change the current motor of the conveyor with a motor with greater torque and speed to optimize the robot's product sorting time Replace the travel switches with optical sensors to calibrate the robot's joints more accurately REFERENCE [1]L Crane, "Battle of the billionaires," ed: Elsevier, 2021 [2]T J J S J Soon, "QR code," vol 2008, pp 59-78, 2008 [3]G Bradski and A J D D s j o s t Kaehler, "OpenCV," vol 3, 2000 [4]https://www.shutterstock.com/vi/image-vector/qr-code-vector-attention-realisticdetail-89748973 [5]https://www.researchgate.net/figure/The-history-of-symbols-used-in-barcode-imagerecognition_fig15_281190312 [6] https://www.researchgate.net/figure/The-Development-of-QR-Code_fig3_267828104 based Artificial Neural Network (ANN) Method” [7] http://qrcode.meetheed.com/question7.php 51 [8] https://kinhtedothi.vn/ha-noi-day-manh-trien-khai-viec-quet-ma-qr-de-quan-ly-thongtin-nguoi-ra-vao-co-quan-dia-diem-phuc-vu-phong-chong-dich.html [9] Adrian-Vasile Duka (Dec 2014) “Neural network based inverse kinematics solution for trajectory tracking of a robotic arm” Presented in The 7th International Conference Interdisciplinarity in Engineering Available: www.sciencedirect.com [10] W S Mada Sanjaya, Dyah Anggraeni, Madinatul Munawwaroh, M Yusuf S Nurasyidiek, Darmawan Setia Rahayu, Ahmad Samsudin, Ikhsan Purnama Santika, and Endah Kinarya Palupi (2018) “Colored Object Sorting using DoF Robot Arm based Artificial Neural Network (ANN) Method” [11] Bruno Siciliano, Lorenzo Sciavicco, Luigi Villani, Giuseppe Oriolo (Aug 2008) “Robotics: Modelling, Planning and Control” [12] Documents and Tutorials of OpenCV Available: https://docs.opencv.org/4.x/d4/db1/tutorial_documentation.html [13]TS Nguyễn Đức Thành, Kỹ thuật Robot [14]PGS TS Nguyễn Trường Thịnh (2016), Kỹ thuật robot, NXB Đại Học Quốc Gia TP HCM 52 APPENDIX https://drive.google.com/drive/folders/1yTSqBGcILKwvYEkrFLER_TWuY5yr4fi?usp=share_link 53 S K L 0