Đồ án tốt nghiệp - Đề tài: Ứng dụng xử lý ảnh và cánh tay robot Nachi công nghiệp phân loại sản phẩm theo hình dạng

Đồ án tốt nghiệp đã báo cáo HK1 - 2024 tại trường Đại học sư phạm kỹ thuật Hồ Chí Minh với điểm chấm 8.2/10 ở hội đồng bảo vệ Khoa Điện - Điện tử.

MINISTRY OF EDUCATION & TRAINING

HCMC UNIVERSITY OF TECHNOLOGY AND EDUCATIONFACULTY OF INTERNATIONAL EDUCATION

AUTOMATION AND CONTROL ENGINEERING TECHNOLOGY -⸙∆⸙ -

GRADUATION THESIS

IMAGE PROCESSING APPLICATION FORPRODUCT CLASSIFICATION AND ROBOT

CONTROL

AUTOMATION AND CONTROLENGINEERING TECHNOLOGY

INDEPENDENCE - FREEDOM –HAPPINESS

HCMC, January 6th, 2024

GRADUATION PROJECT ASSIGNMENT

Student’s name: Nguyen Truong ThinhID: 18151038

Class: 18151CLA2 & 19151CLA3

Major: Automation And Control Engineering Technology

I PROJECT NAME: Image Processing Application For Product Classification And Robot Control.

Content 5: Write a program for object detecting by OpenCV.

Content 6: Link and transfer data between camera and PC to control robot.Content 7: Write a program for creating the Socket communication between robot controller and PC by opencv.

Content 8: Write a program to detect the edges of an object.

Content 9: Write a program for sending the tracked position of an object to the robot.

Content 10: Debugging.

Content 11: Performance evaluation.

III.DATE OF ASSIGNMENT: 21/8/2023.IV TASK COMPLETION DATE: 30/12/2023.

V INSTRUCTOR'S FIRST AND LAST NAME: Assoc Prof Dr Le My Ha.

CHAIR OF THE PROGRAM ADVISOR

(Sign with full name) (Sign with full name)

HCMUTE FACULTY OFINTERNATIONAL EDUCATIONAUTOMATION AND CONTROLENGINEERING TECHNOLOGY

SOCIALIST REPUBLIC OFVIETNAM

INDEPENDENCE - FREEDOM –HAPPINESS

HCMC, January 6th, 2024

ADVISOR’S EVALUATION SHEET

Name of Student 1: Ngo Trieu Vy ID: 19151091Name of Student 2: Nguyen Truong Thinh ID: 18151038Major: Control and Automation Engineering Technology.

Project title: Image Processing Application For Product Classification And Robot Control.

Advisor: Assoc Prof Dr Le My Ha.

1.Content and workload of the project:

-

5. Overall evaluation: (Excellent, Good, Fair, Poor)

HCMUTE FACULTY OFINTERNATIONAL EDUCATIONAUTOMATION AND CONTROLENGINEERING TECHNOLOGY

SOCIALIST REPUBLIC OFVIETNAM

INDEPENDENCE - FREEDOM –HAPPINESS

HCMC, January 6th, 2024

PRE-DEFENSE EVALUATION SHEET

Name of Student 1: Ngo Trieu Vy ID: 19151091Name of Student 2: Nguyen Truong Thinh ID: 18151038Major: Control and Automation Engineering Technology.

Project title: Image Processing Application For Product Classification And Robot Control.

Advisor: Assoc Prof Dr Le My Ha.

1.Content and workload of the project:

-

5.Reviewer questions for project valuation

Ho Chi Minh City, …/ …/ 2024

(Sign with full name)

HCMUTE FACULTY OFINTERNATIONAL EDUCATIONAUTOMATION AND CONTROLENGINEERING TECHNOLOGY

SOCIALIST REPUBLIC OFVIETNAM

INDEPENDENCE - FREEDOM –HAPPINESS

Project title: Image Processing Application For Product Classification And Robot Control.

Advisor: Assoc Prof Dr Le My Ha.

1.Content and workload of the project:

-

Ho Chi Minh City, …/ …/ 2024

COMMITTEE MEMBER

(Sign with full name)

Completing this project was a long journey of 4 years at university and it wasa blessing that we were finally able to complete it We would like to sincerely thankthe teachers in the Department of Mechanical Engineering, especially the teachersin the Mechatronics major who have taught me as well as my fellow students in theK18 course a student journey.

We would like to send our sincere thanks to our guide, Associate Professor,PhD Le My Ha is the teacher who enthusiastically guided and communicated to

improve me throughout the process of implementing the project Sincere thanks to

Mr Do Tan Khoa, Mr Tran Nguyen Hai The people at the SHTP Training CNC

zone training center provided equipment and necessary knowledge for me to use asa foundation to complete this project.

In particular, I would like to thank my friends and family members whosupported me during this time It is truly an extremely important and difficultmilestone The words of encouragement and support from behind have helped mestep forward and given me more hope to complete it Wishing all the current andfuture students of the K18 and K19 course many successes.

We sincerely thank you!

Ngo Trieu VyNguyen Truong Thinh

GRADUATION PROJECT ASSIGNMENT i

ADVISOR’S EVALUATION SHEET iii

PRE-DEFENSE EVALUATION SHEET v

DEFENSE COMMITTEE MEMBER EVALUATION SHEET vii

LIST OF FIGURES xiii

LIST OF TABLES xv

CHAPTER 1: PROJECT OVERVIEW 16

1.1 Reason for this project: 16

1.2 Application of robot hand with conveyor system 16

1.2.1 Introduction to conveyor system 16

1.2.2 Application of robot arm with conveyor system 17

1.3 Example about a Robotic Automatic Assembly System Based on Vision [3]191.4 Objective, task and scope of the thesis 22

2.2.4 Step 4 32

2.3 Tracking Algorithm with Conveyor Belt 32

2.3.1 Harris corner algorithm [4] 32

2.3.2 Hough circle algorithm [5] 34

2.3.3 KLT algorithm (Kanade Lucas Tomasi feature tracker) 34

2.4 Determining Object Center Coordinate 35

2.4.1 The center of the square object 35

2.4.2 The center of the circular object 36

CHAPTER 3: SYSTEM DESIGN 37

3.3.2 Time-of-Flight camera (ToF camera) 44

3.3.3 Structure light camera 45

3.3.4 LIDAR 46

3.3.5 Phone camera 48

3.4 PLC and CFD controller 48

3.5 Overview of Robot Nachi MZ07 50

3.5.1 Forward kinetic analysis 53

3.5.2 Inverse kinematic analysis 56

3.5.3 Adjust the parameters of the initial tool 59

3.5.4 Robot motion options 60

3.6 Design of gripper end-effector 61

3.6.1 Calculation of gripping force 61

3.6.2 Fixture for the gripping tool head 64

3.7 Robot Nachi communication 66

3.7.1 Introduction to TCP/IP Communication Method with Nachi Robot 66

3.7.2 Introduction to Robot Communication and Control Program 68

CHAPTER 4: EXPERIMENTAL RESULT 70

4.1.4 Experimental Results and Evaluation 74

4.2 Link to experimental video 75

CHAPTER 5: CONCLUSION 76

5.1 Conclusion 76

5.2 The limitations of the thesis 77

5.3 The directions for future research 77

LIST OF FIGURE

Figure 1.1 Conveyor belts used in assembly lines 17

Figure 1.2 Manipulator in computer case assembly line 18

Figure 1.3 Robot sorts trash for recycling 18

Figure 1.4 Phone components 19

Figure 1.5 Assembly process of the article 20

Figure 1.6 Process and steps of the assembly system 20

Figure 1.7 The calibration method used in the paper's system 21

Figure 1.8 The image processing method used in the article 21

Figure 1.9 Matching algorithm and assembly step of the article 22

Figure 1.10 Parameter table evaluates the results of the article 22

Figure 2.1 Main Flowchart 24

Figure 2.2 Sub-Flowchart 1 25

Figure 2.3 Sub_Flowchart 2 26

Figure 2.4 Sub_Flowchart 3 27

Figure 2.5 Camera in fixed position 28

Figure 2.6 Camera attached to the robot arm 29

Figure 2.7 Fixed position for the camera 30

Figure 2.8 The result after training yields coordinates (X, Y, Z) in real space 31

Figure 2.9 Processing input images using matlab 31

Figure 3.1 Delta Robot FANUC M-2ia 37

Figure 3.2 Scara Robot 38

Figure 3.3 FANUC Robot Serial-Link 39

Figure 3.4 Pneumatic Gripper 41

Figure 3.5 Vacuum Gripper 42

Figure 3.6 Magnetic gripper 42

Figure 3.7 On field vaccuum gripper 43

Figure 3.8 3D Stereo Camera Multi Sense S7 44

Figure 3.9 Microsoft KinectV2 45

Figure 3.10 Microsoft KinectV1 46

Figure 3.12 Iphone 8 Plus 48

Figure 3.13 Mitsubishi PLC 49

Figure 3.14 CFD controller 49

Figure 3.15 Nachi Robot MZ07 50

Figure 3.16 MZ07 working space 52

Figure 3.17 Coordinate system of manipulator axes 53

Figure 3.18 Parameters setting for tool head in Pendant 60

Figure 3.20 The symbol for a vacuum valve 62

Figure 3.21 Forces affecting the object 62

Figure 3.22 Suction cup DP-S25 63

Figure 3.23 The vacuum suction cup valve ZU from the SMC brand 64

Figure 3.24 Pneumatic suction head Đài Loan 64

Figure 3.25 Flange surface for attaching the tool on Nachi MZ07 robot 65

Figure 3.26 Hitachi Bebicon 65

Figure 3.27 TCP/IP Signal Transmission and Reception Process 68

Figure 3.28 User Task program 69

Figure 4.1 Overview of the system 70

Figure 4.2 Connection diagram 71

Figure 4.3 Analyze the system's input and output 71

Figure 4.4 Tracking square object 72

Figure 4.5 Tracking circular object 73

LIST OF TABLES

Table 3.1 Operating range of MELFA RV-4FR-R 51

Table 3.2 Denavit-Hartenberg table 53

Table 3.3 Robot dimension (mm) 54

Table 3.4 Type of interpolation 61

Table 4.1 Discrepancy table between calibrated image coordinate distances and real-world spatial distances between two image points 74

Table 4.2 The absolute error of the distance between the object's center coordinates and the actual position 75

CHAPTER 1: PROJECT OVERVIEW

1.1 Reason for this project:

Currently, with modernized industry, with the development of moderntechniques, industry 4.0 needs to produce a lot of components or machinery, or inthe commodity production industry, it is also necessary to modernization.Classifying is an extremely essential step, to save time, effort and money,manufacturing manipulators to serve this need in industry has been widelydeveloped widely in our country.

In traditional manufacturing assembly lines, conveyors are widely used tomove components from one workstation to another The conveyor must stop at eachworkstation for a specific amount of time and wait until the assembly operation iscomplete before the conveyor can move the components to another workstation.Conveyor downtime accumulates and lengthens the entire assembly process Thiscauses a decrease in overall production efficiency, especially in automatedassembly lines To eliminate such drawback, the concept of conveyor tracking wasdeveloped to ensure consistent assembly line workflow by performing the assemblyprocess while the conveyor belt is still running.[ CITATION Che97 \l 1033 ]

In a conveyor tracking system, the conveyor not only serves the purpose ofmaterial handling but also works in conjunction with the automation process.Furthermore, the Conveyor robot tracking system is also capable of providinginformation about the continuously changing position of components on theconveyor belt to the robot controller through the use of vision system[ CITATIONPap93 \l 1033 ] This thesis presents the use of the Nachi MZ07 robot of the Serial

Conveyor systems are an important technology today and are widely used inmany different industries In general, conveyor systems have become anindispensable part of many modern industries, helping to optimize the moving andtransportation process, increase efficiency and minimize labor costs Technologiesand designs are constantly being improved to meet the increasing demands in themodern world of manufacturing and logistics.

Figure 1.1 Conveyor belts used in assembly lines

1.2.2 Application of robot arm with conveyor system

Manipulators (industrial robots) combined with conveyor systems are apowerful combination in industry and have many useful applications Below aresome examples of applications of manipulators with conveyor systems:

• Automatic assembly system• Product handling and packaging• Mass production

• Food industry• Quality check

• Textile and garment industry

Figure 1.2 Manipulator in computer case assembly line

Figure 1.3 Robot sorts trash for recycling

Overall, the combination of manipulators and conveyor systems brings many

1.3 Example about a Robotic Automatic Assembly System Based on Vision[ CITATION Rui20 \l 1033 ]

The scientific article below presents the capabilities of robots in automaticphone assembly lines, the precise assembly capabilities of robots and theimportance of maximizing the application of robots to optimize cost savings feesinstead of hiring labor The article presents the process of assembling phone

Figure 1.4 Phone components

components and methods from tracking on conveyor to none tracking on conveyor,thereby providing comparisons and evaluations to choose the optimal method pros.There is also mention of techniques such as Camera Calibration and Robot-Conveyor calibration system or Camera-Robot-Conveyor calibration system.

In addition, the phone has good image input quality, making it suitable for current image processing applications.

Figure 1.4 Phone components

Figure 1.5 Assembly process of the article

In the article, it is shown that the assembly of mobile phones mainly includesthree parts: main board, screen and back cover The assembly system is mainlycomposed of a robot controller, the motion module is the conveyor, the visionmodule is the camera, and the assembly module is the manipulator, as illustratedabove.

Figure 1.6 Process and steps of the assembly system

The above procedure describes camera calibration, component position detection, camera dynamics (dynamic capture), and template matching algorithm.

The author implemented the fixed camera calibration method by ZhangCalibration, the 4-point calibration method for the conveyor-robot system, the eye-

in-hand camera calibration method to convert the camera coordinate system to themanipulator coordinate system Then use a 2D image processing algorithm toextract component images, contours, rotation angles and component coordinates.Combined with processing and reading the conveyor encoder signal to transmitsignals to the robot to perform the picking task and finally assemble it into anexternal fixed tray.

Figure 1.7 The calibration method used in the paper's system

Figure 1.8 The image processing method used in the article

Figure 1.9 Matching algorithm and assembly step of the article.

Finally, experimental results show that the maximum error of positioning isbetween 0.5mm and 0.8 degrees The robot can successfully grasp the part with a100% success rate at different conveyor speeds, even when incomplete informationabout the target part is provided The key technologies developed in this article willbe applied to key equipment in the automatic robot production line This willimprove the standard performance of products, greatly increase the marketcompetition of target products, and create a large market space.

Figure 1.10 Parameter table evaluates the results of the article

1.4 Objective, task and scope of the thesis1.4.1 Objectives

 Assemble conveyor belts to move objects. Design the camera position to track objects.

 Program and control the position of the robot to pick up objects with highand optimal accuracy.

 Design an image processing program using the OpenCV library to improvethe object detection algorithm according to the object's velocity and positioneven when that velocity is unstable.

1.4.3 Scope of topic implementation

The topic applies the knowledge of robotics, electromechanics andprogramming that has been learned at school, while combining it with references toresearch and documents The topic uses phone cameras and hand robots Themachine and conveyor perform a demo run to check the algorithm results Providecomments on the errors of the algorithm and structure and finally propose solutionsthat can be used in the future.

Furthermore, the scope of the topic revolves around object classificationbased on shape, with additional improvements focused on enhancing speed duringexperimentation.

The topic is delimited by the number of object types, specifically focusing onthe shapes of circles and squares.

The robotic arm is constrained to work based on each object moving on theconveyor belt The conveyor belt operates at a fixed speed.

CHAPTER 2: THEORIES

2.1 Algorithm Flowchart2.1.1 Main Flowchart

Figure 2.11 Main Flowchart.

 We will start with each of the following steps:

2.1.1.1 Step 1

We initialize a socket connection for the computer with the robot controller.And then, we initialize variables to define that connection, including the program for robot movement and activating I/O.

Details about the definition and initialization are available in the appendix section.

2.1.1.2 Step 2

We proceed to perform the image processing.

Figure 2.13 Sub_Flowchart 2.

3 Tracking circular and square objects.

a) Camera in fixed position:

Figure 2.15 Camera in fixed position

b) Attached to the robot arm:

Figure 2.16 Camera attached to the robot arm

 Observe the object more clearly.

 No need to recalibrate when moving the robot system.

 Slow performance While the camera on the robotic arm can adjust itsviewing angle, it may encounter difficulties in providing acomprehensive view compared to a stationary camera This could be aparticular issue when the need arises to observe a wide area.

 Not suitable for high-performance applications The tracking

capability of moving objects may be subpar in certain cases, and this can be attributed to factors such as the weight and size of the robotic arm.

 Conclusion:

The types of cameras used in the project vary in size and weight, making theinstallation relatively complex This complexity would pose difficulties andobstacles in moving and adjusting the angles of the camera Therefore, the decision

Figure 2.17 Fixed position for the camera

2.1.1 Step 1

Using Matlab to process 15 input images, from which the transformation matrix KK is determined:

515.6772 0 331.980 511.4329 236.991

Figure 2.18 The result after training yields coordinates (X, Y, Z) in real space.

2.1.2 Step 2

After obtaining the transformation matrix KK, we determine the camera's intrinsic and extrinsic parameters.

1( 1 p ) w

R K Zm tM

(2.2)After using Matlab, we get the value of the transfomation matrix Mw:

0.06365* 1.43391* 398.04720.097366 * 1.44537 * 271.3566

2.3 Tracking Algorithm with Conveyor Belt

2.3.1 Harris corner algorithm[ CITATION Alg \l 1033 ]

The algorithm informs us that corners are regions in the image with significantintensity variations for the displacement of (u, v) in all directions This is illustratedas follows:

We have to maximize this function E(u,v) for corner detection That means wehave to maximize the second term Applying Taylor Expansion to the aboveequation and using some mathematical steps (please refer to any standard text booksyou like for full derivation), we get the final equation as:

( , ) [ ]u

E u vuv Mv

(These can be easily found using cv.Sobel()).

Then comes the main part After this, they created a score, basically an equation, which determines if a window can contain a corner or not.

are the eigenvalues of M

So the magnitudes of these eigenvalues decide whether a region is a corner, anedge, or flat.

 When |R| is small, which happens when 1 and 2 are small, the region

In the line detection case, a line was defined by two parameters (r,θ) In the) In thecircle case, we need three parameters to define a circle:

: ( center, center, )

(2.6)Where (xcenter,ycenter)

define the center position (green point) and r is theradius, which allows us to completely define a circle.

For efficiency, OpenCV implements a detection method slightly trickier thanthe standard Hough Transform: The Hough gradient method, which is made up oftwo main stages The first stage involves edge detection and finding the possiblecircle centers and the second stage finds the best radius for each candidate center.For more details, please check the book Learning OpenCV or your favoriteComputer Vision bibliography.

2.3.3 KLT algorithm (Kanade Lucas Tomasi feature tracker)

The Kanade-Lucas-Tomasi (KLT) tracking algorithm is based on identifyingand tracking the motion of feature points in consecutive frames Below is a moredetailed description with the core algorithmic formulas:

Feature Point Identification:

Choose a set of feature points in the initial frame.

Compute Gradient:

Use a gradient method (usually Sobel) to compute partial derivatives with

respect to x and y in the frame, generating gradient matrices Ixand Iy

Compute Derivative Matrices

Compute matrices of partial derivatives of feature points over time Let It be

the derivative with respect to time.

These matrices are synthesized into a Jacobian matrix J, where:

( ) ( )( ) ( )

Solve System of Equations:

The goal is to solve the linear system of equations: J P. It, where _P is a vector containing changes in the positions of feature points.

This equation can be solved using methods like the Least Squares method.

Here, k is the index of the current frame.

2.4 Determining Object Center Coordinate2.4.1 The center of the square object

_ _ _ / / 2_ _ _ / / 2

center xmin xmax xcenter ymin ymax y

Therefore, the center of the square object has coordinates at (xmax,ymax).

2.4.2 The center of the circular object

Thanks to the Hough circle algorithm, we can find the coordinates of thecenter of the circular object by the openCV library.

In addition, the library also assists us in obtaining the radius of that circle.The Hough Circle Detection algorithm in OpenCV helps determine thecenter and radius of circles in an image

You need to convert the image to grayscale, optionally blur it, and then usethe cv2.HoughCircles function to detect circles.

The result is an array containing parameters for the detected circles You candraw circles and their centers on the image to display the results

Adjust parameters such as minDist, param1, param2, minRadius, andmaxRadius to optimize the outcome.

CHAPTER 3:SYSTEM DESIGN

3.1 Robot classification3.1.1 Delta robot

Delta robots are famous for their high speed and acceleration Their designmakes them faster than other types of industrial robots Most industrial robots havemotors housed in the robot arms, making them suitable for heavy-duty applications.However, placing the motor in the robot arm adds weight and prevents fast speeds.With the Delta robot, all the motors are located in the main body of the robot abovethe work area This keeps the main weight of the robot fixed All movements comefrom the extremely light robot arm, allowing for low inertial weight, achievingextremely fast operating speeds and accelerations.[CITATION htt \l 1033 ]

Figure 3.20 Delta Robot FANUC M-2ia

Advantage:

 It has the highest processing speed among robot mechanisms.

Applications:

 Delta robots are ideal for assembly, distribution, pick and place, materialhandling, part converting and packaging applications Parts processed byDelta robots should be light and simple in shape, as these robots operate atfast speeds Heavy or complex parts are not suitable for the speed andacceleration of the Delta robot.

3.1.2 Scara Robot

The SCARA robot is a manipulator with four degrees of freedom This typeof robot has been developed to improve speed and application in repetitive taskssuch as Pick and Place applications picking objects from one location to another.[ CITATION SCA \l 1033 ]

Figure 3.21 Scara Robot

 In general, SCARA robots can operate at high speeds and are commonlyused for pick-and-place or assembly operations where high speed andaccuracy are required Existing SCARA robots can achieve accuracy below10 microns (0.01mm) This precision is comparable to 20 microns of a six-axis robot.

 The compact layout of the SCARA robot also makes it easy to move for