Nghiên cứu điều khiển rô bốt tay máy di động bám mục tiêu trên cơ sở sử dụng thông tin hình ảnh tt tiếng anh

MINISTRY OF EDUCATION AND TRAINING VIETNAM ACADEMY OF SCIENCE AND TECHNOLOGY GRADUATE UNIVERSITY OF SCIENCE AND TECHNOLOGY LE VAN CHUNG A RESEARCH INTO THE CONTROL OF MOBILE ROBOT MANIPULATOR TO TRACK TARGET BASED ON VISUAL INFORMATION MAJOR: Control engineering and automation CODE: 9.52.02.16 SUMMARY OF DOCTORAL THESIS IN TECHNICAL Ha Noi - 2019 The thesis was completed in: Graduate University of Science and Technology (GUST) - Vietnam Academy of Science and Technology Supervisors: Assoc Prof Pham Thuong Cat Dr Pham Minh Tuan Referee 1: Referee 2: Referee 3: The thesis will be examined by Examination Board of Vietnam Academy of Science and Technology, at: The thesis can be found at: [1] [2] [3] [4] [5] [6] [7] LIST OF WORKS RELATED TO THE THESIS HAS BEEN PUBLISHED Le Van Chung (2018), “Robust Visual Tracking Control of Pan-tilt Stereo Camera System”, International Journal of Imaging and Robotics, Vol 18 (1), pp 45 – 61 Le Van Chung, Pham Thuong Cat (2015), “A new control method for stereo visual servoing system with pan tilt platform”, Journal of Computer Science and Cybernetics,Vol 31 (2), pp 107 – 122 Le Van Chung, Pham Thuong Cat (2016), “Optimal tracking a moving target for integrated mobile robot – pan tilt – stereo camera”, Advanced Intelligent Mechatronics AIM IEEE Conference, Banff, Canada July 12-15, pp 530 – 535 Le Van Chung, Duong Chinh Cuong (2016), “Design Adaptive-CTC Controller for Tracking Target used Mobile Robot-pan tilt-stereo camera system”, International Conference on Advances in Information and Communication Technology, ICTA, Thai Nguyen, Dec 12-14, pp 217 – 227 Le Van Chung, Pham Thuong Cat (2014), “Robust visual tracking control of pan tilt – stereo camera system g i pp.167-173 hu g h h g t i u h 013), “Phương pháp điều khiển hệ servo thị giác stereo sử dụng bệ Pan-Tilt g - 382 hu g “Phát triển hệ pan/tilt – nhiều camera bám mục tiêu di động”, Thai Nguyen University Journal of Science and Technology, 116(02), tr 41-46 INTRODUCTION In recent years, there has been a great deal of research on robotics control using visual information But the achieved results still reveal some limitations For example, using a camera on a mobile robot only allows full tracking of the target's information when knowing the target's moving plane As with the use of two cameras on the pan tilt system, but not considering the deterioration of the Jacobian matrix affecting the grip capacity of the system Besides, the mathematical model of a robot is often difficult to achieve the required accuracy because there are many unspecified parameters in the system such as measurement of parameters or coefficient of friction, inertia mo e t etc…, usually changes during operation In addition, it is difficult to optimize the parameters in the robot controllers to achieve the desired accuracy With the above reasons, the author has chosen the topic: “A research into the control of mobile robot manipulator to track target based on visual information to develop some control algorithms that use image information with many uncertain parameters The objective of the thesis The main object of the research is to focus on pan-tilt robots and mobile robots with wheels Scope of research Researching methods of controlling pan-tilt tracking moving target using image information from cameras with many uncertain parameters Developing algorithms for controlling integrated system include mobile robot, pan-tilt and two cameras tracking moving targets New findings of the doctoral dissertation 1The application of artificial neural networks in compensate for uncertainties in a pan-tilt system model with two cameras Based on that, the kinematic and dynamic controllers are constructed for the pan-tilt two cameras system to track moving targets with uncertain parameters 2The thesis has developed a dynamic model for the integrated system that including mobile robot, pan-tilt and stereo cameras Also, the thesis has built two control methods, sliding mode controller and quadratic perfofmance optimal controller for the above integrated system The layout of the thesis Chapter 1: Overview Chapter 2: Developing controller for pan-tilt stereo camera system to track the moving target Chapter 3: Some improvements in controlling servo system to track the target Chapter 4: Developing control method for mobile robot CHAPTER OVERVIEW 1.1 Problem? In order to control a robot system using two cameras to work better, the problem is: The first: Developing methods to control pan-tilt systems using image information to track moving targets when there are uncertain parameters The second: Build a Jacobian matrix image is a square matrix for the system to track the moving target for the better system The third: Developing some methods to control the integrated system combining a mobile robot with a pan-tilt robot that carries two cameras to track the moving target and be able to move closer to the target in the space 1.2 Overview of controlling robots using visual information When using a pan-tilt robot with two cameras, another problem poses the degradation of the Jacobi matrix when taking inverse pseudocode When using a camera, the Jacobian matrix of the pan-tilt system - a camera is square and invertible But when using cameras, the Jacobian matrix of the system will be (3x6), the Jacobian matrix of the system will be (3x2) So in transformations we have to take the inverse pseudo cause that is the cause of singularities Control robots using visual information Classical method Kinematic Dynam -ic Modern method Combined with neural network Classical Modern methods methods with with neuron neuron network network Optimal control, adaptable Sliding mode Optimal paramet Optimal ers output Hình 1.2 Some methods of controlling robots 1.3 The research issues of the thesis - Building kinematic/dynamic model of pan-tilt robot and developing classical control method combining with Neural Network to get better results including: - Research and improve to build a square image Jacobian matrix - Developing advanced control methods for robots when having uncertain parameters - Building dynamic models for the integrated system that including mobile robots, pan-tilt with cameras and controllers for the above system - Using Lyapunov stability method and Barbalat's lemma proves stability and Matlab to verify results Chapter DEVELOPING CONTROLLER FOR PAN TILT STEREO CAMERA TO TRACK MOVING TARGET In this chapter, the kinematic control algorithm combined with the neural network is built to control the rotation angle of the pan - tilt robot so that the target image is always maintained at the desired position on the model of the image frames The content of the chapter consists of main parts: building a kinematic model of the pan-tilt stereo camera system, designing control algorithm, verifying and comparing with the controller not associated with Neural network and conclusions 2.1 Kinematic model of stereo visual servoing system with uncertain parameters 2.1.1 Determination of Image Jacobian matrix: Fig Pan-tilt PTU-D48E-Series & camera coordinates The velocity of t rget’s i ge o c er s: (2.9) m = Jimag (m) v - is the image Jacobi matrix: v is the velocity vector of the camera system J imag (m) U L (U L U R ) U LVL f U L2 U LU R fB f 2f UR UL B VL (U L U R ) fB f VL2 f VL (U L U R ) 2f U R (U L U R ) U RVL U U f UR L R fB f 2f UR UL B J imag (m) UR UL B VL UL UR 2f (2.10) VL 2.1.2 Kinematic equations of Pan-Tilt platform: Denote J robot the Jacobian of the Pan-Tilt platform, we have: x = J robot (q)q (2.11) Fig 2 Camera system model 2.1.3 Formulation of stereo visual servoing problem with uncertain parameters: We calculate the image feature error: ε m md The kinematic control problem of stereo visual servoing is to find control law q = K (ε) to control the system track the moving target so the tracking error ε converges to zero m = Jimag (m)u ; v = x = J robot (q)q ; q = K(ε) 2.2 Control law design In this chapter, a neural control method is proposed for Pan Tilt - stereo camera system to track a moving object when there are many uncertainties in the parameters of both camera and Pan-Tilt platform ˆ +f -m (2.21) ε = m - md = Jq -Kε + f + u1, d The structure of the chosen artificial neural network is of RBF type as shown in Fig.6 It has three layers: the input layer includes the three components of error ε , the output layer includes linear neurons and hidden layer contains neurons with Gaussian output function: j exp j cj j ; j = 1,2,3 (2.22) Fig Structure of proposed visual tracking system with many uncertain parameters Control Method 1: The stereo camera system described by the model in Esq (2.9), (2.11), with uncertain parameters controlled by the neural network defined by Eqs (2.22), (2.23) will track moving targets with the error ε, ε  if the speed of the Pan-Tilt joints is determined by the Eqs (2.25), (2.26), (2.27) and learning rules (2.28): (2.25) q Jˆ + [(md - Kε) + u1, ] Jˆ + (md - Kε) + Jˆ +u1, = u0 + u1 u0 = Jˆ + md -Kε (2.26) 1)Wσ - u1, = ( ε ε (2.27) u1 = Jˆ + u1, εσ T (2.28) where K is a positive definite symmetric matrix K = K > , the coefficients   1,   Proof: Chose a positive definite candidate Lyapunov function as follows: W T V V T ε ε εT Kε w iT w i (2.29) i ε (2.37) 2.3 SIMULATION Simulation 1: Fixed target Camera center at the initial time: m(0) = [-40, 30, 0] (pixel); Image coordinates of the target stand still at mt= [-20, 0, 20] (pixel); Fig Image feature error when using neural control u1, Fig Image feature error when no neural control is used u1, = Simulation 2: Moving target in a straight line Moving target from point A(0m,1.8m,0m) to B(-0.3m, 1.8m, 0.5m) on the plane ZCOCXC Fig Tracking error coordinates when the target moves along a straight line Fig 10 Image feature error coordinates when the target moves along a straight line 10 Lyapunov stability theory The simulation results with the uncertainties up to 20% in the case of fixed target, moving target in a straight line or circular arc, show that the tracking error converges to zero These results are consistent with the theoretical CHAPTER SOME IMPROVEMENTS IN CONTROLLING SERVO SYSTEM TO TRACK TARGET As in chapter 2, the author has noticed that in order to control the pan- tilt system with two cameras to track target work well, the control problem has some issues to solve as follows: - Firstly, it is necessary to build a square image Jacobian matrix so that performing the inverse and avoid singular points leading to losing grip - The second is building dynamic controllers in combination with neural networks to compensate for the effects of uncertainty parameters inside the model as well as external noise - The third is to optimize some parameters in the neural network to get better outputs In Part of this chapter, the author built a 3D model for two camera system to obtain the full Jacobian matrix In part 2, the author built the dynamic controller using neural network with optimized parameters, the stability of the system is demonstrated by Lyapunov method and Barballat lemma Part is the simulation results Finally, some conclusions 3.1 3D visual model for eye-in-hand stereo camera system 3.1.2 3D virtual stereo camera model systems A 3D visual space is built according to the following steps: First step, from geometrical relations between the target and that feature images we calculated the coordinates of the target point T xc  x y z in the camera coordinate frame Oc X cYc Zc Second step, a reference coordinate frame Ov X v Yv Zv with the origin located at the same position as Oc X cYc Zc is defined In order to transform OC to OV, the rotation matrix C Rv (Fig 3.2) is used The 11 Fig 3.2 3D visual stereo camera model T projection of x c  x y z in Oc X cYc Zc is defined in Ov X v Yv Zv T as x v  zv xv y v  Third step, the reference coordinate frame Ov X v Yv Zv is used to define two virtual camera's frame Ov1 X v1Yv1Zv1 , Ov X v 2Yv Zv associated with stereo cameras Their location on Xv and Zv axes are far away from Ov the distances λ Last step, the virtual camera model is combined with 3D visual camera model to construct a 3D virtual Cartesian space having feature point vector denoted as: x s  zv1 zv xv1 T Jvimg is the visual Jacobian matrix: x s  fv Jvimg x v (3.9) Jvimg     xv   xv      ( zv   )  yv  ( z v  )  3.1.3 Avoid singularity zv (  x v ) zv          zv    (3.10) 12 The singularity of Jvimg that can be avoided by choosing the r eter λ such s λ > x xv, zv) 3.1.4 Stereo visual servoing problem with uncertain parameters   fv Jvimg v R C x (3.19) x s  (Jˆ  J)q 3.2 Dynamics of robot manipulator with uncertainties The dynamics of a serial n-link rigid robot with friction, and uncertainty can be written as follows:   C(q, q  )q   g(q)  d(t )  τ (3.20) M(q)q 3.3 Robust neural control of stereo camera system with pan tilt robot 3.3.1 Construction of robust controller The control law is chosen as follows: τ  A(K Dε  K Pε)  b  τ NR  τ  τ1 , (3.30) τ  A(K D ε  K P ε)  b , (3.31) 3.3.2 Layer construction of RBF neural network The structure of choosing artificial neural network is a Radial Basis Function (RBFNN) network It has layers Input layer is vecto s  s1 s2 s3  Hidden layer computation The hidden layer consists of neurons with output function calculated by Gaussian form Output layer The output values of the network are approximate function f1 Control Method 2: The image error dynamics (3.32b) of the uncertain pan-tilt – Stereo camera tracking system (3.19), (3.20) will be asymptotically stable with the error ε  if the control torque is chosen by following (3.36), (3.37) and online learning rules (3.38): τ  A(K D ε  K P ε)  b  τ NR (3.36) T  ε  τ NR  A   1 Wσ    ε   W  sσT , (3.37) (3.38) 13 Fig 3.6 Structure of proposed visual tracking system Proof: We choose the candidate Lyapunov function as follows: V ( s, W )  T s s 2 w T i wi (3.39) i 1 Taking the derivative of V along time, yields: V  sT Gs   s    s  s V    2sT G  βT    Wσ    β  Gs     s  s    (3.49) (3.50) s We found that V is bounded because s, β are bounded, is the s unit vector, G is the positive-definite constant matrix and ,   Thus V is uniformly continuous According to Barbalat's lemma, we have s  when t  and it forces ε  and the system is asymptotically stable Simulation results Simulation 1: Moving target in a straight line from point A (0m,3m,0m) to B (-0.5m, 3m, -0.3m) 14 Fig 3.7 Tracking error coordinates when moving target in a straight line Simulation 2: Moving target in a circle O (0, 0, 0), r = Fig 3.9 Tracking error coordinates when moving target follows a circle 15 a) b) c) Fig 3.11.a) Tracking error coordinates in X, Z axes b) Torques of pan and tilt joints c) Joints angle q Simulation 3: Moving target in a rectangle with changing velocity Moving target follows the rectangle as figure from point (-1m, 3m, 0m) to (-1m, 3m, 1m) in 10 seconds In the next 10 seconds, target moves from point (-1m, 3m, 1m) to (1m, 3m, 1m) and the same to return to the starting point Fig 3.12 Tracking error coordinates when moving target is in a rectangle with changing velocity 16 a) b) c) Fig 3.13 a) Tracking error coordinates in X, Z axes b) Torques of pan and tilt joints c) Joints angle q Simulation 4: Moving target in space with random velocity and direction The start of target at point (3, 3, 0.5) in O0 coordinates; yt = 0.1t The movement of the target following x, z axes is plane-parallel motion: ≤ t