Development of machine vision systems on object classification and measurement for robot manipulation

國立高雄科技大學 機械工程系 博士班 博士論文 操縱機器人所需物件分類和量測機械視覺系統之開發 Development of machine vision systems on object classification and measurement for robot manipulation Graduate student: Ngo Ngoc Vu 研究生：吳玉武 指導教授：許光城 教授 Advisor: Prof Quang-Cherng Hsu 中華民國 108 年 月 i 操縱機器人所需物件分類和量測機械視覺系統之開發 Development of machine vision systems on object classification and measurement for robot manipulation Graduate student: Ngo Ngoc Vu 研究生：吳玉武 Advisor: Prof Quang-Cherng Hsu 指導教授：許光城 教授 國立高雄科技大學 機械工程系 博士論文 A Dissertation Submitted to Department of Mechanical Engineering National Kaohsiung University of Science and Technology in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Mechanical Engineering January, 2019 Kaohsiung, Taiwan, Republic of China 中華民國 108 年 月 ii iii 操縱機器人所需物件分類和量測機械視覺系統之開發 研究生：吳玉武 指導教授：許光城 教授 國立高雄科技大學 機械工程系 博士班 中文摘要 本研究介紹了機器視覺系統在物體分類和機器人操縱測量方面的研究。首先， 在不同的照明條件下開發了一套可用於金屬零件的自動分類和量測的機械視覺系 統，並將其應用於具 個自由度（DOF）的機器手臂。為了獲得準確的定位資訊， 整個圖像藉由工作平台上方的 CMOS 攝影機來進行取像。本研究中亦探討不同照明 條件下對本系統的影響，在正向打光的條件中有四種不同的打光狀態，且在每種條 件，皆使用全域和局部閾值來獲得較佳的圖像品質。研究中主要利用張氏方法、線 性轉換函式與二階轉換函式來取得影像座標與世界座標的關係。 實驗結果顯示，在背向打光相較於前向打光其所獲得的物體中心更為準確，而 在校正結果的部分，二階轉換函式比其他校正方法更準確。透過使用二階轉換函式 的校正偏差，在 X 和 Y 軸其最大正偏差分別為 0.48 mm 和 0.38 mm 而最大負偏差分 別為-0.34 mm 和-0.43 mm。 本研究亦開發了一套機械視覺系統於六自由度(DOF)的機械手臂用以進行物件 顏色的分類和座標的量測。整個圖像藉由工作平台上方的兩個相機以藉此獲得準確 的定位資訊，並在二維(2-D)和三維(3-D)的校正過程中採用二階轉換函式與透視投影 法來建立影像坐標和世界坐標間的關係。在二維校正的部分，於 X 和 Y 軸上的最大 正偏差分別為 1.29 mm 和 1.12 mm；而最大負偏差分別為-1.48 mm 和-0.97 mm。在 三維校正的部分，X，Y 和 Z 方向的偏差分別為 0.07、-0.418 和-0.063 mm。本研究 所提出的視覺識別系統可以對物件進行分類並取得該物件之三維坐標。 關鍵字：機械視覺、機械手、相機校正、影像分析、物件分類、光源 i Development of machine vision systems on object classification and measurement for robot manipulation Graduate student: Ngo Ngoc Vu Advisor: Prof Quang-Cherng Hsu Department of Mechanical Engineering National Kaohsiung University of Science and Technology ABSTRACT This research presents development of machine vision systems on object classification and measurement for robot manipulation Firstly, a machine vision system for the automatic metal part classification and measurement process is developed under different lighting conditions, and has been applied to the operation of a robot arm with degrees of freedom (DOF) In order to obtain accurate positioning information, the overall image is captured by a CMOS camera which is mounted above the working platform The effects of back-lighting and frontlighting conditions to the proposed system were investigated With the frontlighting condition, four different conditions were performed For each condition, global and local threshold operations were used to obtain good image quality The relationship between the image coordinates and the world coordinates was determined through Zhang’s method, the linear transformation and the quadratic transformation during the calibration process Experimental results show that in a back-lighting environment, the image quality is improved, such that the positions of the centers of objects are more accurate than in a front-lighting environment According to the calibration results, the quadratic transformation is more accurate ii than other methods By calculating the calibration deviation using the quadratic transformation, the maximum positive deviation is 0.48 mm and 0.38 mm in the X and Y directions, respectively The maximum negative deviation is -0.34 mm and -0.43 mm in X and Y directions, respectively The proposed system is effective, robust, and can be valuable to industry The second, a machine vision system for color object classification and measurement process for robot arm with six degree of freedom (DOF) is developed In order to obtain accurate positioning information, the overall image is captured by a double camera C615 and a camera C525 which are mounted above the working platform The relationship between the image coordinate and the world coordinate is performed through calibration procedure The quadratic transformation and generalized perspective transformation algorithms were used to transform coordinates in 2-D and 3-D calibration process, respectively According to calibration results, with 2-D calibration, the positive maximum deviation is 1.29 mm and 1.12 mm in X and Y directions, respectively The negative maximum deviation is -1.48 mm and -0.97 mm in X and Y directions, respectively With 3-D calibration, the deviation is 0.07 mm, -0.418 mm, and -0.063 mm in X, Y and Z directions, respectively The proposed system can catch the three dimensional coordinates of the object and perform classification and assembly automatic operations by the data from visual recognition system Keywords: machine vision, robot arm, camera calibration, image analysis, object recognition, lighting source iii ACKNOWLEDGMENTS The fulfillment of over three years of study at National Kaohsiung University of Science and Technology (NKUST) has brought me into closer relations with many enthusiastic people who wholeheartedly devoted their time, energy, and support to help me during my studies Therefore, this is my opportunity to acknowledge my great debt of thanks to them I wish to express my thanks and gratitude to my academic supervisor, Prof Dr Quang-Cherng Hsu, for his continuous guidance, valuable advice, and helpful supports during my studies He has always been supportive of my research work and gave me the freedom to fully explore the different research areas related with my study I wish to acknowledge my deepest thanks to Vietnam Ministry of Education and Taiwan Ministry of Education for giving me a great opportunity, necessary scholarships to study at NKUST via VEST500 scholarship which is corporation between Vietnam Government and Taiwan Government, and many enthusiastic helps during my time in NKUST I am also particularly grateful to Thai Nguyen University of Technology (TNUT) provided me unflagging encouragement, continuous helps and support to complete this course My gratitude also goes to all of the teachers, Dean and staffs of Department of Mechanical Engineering at NKUST for their devoted teaching, great helping and thoughtful serving during my study iv I would also like to express my sincere gratitude to all of my colleagues at the Precision and Nano Engineering Laboratory (PANEL), Department of Mechanical Engineering, NKUST I want to express my sincere thanks to all my Vietnamese friends in NKUST for their helpful sharing and precious helping me over the past time I also wish to express my gratitude to all those who directly or indirectly helped me during my study in NKUST Finally, my special thanks to my dad Ngo The Long and my mom Vu Thi Hai, to my older sister Ngo Thi Phuong, to my adorable wife Duong Thi Huong Lien, to two lovely little daughters Ngo Duong Anh Thu and Ngo Phuong Linh, who are the most motivation for me over years in Taiwan! v CONTENTS 中文摘要…………………………………………………………………………….i ABSTRACT……………………………………………………………………… ii ACKNOWLEDGMENTS……………………………………………………… iv CONTENTS……………………………………………………………………… vi LIST OF FIGURES…………………………………………………………… xii LIST OF TABLES…………………………………………………………… xvi NOMENCLATURE…………………………………………………………… xvii Chapter Introduction…………………………………………………………….1 1.1 Motivation of the research……………………………………………………1 1.2 Scopes of the research……………………………………………………… 1.3 Contributions…………………………………………………………………9 1.4 Organization of the dissertation…………………………………………… Chapter Theory of image processing and machine vision……………………12 2.1 Image processing system……………………………………………………12 2.1.1 Basics in image processing…………………………………………… 13 2.1.1.1 Pixels…………………………………………………………… 13 2.1.1.2 Resolution of image…………………………………………… 13 vi 2.1.1.3 Gray level……………………………………………………… 14 2.1.1.4 Histogram……………………………………………………… 15 2.1.1.5 Image presentation……………………………………………….16 2.1.1.6 Color models…………………………………………………… 17 2.1.1.7 Neighbors of pixels………………………………………………18 2.1.2 Morphological image processing …………………………………… 19 2.1.2.1 Erosion operation……………………………………………… 19 2.1.2.2 Dilation operation……………………………………………… 19 2.1.2.3 Opening and closing operations …………………………………20 2.1.3 Blob analysis………………………………………………………… 21 2.1.3.1 Goal of blob analysis ……………………………………………21 2.1.3.2 Feature extraction ………………………………………………21 2.1.3.3 Steps to perform blob analysis …………………………………22 2.2 Machine vision system …………………………………………………….22 2.2.1 Lighting design ………………………………………………… .23 2.2.2 Lens …………………………………………………………… … 25 2.2.3 Image sensors ………………………………………………… … 27 Chapter Coordinate calibration methods and camera calibration………….30 3.1 Two-Dimensional coordinate calibration………………………………… 30 vii ... National Kaohsiung University of Science and Technology ABSTRACT This research presents development of machine vision systems on object classification and measurement for robot manipulation Firstly,... 關鍵字：機械視覺、機械手、相機校正、影像分析、物件分類、光源 i Development of machine vision systems on object classification and measurement for robot manipulation Graduate student: Ngo Ngoc Vu Advisor: Prof Quang-Cherng Hsu Department of Mechanical... computer vision system for detection and autonomous object manipulation placed randomly on a target surface and controls an educational robotic arm with degree of freedom (DOF) to pick it up and place