Development of machine vision systems on object classification and measurement for robot manipulation

國立高雄科技大學 機械工程系 博士班 博士論文 操縱機器人所需物件分類和量測機械視覺系統之開發 Development of machine vision systems on object classification and measurement for robot manipulation Graduate student: Ngo Ngoc Vu 研究生：吳玉武 指導教授：許光城 教授 Advisor: Prof Quang-Cherng Hsu 中華民國 108 年 月 i 操縱機器人所需物件分類和量測機械視覺系統之開發 Development of machine vision systems on object classification and measurement for robot manipulation Graduate student: Ngo Ngoc Vu 研究生：吳玉武 Advisor: Prof Quang-Cherng Hsu 指導教授：許光城 教授 國立高雄科技大學 機械工程系 博士論文 A Dissertation Submitted to Department of Mechanical Engineering National Kaohsiung University of Science and Technology in Partial Fulfillment of the Requirements for the Degree of Doctor of Philosophy in Mechanical Engineering January, 2019 Kaohsiung, Taiwan, Republic of China 中華民國 108 年 月 ii iii 操縱機器人所需物件分類和量測機械視覺系統之開發 研究生：吳玉武 指導教授：許光城 教授 國立高雄科技大學 機械工程系 博士班 中文摘要 本研究介紹了機器視覺系統在物體分類和機器人操縱測量方面的研究。首先， 在不同的照明條件下開發了一套可用於金屬零件的自動分類和量測的機械視覺系 統，並將其應用於具 個自由度（DOF）的機器手臂。為了獲得準確的定位資訊， 整個圖像藉由工作平台上方的 CMOS 攝影機來進行取像。本研究中亦探討不同照明 條件下對本系統的影響，在正向打光的條件中有四種不同的打光狀態，且在每種條 件，皆使用全域和局部閾值來獲得較佳的圖像品質。研究中主要利用張氏方法、線 性轉換函式與二階轉換函式來取得影像座標與世界座標的關係。 實驗結果顯示，在背向打光相較於前向打光其所獲得的物體中心更為準確，而 在校正結果的部分，二階轉換函式比其他校正方法更準確。透過使用二階轉換函式 的校正偏差，在 X 和 Y 軸其最大正偏差分別為 0.48 mm 和 0.38 mm 而最大負偏差分 別為-0.34 mm 和-0.43 mm。 本研究亦開發了一套機械視覺系統於六自由度(DOF)的機械手臂用以進行物件 顏色的分類和座標的量測。整個圖像藉由工作平台上方的兩個相機以藉此獲得準確 的定位資訊，並在二維(2-D)和三維(3-D)的校正過程中採用二階轉換函式與透視投影 法來建立影像坐標和世界坐標間的關係。在二維校正的部分，於 X 和 Y 軸上的最大 正偏差分別為 1.29 mm 和 1.12 mm；而最大負偏差分別為-1.48 mm 和-0.97 mm。在 三維校正的部分，X，Y 和 Z 方向的偏差分別為 0.07、-0.418 和-0.063 mm。本研究 所提出的視覺識別系統可以對物件進行分類並取得該物件之三維坐標。 關鍵字：機械視覺、機械手、相機校正、影像分析、物件分類、光源 i Development of machine vision systems on object classification and measurement for robot manipulation Graduate student: Ngo Ngoc Vu Advisor: Prof Quang-Cherng Hsu Department of Mechanical Engineering National Kaohsiung University of Science and Technology ABSTRACT This research presents development of machine vision systems on object classification and measurement for robot manipulation Firstly, a machine vision system for the automatic metal part classification and measurement process is developed under different lighting conditions, and has been applied to the operation of a robot arm with degrees of freedom (DOF) In order to obtain accurate positioning information, the overall image is captured by a CMOS camera which is mounted above the working platform The effects of back-lighting and frontlighting conditions to the proposed system were investigated With the frontlighting condition, four different conditions were performed For each condition, global and local threshold operations were used to obtain good image quality The relationship between the image coordinates and the world coordinates was determined through Zhang’s method, the linear transformation and the quadratic transformation during the calibration process Experimental results show that in a back-lighting environment, the image quality is improved, such that the positions of the centers of objects are more accurate than in a front-lighting environment According to the calibration results, the quadratic transformation is more accurate ii than other methods By calculating the calibration deviation using the quadratic transformation, the maximum positive deviation is 0.48 mm and 0.38 mm in the X and Y directions, respectively The maximum negative deviation is -0.34 mm and -0.43 mm in X and Y directions, respectively The proposed system is effective, robust, and can be valuable to industry The second, a machine vision system for color object classification and measurement process for robot arm with six degree of freedom (DOF) is developed In order to obtain accurate positioning information, the overall image is captured by a double camera C615 and a camera C525 which are mounted above the working platform The relationship between the image coordinate and the world coordinate is performed through calibration procedure The quadratic transformation and generalized perspective transformation algorithms were used to transform coordinates in 2-D and 3-D calibration process, respectively According to calibration results, with 2-D calibration, the positive maximum deviation is 1.29 mm and 1.12 mm in X and Y directions, respectively The negative maximum deviation is -1.48 mm and -0.97 mm in X and Y directions, respectively With 3-D calibration, the deviation is 0.07 mm, -0.418 mm, and -0.063 mm in X, Y and Z directions, respectively The proposed system can catch the three dimensional coordinates of the object and perform classification and assembly automatic operations by the data from visual recognition system Keywords: machine vision, robot arm, camera calibration, image analysis, object recognition, lighting source iii ACKNOWLEDGMENTS The fulfillment of over three years of study at National Kaohsiung University of Science and Technology (NKUST) has brought me into closer relations with many enthusiastic people who wholeheartedly devoted their time, energy, and support to help me during my studies Therefore, this is my opportunity to acknowledge my great debt of thanks to them I wish to express my thanks and gratitude to my academic supervisor, Prof Dr Quang-Cherng Hsu, for his continuous guidance, valuable advice, and helpful supports during my studies He has always been supportive of my research work and gave me the freedom to fully explore the different research areas related with my study I wish to acknowledge my deepest thanks to Vietnam Ministry of Education and Taiwan Ministry of Education for giving me a great opportunity, necessary scholarships to study at NKUST via VEST500 scholarship which is corporation between Vietnam Government and Taiwan Government, and many enthusiastic helps during my time in NKUST I am also particularly grateful to Thai Nguyen University of Technology (TNUT) provided me unflagging encouragement, continuous helps and support to complete this course My gratitude also goes to all of the teachers, Dean and staffs of Department of Mechanical Engineering at NKUST for their devoted teaching, great helping and thoughtful serving during my study iv I would also like to express my sincere gratitude to all of my colleagues at the Precision and Nano Engineering Laboratory (PANEL), Department of Mechanical Engineering, NKUST I want to express my sincere thanks to all my Vietnamese friends in NKUST for their helpful sharing and precious helping me over the past time I also wish to express my gratitude to all those who directly or indirectly helped me during my study in NKUST Finally, my special thanks to my dad Ngo The Long and my mom Vu Thi Hai, to my older sister Ngo Thi Phuong, to my adorable wife Duong Thi Huong Lien, to two lovely little daughters Ngo Duong Anh Thu and Ngo Phuong Linh, who are the most motivation for me over years in Taiwan! v CONTENTS 中文摘要…………………………………………………………………………….i ABSTRACT……………………………………………………………………… ii ACKNOWLEDGMENTS……………………………………………………… iv CONTENTS……………………………………………………………………… vi LIST OF FIGURES…………………………………………………………… xii LIST OF TABLES…………………………………………………………… xvi NOMENCLATURE…………………………………………………………… xvii Chapter Introduction…………………………………………………………….1 1.1 Motivation of the research……………………………………………………1 1.2 Scopes of the research……………………………………………………… 1.3 Contributions…………………………………………………………………9 1.4 Organization of the dissertation…………………………………………… Chapter Theory of image processing and machine vision……………………12 2.1 Image processing system……………………………………………………12 2.1.1 Basics in image processing…………………………………………… 13 2.1.1.1 Pixels…………………………………………………………… 13 2.1.1.2 Resolution of image…………………………………………… 13 vi 2.1.1.3 Gray level……………………………………………………… 14 2.1.1.4 Histogram……………………………………………………… 15 2.1.1.5 Image presentation……………………………………………….16 2.1.1.6 Color models…………………………………………………… 17 2.1.1.7 Neighbors of pixels………………………………………………18 2.1.2 Morphological image processing …………………………………… 19 2.1.2.1 Erosion operation……………………………………………… 19 2.1.2.2 Dilation operation……………………………………………… 19 2.1.2.3 Opening and closing operations …………………………………20 2.1.3 Blob analysis………………………………………………………… 21 2.1.3.1 Goal of blob analysis ……………………………………………21 2.1.3.2 Feature extraction ………………………………………………21 2.1.3.3 Steps to perform blob analysis …………………………………22 2.2 Machine vision system …………………………………………………….22 2.2.1 Lighting design ………………………………………………… .23 2.2.2 Lens …………………………………………………………… … 25 2.2.3 Image sensors ………………………………………………… … 27 Chapter Coordinate calibration methods and camera calibration………….30 3.1 Two-Dimensional coordinate calibration………………………………… 30 vii Chapter Conclusions and future works 6.1 Conclusions This study presented the development classification and measurement systems including the metal part and the color object classification system using machine vision According to this research work, the conclusions were drawn below: 6.1.1 Conclusion for the classification and measurement system for metal parts Image processing technology was applied to process overall data image Multiple lighting sources were utilized to assess the system’s ability to identify and classify objects Under backlighting source, contour extraction of target objects within a binary image is better However, under different front-lighting sources, to obtain contour of the target objects, the global and local thresholding operations were used to control gray threshold values To determine position information in physical space from acquired images, calibration procedure was set and compared to select best accuracy calibration method for the proposed system Firstly, camera calibration was performed using Matlab software to determine the intrinsic and extrinsic parameters of camera, and then calibration accuracy was accessed Besides, the linear transform was used to mutate coordinates between the image coordinate system and the world coordinate system However, calibration error of this method was high To reduce deviation, the quadratic transformation using least square method was used This method was more complex than linear transform, but the calibration error was smaller Lighting condition affects to precise of classification and position of objects With back-lighting condition, the proposed system can classify and determine the position of objects easier and more accurate Front lighting and some room-lighting with black 100 cover are suitable to use for improving uniformity of light for all objects (bolts, nuts and washers) With the proposed system, all components were classified accurately 100% Positions of them in the world coordinate system were determined for controlling robot arm during working process 6.1.2 Conclusion for the classification and measurement system for color objects In this study, a color object classification and measurement system for a robotic arm was developed, using a machine vision system With this system, the robotic arm could accurately recognize and classify different color objects The conclusions drawn from this research are summarized as follows: For the 2-D object classification and calibration work, the CMOS camera C525 was able to successfully capture the data related to location The quadratic transformation, which is a plane-to-plane transformation, was used to determine the relationship between the image coordinates and the world coordinates Calibration errors of this method is in the range of resolution of the proposed 2-D system For the 3-D object recognition and calibration work, an automatic determination of a position of interest on the calibration pattern was performed successfully The perspective transformation was used to determine the relationship between the image coordinates and the world coordinates Concerning the image calibration, the conversion process from the world coordinate system to the image coordinate system, as well as the reverse process, of five calibration points (from point to point 5) was successful The error was small and within a suitable range of accuracy for the proposed system Algorithms were developed which could ascertain the orientations of different geometries, allowing the robotic arm to position the triangles into their respective 101 holes accordingly By using the RGB colour model, and choosing suitable threshold values for each colour, including red, blue, green, and yellow, all the assembly modules and their associated parts, were recognised The program was able to present the ID, name, and centre lines of the objects and holes accurately All assembly parts and models were classified accurately 100% Positions of them in the world coordinate system were determined for controlling robot arm during working process 6.2 Future works and suggestions In addition to the work realized in this study, there are several points which need to be further investigated and improved in the future, as follows: a Machine vision has been applied in many fields to increase productivity and decrease product price In mechanical engineering, machine vision systems can be developed for defect inspection of online complex geometries or hazardous environment with humans It can be used to replace for non-contact measurement devices which is high cost such as ATOS or contact measurement devices such as CMM (Coordinate Measuring Machine) b New calibration methods and new calibration patterns will be developed to improve accuracy of machine vision based systems c Advanced open software packages will be used such as C# with image library Emgu CV, C++, Python with image library OpenCV or close software packages such as Matlab or Halcon for complex geometry recognition and measurement 102 List of publications Journal papers (SCI) Ngo, N V; Hsu, Q C; Hsiao, W L; Yang, C J (2017) Development of a simple three-dimensional machine-vision measurement system for in-process mechanical parts Advances in Mechanical Engineering, 9(10), IF: 0.848, DOI 1687814017717183 Hsu, Q C; Ngo, N V; Ni, R H (2018) Development of a Faster Classification System for Metal Parts Using Machine Vision under Different Lighting Environments The International Journal of Advanced Manufacturing Technology, Vol 76, No 1-4, pp 247-254, IF: 2.601, DOI: 10.1007/s00170-018-2888-7 Ngo, N V; Glen, A P; Hsu, Q C (2019) Development of a Color Object Classification and Measurement System using Machine Vision International Journal of Precision Engineering and Manufacturing (Under review) Journal papers (EI) Ngo, N V; Hsu, Q C; Li, W H; Huang, P J (2017) Optimizing Design of Two-dimensional Forging Preform by Bi-directional Evolutionary Structural Optimization Method Procedia Engineering, 207, 520-525 DOI: https://doi.org/10.1016/j.proeng.2017.10.815 International Conference Papers Hsu, Q C; Ngo, N V; Chen, K M (2016) Development of Positioning System for Edge-milling of Different Wind Deflector by Machine Vision IEEE ICASI 2016, International Conference on Applied System Innovative, Okinawa, Japan Hsu, Q C; Ngo, N V; Lin, H W; Jui, Y C (2016) Development of Simple Equipped 3D Measurement System for in-process Mechanical Parts IEEE ICASI 2016, International Conference on Applied System Innovative, Okinawa, Japan 103 Wu, N J; Hsu, Q C; Do, T V; Ngo, N V (2017) Study on Three Dimensional Metrology System with Automatic Calibration and Spot Guidance Technology for Metal Product Measurement The 14th International Conference on Automation Technology, Kaohsiung, Taiwan Ngo, N V; Hsu, Q C; Ni, R H (2017) Development of Smart Faster Classification and Assembly System for Robot Arm under Non-Control Lighting Environment The 14th International Confer on Automation Technology, Kaohsiung, Taiwan Hsu, Q C; Jian, Y L; Ngo, N V (2018) Development on Automatic Optical Measurement System for Ring-Shaped Workpiece Contour with Economical Cost and High Performance The 6th International Symposium on Sensor Science and 4th SPINTECH Technology Thesis Award, Taiwan 104 References [1] Leemans, V., Magein, H., & Destain, M F (2002) AE-automation and emerging technologies: On-line fruit grading according to their external quality using machine vision Biosystems Engineering, 83(4), 397-404 [2] Blasco, J., Aleixos, N., & Moltó, E (2003) Machine vision system for automatic quality grading of fruit Biosystems engineering, 85(4), 415-423 [3] Cubero, S., Aleixos, N., Moltó, E., Gómez-Sanchis, J., & Blasco, J (2011) Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables Food and Bioprocess Technology, 4(4), 487-504 [4] Xiang, R., He, W., Zhang, X., Wang, D., & Shan, Y (2018) Size measurement based on a two-camera machine vision system for the bayonets of automobile brake pads Measurement, 122, 106-116 [5] Balasundaram, M K., & Ratnam, M M (2014) In-process measurement of surface roughness using machine vision with sub-pixel edge detection in finish turning International journal of precision engineering and manufacturing, 15(11), 2239-2249 [6] Mishra, A., Sainul, I A., Bhuyan, S., Deb, S., Sen, D., & Deb, A K (2018) Development of a Flexible Assembly System Using Industrial Robot with Machine Vision Guidance and Dexterous Multi-finger Gripper In Precision Product-Process Design and Optimization (pp 31-71) Springer, Singapore [7] Stone, K., Cintron-Aponte, A., & Simons, B (2018) U.S Patent Application No 15/288,245 [8] Huang, S H., & Pan, Y C (2015) Automated visual inspection in the semiconductor industry: A survey Computers in industry, 66, 1-10 [9] Najrabi, D., Hamghalam, M., & Ayatollahi, A (2018) Diagnosis of astrocytoma and globalastom using machine vision In Fuzzy and Intelligent Systems (CFIS), 2018 6th Iranian Joint Congress on IEEE, 152-155 [10] Možina, M., Tomaževič, D., Pernuš, F., & Likar, B (2013) Automated visual inspection of imprint quality of pharmaceutical tablets Machine vision and applications, 24(1), 63-73 105 [11] Chauhan, V., & Surgenor, B (2017) Fault detection and classification in automated assembly machines using machine vision The International Journal of Advanced Manufacturing Technology, 90(9-12), 2491-2512 [12] Kumar, G S., Natarajan, U., & Ananthan, S S (2012) Vision inspection system for the identification and classification of defects in MIG welding joints The International Journal of Advanced Manufacturing Technology, 61(9-12), 923-933 [13] Huang, C K., Liao, C W., Huang, A P., & Tarng, Y S (2008) An automatic optical inspection of drill point defects for micro-drilling The International Journal of Advanced Manufacturing Technology, 37(11-12), 1133-1145 [14] Park, C., Choi, S., & Won, S (2010) Vision-based inspection for periodic defects in steel wire rod production Optical Engineering, 49(1), 017202 [15] Dworkin, S B.; Nye, T J (2006) Image processing for machine vision measurement of hot formed parts Journal of Materials Processing Technology, 174(1), 1-6 [16] Derganc, J., Likar, B., Pernuš, F (2003) A machine vision system for measuring the eccentricity of bearings Computers in Industry, 50(1), 103-111 [17] Shuxia, G., Jiancheng, Z., Xiaofeng, J., Yin, P., Lei, W (2011) Mini milling cutter measurement based on machine vision Procedia Engineering, 15, 1807-1811 [18] Hsu, Q C., Lin, C W., Chen, J Y (2012) Development of an automatic optical inspection system for defect detection of dental floss picks In Advanced Intelligent Mechatronics (AIM), 2012 IEEE/ASME International Conference, 444-449 [19] Ngo, N V., Hsu, Q C., Hsiao, W L., Yang, C J (2017) Development of a simple three-dimensional machine-vision measurement system for in-process mechanical parts Advances in Mechanical Engineering, 9(10), 1-11 [20] Ali, M H., Kurokawa, S., & Uesugi, K (2014) Camera based precision measurement in improving measurement accuracy Measurement, 49, 138-147 [21] Rejc, J., Kovačič, F., Trpin, A., Turk, I., Štrus, M., Rejc, D., & Munih, M (2011) The mechanical assembly dimensional measurements with the automated visual inspection system Expert Systems with Applications, 38(8), 10665-10675 [22] Martínez, S S., Vázquez, C O., García, J G., & Ortega, J G (2017) Quality inspection of machined metal parts using an image fusion technique Measurement, 111, 374-383 106 [23] Bhuvanesh A., Mani MR (2007) Automatic detection of stamping defects in leadframes using machine vision: Overcoming translational and rotational misalignment The International Journal of Advanced Manufacturing Technology, 32, 1201-1210 [24] Quinn, M K., Spinosa, E., & Roberts, D A (2017) Miniaturisation of pressure-sensitive paint measurement systems using low-cost, miniaturised machine vision cameras Sensors, 17(8), 1708 [25] Zhang, X., Zhang, J., Ma, M., Chen, Z., Yue, S., He, T., & Xu, X (2018) A High Precision Quality Inspection System for Steel Bars Based on Machine Vision Sensors, 18(8), 2732 [26] Cubero, S.; Aleixos, N.; Moltó, E.; Gómez-Sanchis, J.; Blasco, J (2011) Advances in machine vision applications for automatic inspection and quality evaluation of fruits and vegetables Food and Bioprocess Technology, 4(4), 487-504 [27] Jarimopas, B.; Jaisin, N (2008) An experimental machine vision system for sorting sweet tamarind Journal of Food Engineering, 89(3), 291-297 [28] Blasco, J.; Aleixos, N.; Roger, J M.; Rabatel, G.; Molto, E (2002) AE-automation and emerging technologies: robotic weed control using machine vision Biosystems Engineering, 83(2), 149-157 [29] Di Fulvio, G., Frontoni, E., Mancini, A., and Zingaretti, P (2014) A stereovision system for dimensional measurements in industrial robotics applications In Mechatronic and Embedded Systems and Applications (MESA), 2014 IEEE/ASME 10th International Conference, IEEE, 1-6 [30] Phansak, N., Pichitra, U., Kontorn, C (2016) Using machine vision for flexible automatic assembly system The 20th International Conference on Knowledge Based and Intelligent Information and Engineering Systems, KES2016, York, United Kingdom [31] Tsarouchi, P., Matthaiakis, S A., Michalos, G., Makris, S., Chryssolouris, G (2016) A method for detection of randomly placed objects for robotic handling CIRP Journal of Manufacturing Science and Technology, 14, 20-27 [32] Iscimen, B., Atasoy, H., Kutlu, Y., Yildirim, S., Yildirim, E (2015) Smart robot arm motion using computer vision Elektronika ir Elektrotechnika, 21(6), 3-7 107 [33] Rai, N., Rai, B., Rai, P (2014) Computer vision approach for controlling educational robotic arm based on object properties In Emerging Technology Trends in Electronics, Communication and Networking (ET2ECN), The 2nd International Conference, 1-9 [34] Tsarouchi, P., Michalos, G., Makris, S., Chryssolouris, G (2013) Vision system for robotic handling of randomly placed objects Procedia CIRP, 9, 61-66 [35] Juang, J G., Tsai, Y J., Fan, Y W (2015) Visual recognition and its application to robot arm control Applied Sciences, 5(4), 851-880 [36] Shin, K Y., and Mun, J H (2012) A multi-camera calibration method using a 3-axis frame and wand International Journal of Precision Engineering and Manufacturing, 13(2), 283-289 [37] Ji, Y.; Yamashita, A.; Asama, H (2017) Automatic calibration of camera sensor networks based on 3D texture map information Robotics and Autonomous Systems, 87, 313-328 [38] Deng, L., Lu, G., Shao, Y., Fei, M., and Hu, H (2016) A novel camera calibration technique based on differential evolution particle swarm optimization algorithm Neurocomputing, 174, 456-465 [39] You, X., and Zheng, Y (2016) An accurate and practical calibration method for roadside camera using two vanishing points Neurocomputing, 204, 222-230 [40] Ivo, S., Musić, J., Zanchi, V (2013) Improved structured light 3D scanner with application to anthropometric parameter estimation Measurement, 46(1), 716-726 [41] Davies E.R (1998) Automated Visual Inspection, Machine Vision Second ed., Academic Press, New York, Chapter 19, 471–502 [42] Sanz, J.L.C., Petkovic, D (1988) Machine vision algorithm for automated inspection of thin-film disk heads IEEE Transactions on PAMI 10, 830-848 [43] Bahlmann, C., Heidemann, G., Ritter, H (1999) Artificial neural networks for automated quality control of textile seams Pattern Recognition, 32, 1049-1060 [44] Tucker, J.W (1989) Inside beverage can inspection: an application from start to finish Proceecings of the Vision’89 Conference [45] Novini, A.R (1990) Fundamentals of machine vision inspection in metal container glass manufacturing Vision’90 Conference 108 [46] Ker, J., Kengskool, K (1990) An efficient method for inspecting machine parts by a fixtureless machine vision system Vision’90 Conference [47] Torres, T., Sebastian, J.M., Aracil, R., Jimenez, L.M., Reinoso, O (1998) Automated real-time visual inspection system for high-resolution superimposed printings Image and Vision Computing, 16, 947-958 [48] Shafarenko, L., Petrou, M., Kittler, J (1997) Automatic watershed segmentation of randomly textured color images IEEE Transactions on Image Processing, 6, 1530-1543 [49] Li, H., Lin, J.C (1994) Using fuzzy logic to detect dimple defects of polisted wafer surfaces IEEE Transactions on Industry Applications, 30, 1530-1543 [50] Frank, Y.S (2009) Image processing and mathematical morphology fundamental and applications CRC Press, Taylor & Francis Group [51] Scott, E.U (1998) Computer vision and image processing: a practical approach using CVIP tools Prentice-Hall International, Inc [52] Rafael, C.G., Richard, E.W (2002) Digital Image Processing Prentice-Hall, Upper Saddle, New Jersey [53] Ramesh, J., Rangachar, K., Brian G.S (1995) Machine Vision McGraw-Hill [54] Matrox Image Library, version 8, User guide, Manual no 10513-301-0880, June 1, 2005 [55] David, F.R., Alan, J A (1990) Methematical Elements for Computer Grathics McGraw-Hill [56] Erwin, K (1999) Advanced Engineering Mathematics John Wiley & Sons, Inc [57] Fu, K.S., Gonza´lez, R.C., Lee, C.S.G (1997) Robotics: control, sensing, vision, and intelligence New York: McGraw-Hill [58] Oliver, F (2001) Three-Dimension Computer Vision: A geometry viewpoint The MIT Press, Cambridge, Massachusetts, London, England [59] Gary, B., Adrian, K (2008) Learning OpenCV: Computer vision with the OpenCV library O’REILLY [60] Peter, C (2013) Robotics, Vision and Control: Fundamental Algorithms in Matlab Springer [61] http://www.hiwin.tw/download/Tech_doc/Robot/RA605_Manipulator_manual (EN).pdf 109 [62] Tsai, R.Y (1987) A versatile camera calibration technique for high-accuracy 3D machine vision metrology using off-the-shelf TV cameras and lenses IEEE Journal of Robotics and Automation, 3, 323-344 [63] Zhang, Z (2000) A flexible new technique for camera calibration IEEE Transactions on pattern analysis and machine intelligence, 22,1330-1334 [64] Zhang, W., Yuchen, B., Feng, D (2012) Comparision of calibration methods for bubbly flow video image Control and Decision Conference (CCDC), 2012 24th Chinese IEEE [65] https://www.mathworks.com/help/vision/examples/measuring planar objects with a calibrated camera html [66] Chen, H.T (2016) Geometry-based camera calibration using five point correspondences from a single image IEEE Transactions on Circuits and Systems for Video Technology, 27, 2555-2566 [67] Kang, D.J., Wang, H.L (2010) Automatic circle pattern extraction and camera calibration using fast adaptive binarization and plane homography International Journal of Precision Engineering and Manufacturing, 11,13-21 [68] Hsu, Q.C (2003) Comparison of different analysis models to measure plastic strains on sheet metal forming parts by digital image processing International Journal of Machine Tools and Manufacture ,43, 515-521 [69] John, C.R (2007) The Image Processing Handbook Taylor&Francis Group [70] Pérez, L., Rodríguez, Í., Rodríguez, N., Usamentiaga, R., & García, D F (2016) Robot guidance using machine vision techniques in industrial environments: A comparative review Sensors, 16(3), 335 [71] Kopparapu, S K (2006) Lighting design for machine vision application Image and Vision Computing, 24(7), 720-726 [72] Malamas, E N., Petrakis, E G., Zervakis, M., Petit, L., & Legat, J D (2003) A survey on industrial vision systems, applications and tools Image and vision computing, 21(2), 171-188 [73] Martin, D (2007) A practical guide to machine vision lighting Retrieved, 11(05), 2013 [74] Di Fulvio, G., Frontoni, E., Mancini, A., & Zingaretti, P (2014) A stereovision system for dimensional measurements in industrial robotics applications In 110 Mechatronic and Embedded Systems and Applications (MESA) 2014 IEEE/ASME 10th International Conference on (pp 1-6) IEEE [75] Tsarouchi, P., Michalos, G., Makris, S., & Chryssolouris, G (2013) Vision system for robotic handling of randomly placed objects Procedia CIRP, 9, 61-66 [76] https://www.microsoft.com/en-us/research/project/a flexible new technique for camera calibration [77] Kang, D J., Ha, J E., & Jeong, M H (2008) Detection of calibration patterns for camera calibration with irregular lighting and complicated backgrounds International Journal of Control, Automation, and Systems, 6(5), 746-754 [78] Park, J.H., et al (2009) Easy calibration method of vision system for in-situ measurement of strain of thin films Transactions of Nonferrous Metals Society of China, 19, 243-249 [79] Mersch, S (1987) Overview of machine vision lighting techniques In Optics, Illumination, and Image Sensing for Machine Vision International Society for Optics and Photonics, 728, 36-39 [80] Cowan, C K (1991) Automatic camera and light-source placement using CAD models In Automated CAD-Based Vision, 1991, Workshop on Directions in IEEE, 22-32 [81] Hsu, Q C., Ngo, N V., & Ni, R H (2018) Development of a faster classification system for metal parts using machine vision under different lighting environments The International Journal of Advanced Manufacturing Technology, 76(1-4), 247-254 [82] Tan, W., Xuan L , Chen Y., & Erfu Y (2018) A novel method of 3D motion object’s position perception MATEC Web of Conferences, ISWSO 2018, 246, 1-6 [83] Gao, M., Yan, Y., Yang, Y., Huang, J., & He, Z (2016) A positioning system based on monocular vision for industrial robots In Information Science and Control Engineering (ICISCE), 2016 3rd International Conference on IEEE, 784-788 [84] Mustafa, A A., Shapiro, L G., & Ganter, M A (1999) 3-D object identification with color and curvature signatures Pattern recognition, 32(3), 339-355 [85] Lampert, C H., & Peters, J (2012) Real-time detection of colored objects in multiple camera streams with off-the-shelf hardware components Journal of Real-Time Image Processing, 7(1), 31-41 111 [86] Caleiro, P M., Neves, A J., & Pinho, A J (2007) Color-spaces and color segmentation for real-time object recognition in robotic applications Electrúnica e Telecomunicaỗoes, 4(8), 940-945 [87] Zhang, H., & Cao, Q (2017) Texture-less object detection and 6D pose estimation in RGB-D images Robotics and Autonomous Systems, 95, 64-79 112 Appendices Program for camera calibration using Matlab tool M = [1746.1 425.8665;0 1744.9 464.9518;0 1]: Intrinsic matrix RT = [0.9999 -0.0103 -0.0086 -40.0882;0.0103 0.9999 -0.0027 -126.1172; 0.0086 0.0026 1001.6]: Rotation matrix and Translation vector num = csvread('2.csv'); size_num = size(num); for i = : size(num); a(i,1) = M(1,1)*RT(1,1)+M(1,2)*RT(2,1)+M(1,3)*RT(3,1) - num(i,1)*RT(3,1); %x b(i,1) = M(1,1)*RT(1,2)+M(1,2)*RT(2,2)+M(1,3)*RT(3,2) - num(i,1)*RT(3,2); %x c(i,1) = M(2,2)*RT(2,1)+M(2,3)*RT(3,1) - num(i,2)*RT(3,1); %y d(i,1) = M(2,2)*RT(2,2)+M(2,3)*RT(3,2) - num(i,2)*RT(3,2); %y H(1,1) = a(i,1); H(1,2) = b(i,1); H(2,1) = c(i,1); H(2,2) = d(i,1) e(i,1) = -(M(1,1)*RT(1,4)+M(1,2)*RT(2,4)+M(1,3)*RT(3,4) num(i,1)*RT(3,4)); %x f(i,1) = -(M(2,2)*RT(2,4)+M(2,3)*RT(3,4) - num(i,2)*RT(3,4)); %y G(1,1) = e(i,1); G(2,1) = f(i,1); K = inv(H); L(i,1) = K(1,1)*G(1,1)+K(1,2)*G(2,1); L(i,2) = K(2,1)*G(1,1)+K(2,2)*G(2,1); end csvwrite('NEW_WORLDCOOR.csv', L); 113 A User interface of the classification and measurement system for metal parts B User interface of the classification and measurement system for color objects 114 ... conditions for robot manipulation using quadratic transformation algorithm in calibration process The second object studies on application of machine vision for the color object classification. .. National Kaohsiung University of Science and Technology ABSTRACT This research presents development of machine vision systems on object classification and measurement for robot manipulation Firstly,... computer vision system for detection and autonomous object manipulation placed randomly on a target surface and controls an educational robotic arm with degree of freedom (DOF) to pick it up and place