Development of vision based systems for structural health monitoring

Development of vision-based systems for structural health monitoring Ho Hoai Nam The Graduate School Sejong University Department of Civil and Environmental Engineering Development of vision-based systems for structural health monitoring Ho Hoai Nam A Dissertation Submitted to the Department of Civil and Environmental Engineering, and the Graduate School of Sejong University in partial fulfillment of the requirements for the degree of Doctor of Philosophy December 2013 Approved by Jong-Jae Lee Major Advisor TABLE OF CONTENTS TABLE OF CONTENTS i LIST OF TABLES vi LIST OF FIGURES vii ABSTRACT xii CHAPTER INTRODUCTION 1.1 Introduction 1.2 Scope and Objectives 1.3 Organization CHAPTER FUNDAMENTALS OF DIGITAL IMAGE PROCESSING AND PATTERN RECOGNITION 2.1 Digital image formation and representation 2.1.1 Image formation 2.1.2 Image presentation 2.1.3 Coordinate convention 2.2 Color fundamentals 10 2.2.1 Basic concepts 10 2.2.2 Color models 14 2.3 Basic image processing operations 25 2.3.1 Elementary image processing operators 25 i 2.3.2 Binary image processing operations 26 2.3.3 Geometric transform 27 2.4 Edge detection algorithms 28 2.4.1 Roberts Cross edge detector 29 2.4.2 Sobel edge detector 31 2.4.3 Prewitt gradient edge detector 34 2.4.4 Laplacian edge detector 35 2.4.5 Canny edge detection 36 2.5 Image segmentation algorithms 39 2.5.1 Threshold techniques 39 2.5.2 Edge-based methods 40 2.5.3 Region-based techniques 40 2.5.4 Connectivity-preserving relaxation methods 40 2.6 Pattern recognition 41 2.6.1 Principal component analysis (PCA) 41 2.6.2 Support Vector Machines (SVM) 45 2.6.3 Artificial Neutral Networks (ANN) 49 CHAPTER ADVANCED VISION-BASED SYSTEMS FOR DISPLACEMENT AND ROTATION ANGLE MEASUREMENT 53 3.1 Introduction 53 3.2 Advanced vision-based displacement measurement system 53 ii 3.2.1 Development of the advanced vision-based system 54 3.2.2 Displacement measurement of high-rise buildings 72 3.3 High resolution rotation angle measurement system 85 3.3.1 A vision-based rotation angle measurement system 86 3.3.2 Verification tests 88 3.3.3 Testing on a single-span bridge 93 CHAPTER AN EFFICIENT VISION-BASED 3-D MOTION MONITORING SYSTEM FOR CIVIL STRUCTURES 97 4.1 Introduction 97 4.2 Development of 3-D vision-based motion monitoring system 99 4.3 Experimental verifications 107 CHAPTER AN IMAGE-BASED SURFACE DAMAGE DETECTION SYSTEM FOR CABLE-STAYED BRIDGES 111 5.1 Introduction 111 5.2 Cable inspection system 112 5.3 Damage detection algorithm 114 5.4 Experimental verifications 119 iii CHAPTER SONAR IMAGE ENHANCEMENT APPLYING FOR UNDERWATER CIVIL STRUCTURES 124 6.1 Fundamentals of Sonar 124 6.1.1 Beam pattern 126 6.1.2 Resolution 128 6.1.3 Backscatter 132 6.1.4 Distortions 132 6.2 Reading sonar image file 135 6.3 Sonar image enhancement techniques 136 6.3.1 Median filter 136 6.3.2 Morphological filter 139 6.3.3 Weiner filter 144 6.3.4 Frost filter 146 6.3.5 Contrast enhancement 149 6.4 Efficient image enhancement algorithm 153 CHAPTER CONCLUSIONS AND FUTURE STUDY 156 7.1 Conclusions 156 7.1.1 Synchronized real-time vision-based systems applying for displacement and rotation measurement 156 7.1.2 An 3D motion measurement system for civil infra-structures 157 iv 7.1.3 An efficient image-based damage detection for cable surface in cable-stayed bridges 157 7.1.4 Sonar image enhancement applying for underwater civil structures 158 7.2 Recommendations for future study 158 REFERENCES 160 APPENDICES 175 v LIST OF TABLES Table 3.1: List of hardware components 75 Table 3.2: Static test results 80 Table 3.3: Commercial tilt sensors 86 Table 3.4: Static test results 91 Table 4.1: Static displacement test results 109 Table 4.2: Static rotation test results 109 Table 5.1: Typical camera specifications [95] 113 Table 5.2: Laboratory test results 120 Table 6.1: Rules for Dilation and Erosion 141 vi LIST OF FIGURES Figure 2.1: Model of a digital formation system Figure 2.2: Combination and variant of RGB color Figure 2.3: Coordinate convention of digital images 10 Figure 2.4: Energy spectrum [37] 11 Figure 2.5: Mixture of light and Mixture of pigments [36] 12 Figure 2.6: The wavelengths of the vision spectrum 14 Figure 2.7: RGB color cube 16 Figure 2.8: Example of color, gray and binary image 17 Figure 2.9: CMY color cube 19 Figure 2.10: HIS color cube 22 Figure 2.11: Image rotation 27 Figure 2.12: Shear along horizontal axis 28 Figure 2.13: Roberts Cross convolution kernels 30 Figure 2.14: Results of applying Roberts Cross convolution kernels 31 Figure 2.15: Sobel convolution kernels 32 Figure 2.16: Outputs of Sobel and Roberts edge detectors 34 Figure 2.17: Masks for the Prewitt gradient edge detector 34 Figure 2.18: Three small common kernels of Laplacian edge detector 35 Figure 2.19: Comparison of the edge detection methods 38 vii APPENDICES SOFTWARE USER’S USER MANUAL Synchronized multi-point multi displacement measurement Slave software Select the ROI (Region of Interest) by clicking the appropriate button on the menu 175 Apply : connects the cameras to computers Property : displays the properties of the connected cameras Start capture : begins capturing the images Stop capture : stops capturing the images ROI : clicks the button name to select the corresponding ROI ROI adjustment : adjusts the size of ROI to best fit the target points 176 Master software IP address : types in the IP address of the salve computer Connect : connects the slave computer to the Master software Disconnect : removes the slave computer from the Master software Target panel : inputs the lengths of the target in x and y directions in millimeter-scale Scale : selects the range to display the result 177 178 The 3D motion measurement Main menu Calibration: Taking the calibration process before conducting the measurement for all cameras This is the first step needs to be done Real-time measurement: Conducting the real-time measurement task This job is only conducted when calibration process is successfully carried out Saving video files: Saving the measurement into video files for Off-line measurement Offline measurement: Processing the saved video files in offline mode 179 Calibration Following the basic steps: Choosing the desired camera Click “Apply” button Choosing the correct ROI position Deciding the appropriate size of ROI Entering the correct Target size 180 Real-time measurement The following steps should be followed: Click “Apply” button, there will be a message to confirm the status otherwise the program will be stopped Entering the correct target distance in x direction (Lx) and y direction (Ly) Click “Start Capture” button Choosing the suitable point to begin measurement, and adjusting the appropriate size of ROI for each camera, if needed The displacement of measured by camera, the final displacement, and the rotation value are displayed in the graphs Click “Save” button to store the results, then click “Stop” button to finish saving data 181 Offline measurement The following steps should be followed: Click “Load” button, there will be a message to confirm the status otherwise the program will be stopped Entering the correct target distance in x direction (Lx) and y direction (Ly) Click “Start Capture” button Choosing the suitable measuring point to begin measurement, and adjusting the appropriate size of ROI for each camera, if needed The displacement of measured by camera, the final displacement, and the rotation value are displayed in the graphs Click “Save” button to store the results, then click “Stop” button to finish saving data 182 Saving video file Click “Grab” button to capture images, click “Save” to save into video files, then the click “Stop” when completing the saving process The frame rate also can be set 183 논문초록 영상 기반 구조물 건전도 모니터링 시스템 개발 Ho Hoai Nam 토목환경공학과 세종대학교 최근 영상 수집 장비들의 발전에 따라 다양한 분야에서 영상을 기반으로 한 장비의 개발이 이루어 지고 있다 다양한 장비를 이용해 수집된 영상들은 시간과 공간 정보를 포함하고 있기 때문에 고유 진동수, 감쇠비, 모드형상과 같은 구조물의 동 특성을 추정하는데 사용될 수 있다 이 논문에서는 토목 구조물의 건전도 평가를 위해 사용 될 수 있는 몇 가지 효과적인 영상 기반 시스템 개발에 관한 연구를 수행하였다 일반적으로 대형 토목 구조물의 건전도 평가는 물리량의 측정에서부터 시작된다 계측 가능한 물리량 중 구조물의 변위와 회전각을 측정하는 많은 시스템이 있지만, 이는 고가의 장비, 토목 구조물의 적용성에 한계가 있다 따라서 본 연구에서는 경제적이고 효율적인 영상 기반 변위/회전각 측정 시스템을 개발하였다 개발된 시스템은 일반 캠코더, 영상변환/저장 장치, 컴퓨터 만으로 정밀한 변위 및 회전각 측정이 가능하다 개발된 시스템의 토목 구조물에 대한 적용성을 높이기 위하여 소프트웨어를 개발하였다 개발된 소프트웨어는 계측과 계측을 제어하는 프로그램으로 구성되어 있으며, 계측을 제어하는 프로그램은 계측 프로그램의 시간 동기화와 계측 프로그램에서 계측하는 데이터와 시간 정보를 무선 및 유선으로 전송이 가능하게 개발되었다 개발된 시스템은 실내 실험을 통해 검증을 수행하였다 또한, 대형 토목 구조물에 차원 움직임을 효율적으로 측정하기 위한 시스템을 개발하였다 개발된 시스템은 세 대의 캠코더, 한대의 PC, 영상변환/저장 장치로 구성되어 있으며, 자동으로 최적의 임계치를 판단 할 수 있는 알고리즘을 이용하여 소프트웨어를 개발하였다 이를 통하여 실시간 차원 움직임 계측 가능한 시스템을 개발하였다 개발된 시스템은 정적 및 동적 하중에 대한 실내 실험을 통해 검증하였다 케이블 교량의 증가에 따라 케이블 교량의 유지관리의 필요성이 높아지고 있다 케이블 교량에서 중요한 부재인 케이블의 효율적인 점검을 위하여 영상을 기반으로 케이블 표면의 손상을 자동으로 검출할 수 있는 시스템을 개발하였다 케이블 등반 로봇에 장착된 카메라를 통해 이미지를 촬영하고, 촬영된 이미지를 무선으로 수집한다 수집된 이미지의 효과적인 손상 검출을 위하여 이미지의 노이즈 제거 및 개선 작업을 수행하였다 개선된 이미지를 PCA 기법을 통해 훈련된 데이터를 통해 Mahalanobis square distance 기법을 통해 케이블 외관의 손상을 감지한다 개발된 시스템을 가지 다른 형식의 케이블을 실내실험을 통하여 검증하였고, 실제 교량에 적용을 통해 개발된 시스템의 성능 및 적용성을 검증하였다 최근 늘어나는 기후 변화에 따라 태풍과 홍수에 의한 교량의 피해가 늘어나고 있다 최근 교량 수중 구조물과 하상의 영상을 주변환경의 영향 없이 수집할 수 있는 수중 음파 탐지기를 이용한 수중점검이 연구가 수행되고 있다 수중 음파 탐지기를 통해 수집된 영상을 이용하여 효율적인 점검을 위한 영상 개선을 수행하였다 수집된 영상을 개선하기 위한 작업은 노이즈를 제거하기 위한 작업과 이미지를 선명하게 하는 작업으로 나누어진다 Wiener Filter 와 Median Filter 를 이용하여 수집된 영상의 노이즈를 제거 하기 위해 사용되었으며, 노이즈가 제거된 이미지를 de-blured 와 unsharp masking 기법을 이용하여 히스토그램의 평활화 과정을 거쳐 이미지를 향상 시켰다 제안된 알고리즘은 많은 수중 음향 이미지에 적용 되었으며, 이를 통해 점검의 효울성을 높이게 되었다 ACKNOWLEDGEMENTS First and foremost, I would like to express my sincere gratitude and respect to my advisor, Professor Jong-Jae Lee, for his valuable advices and guidances throughout my doctoral program at the Department of Civil and Environmental Engineering, Sejong University This work would not have been made possible without his knowledgeable guidances and generous support I cannot thank him enough for all he did for me, and I am really lucky to work with him I would like to thank all my committee members Professor Dong-Joo Kim, Professor SeungEock Kim, Professor Dae-Hyun Kim and Professor Jong-Han Lee for their guidances, suggestions and constructive comments while serving on my doctoral committee Many thanks to all my current and former labmates, with my special gratitude goes to YoungSoo Park, Nam-Gyu Kim, Hyeon-Seong Park, Ki-Deok Kim, Se-Hoon Kim, In-Ho Moon, Seong-Yeon Kim, Guiyu Jin, Ji-Ho Jung and Jeong-Hun Park, for their dear help and kind assistance through the years, their cheerful and lively companionship has been a real pleasure They have relieved my struggle and refreshed my view on life I would like to give a big thank-you to all my Vietnamese friends for the selfless help and encouragement during my study in Korea Besides, I am grateful to all faculty members of Hochiminh city Industry and Trade College, Vietnam for their constant emotional support during my doctoral course Finally, a special appreciation goes to my dear parents and younger sister Without their understanding, endless encouragement and support over the years, I could have achieved nothing I would like to dedicate this dissertation to them Ho Hoai Nam Sejong University, Seoul, Korea February, 2014 [...]... 152 Figure 6.21: SONAR image enhancement algorithm 154 Figure 6.22: Results of the proposed algorithm 155 xi ABSTRACT Development of vision- based systems for structural health monitoring Ho Hoai Nam Department of Civil and Environmental Engineering The Graduate School of Sejong University Vision- based systems are regaining their popularity in various engineering disciplines due to a... the main motivations for the development of image -based detection systems which can be used to remotely and non-contact inspected the structure conditions to reduce the inspection time, cost, and to increase the effectiveness, safety of structural health monitoring Displacement is an important physical quantity to be monitored in the framework of structural health monitoring Vision- based displacement... concerns with vision- based systems utilizing in non-contact assessment of civil structures which have low natural frequencies The main objective of this study is to develop vision- based systems for deformation measurement and damage detection of structures The specific objectives of this study are: • Develop an efficient synchronized multi-point vision- based displacement system applying for civil structures... cycle Thus, structural health monitoring plays an important role to ensure structures maintain their structural performance Based on knowledge of structural operational condition, suitable maintenance procedures can be performed to prolong the service life of the structure, and the risk of catastrophic failure should be significantly reduced Two general approaches to structural health monitoring include... information of the target object; hence can be used to extract the object’s dynamic characteristics such as natural frequencies, damping ratios, and mode shapes In this dissertation, some effective vision- based systems that can be used for structural health monitoring have been successfully developed The advanced vision- based systems are developed to remotely measure the displacement and rotation of. .. tornadoes, typhoons etc., so structural health monitoring plays a key role to sustain long service-life of structures The structural health monitoring starts with the inspection of the structural members to determine the present condition For a long time, the inspection has been carried out by field inspectors and/or using contact-type sensors [11, 12] In reality, the inspection of the large structures... predict the remaining service life of the structure Other main advantages of the global methods are ready for automatic processing and have little reliance on engineers’ judgments or on the analytical model of structure A lot of studies in the field of structural health monitoring were extensively surveyed in the Ref [1, 2] With the advent of inexpensive and high performance image capture devices and... processing techniques, vision- based systems become a popular tool of great interest for wide applications in medicine, archaeology, biomechanics, astronomy, public safety, etc The vision- based system is regaining its popularity in various engineering disciplines due to a recent remarkable evolution in cost-effective and high-performance image capture devices In civil engineering, vision- based systems have been... angle measurement for civil structures • Extend the multi-point vision- based displacement for 3D real-time measurement of structures • Develop an effective image -based system for damage detection of bridge cables, and propose an efficient sonar image enhancement algorithm applying for underwater civil structures 1.3 Organization This dissertation consists of seven chapters, a brief summary of chapter-by-chapter... digital image processing applying for civil applications including digital image formation, color models, basic image processing, edge detection, image segmentation, and popular pattern recognition techniques 4 Chapter 3 develops the vision- based displacement and rotation measurement systems for large civil structures The vision- based measurement system consists of software and hardware (lowcost PCs,