(Đồ án hcmute) design, research and implementation of bearing seals defects detection and classifition system using image processing and deep learning

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT MECHATRONICS ENGINEERING DESIGN, RESEARCH AND IMPLEMENTATION OFBEARING SEALS DEFECTS DETECTION AND CLASSIFICATION SYSTEM USING IMAGE PROCESSING AND DEEP LEARNING LECTURER: M.E NGUYEN MINH TRIET STUDENT: HOANG PHI LONG TRAN VIET THANG SKL010447 Ho Chi Minh City, January 2023 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING BACHELOR THESIS DESIGN, RESEARCH AND IMPLEMENTATION OF BEARING SEALS DEFECTS DETECTION AND CLASSIFICATION SYSTEM USING IMAGE PROCESSING AND DEEP LEARNING HOANG PHI LONG Student ID: 18146040 TRAN VIET THANG Student ID: 18146062 Major: MECHATRONICS ENGINEERING Advisor: M.E NGUYEN MINH TRIET Ho Chi Minh City, January 2023 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING BACHELOR THESIS DESIGN, RESEARCH AND IMPLEMENTATION OF BEARING SEALS DEFECTS DETECTION AND CLASSIFICATION SYSTEM USING IMAGE PROCESSING AND DEEP LEARNING HOANG PHI LONG Student ID: 18146040 TRAN VIET THANG Student ID: 18146062 Major: MECHATRONICS ENGINEERING Advisor: M.E NGUYEN MINH TRIET Ho Chi Minh City, January 2023 THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, Day…….Month…….,Year…… GRADUATION PROJECT ASSIGNMENT Student name: Hoang Phi Long Student ID: 18146040 Student name: Tran Viet Thang Student ID: 18146062 Major: Mechatronics Engineering Class: 18146CLA Advisor: M.E Nguyen Minh Triet Phone number: 0996223249 Date of assignment: _ Date of submission: _ Project title: Design, Research and Implementation of bearing seals defects detection and classification system using image processing and deep learning Initial materials provided by the advisor: _ Content of the project: _ Final product: CHAIR OF THE PROGRAM (Sign with full name) ADVISOR (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, Day…….Month…….,Year…… ADVISOR’S EVALUATION SHEET Student name: Hoang Phi Long Student ID: 18146040 Student name: Tran Viet Thang Student ID: 18146062 Major: Mechatronics Engineering Project title: Design, Research and Implementation of bearing seals defects detection and classification system using image processing and deep learning Advisor: M.E Nguyen Minh Triet EVALUATION Content of the project: Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, Day…….Month…….,Year…… ADVISOR (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, Day…….Month…….,Year…… PRE-DEFENSE EVALUATION SHEET Student name: Hoang Phi Long Student ID: 18146040 Student name: Tran Viet Thang Student ID: 18146062 Major: Mechatronics Engineering Class: 18146CLA Project title: Design, Research and Implementation of bearing seals defects detection and classification system using image processing and deep learning Advisor: M.E Nguyen Minh Triet Name of Reviewer: Th.S Vo Lam Chuong EVALUATION Content and workload of the project Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:…………….(in words: ) Ho Chi Minh City, Day…….Month…….,Year…… REVIEWER (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Hoang Phi Long Student ID: 18146040 Student name: Tran Viet Thang Student ID: 18146062 Major: Mechatronics Engineering Class: 18146CLA Project title: Design, Research and Implementation of bearing seals defects detection and classification system using image processing and deep learning Name of Defense Committee Member: EVALUATION Content and workload of the project Strengths: Weaknesses: Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, Day…….Month…….,Year…… COMMITTEE MEMBER (Sign with full name) ACKNOWLEDGEMENT The graduation thesis is a milestone that marks the foundation of a path to our success in life During the project, students can summarize what they have learned, practiced, and apply it to real world challenges Students can also strengthen their soft skills in cooperation, communication, planning workload, and self pressure against work and learn at the same time As a result, they will integrate better into their jobs, matching the standards of recruitment firms The success of this project thanks to the contributions of each member Moreover, we would like to express our gratitude to the teachers at the Department of Mechatronics, Mr Nguyen Minh Triet for guiding us during the research period through the meetings to discuss and overcome difficulties during project implementation, providing majority feedbacks and helpful advice We would like to thank the lecturers of the Faculty for High Quality Training at Ho Chi Minh City University of Technology and Education for their excellent knowledge transmission We received a lot of attentions and assistances from them in our way to define ourselves Many thanks to friends who supported each other during difficult times, provided constant encouragement, and celebrated each achievement Most importantly, deeply thank the parents for their love, unwavering, countless sacrifices to create opportunities for us to fulfill our developments While having our best effort, this project may have limitations so that we are appreciated to receive helpful feedbacks from lecturers and classmates to assist us to develop our professional knowledge and organize our thesis Finally, we wish the teachers to have good health, faith to assist the student's learning journey i ABSTRACT Throughout the years, machine vision technology has made numerous improvements in many aspect of lives, such as medical, service, environment, mechanics, electrics and so on,… Especially in combination of mechanics, electrics and visual processing which were and are having an enormous impact to the development of automotive industry In order to make application of machine vision, we would like to design and implement a bearing seals defect half-automotive detecting and classifying system Moreover, with an idea to reduce number of labors classify product physically and also increasing the precise in output of bearing seals production Following the previous mechatronics project, we were designing and implementing a system that would detect and classify bearing seals that have standard qualified or unqualified So, in this graduation thesis, we desired to improve all sides in mechanical, electrical and visual process Firstly, the quality of detection and classification that can handle more difficult samples by applying not only pure visual processing, which is used by previous project, but also applying machine learning or deep learning to be able to process tiny and precise errors or defects that happen on the bearing seal’s surfaces Secondly, we would like to improve mechanism by adding a mechanical packing structure which Finally, the electrical circuit would reuse the board that we made for the previous project due to the lack of time and property We hope that our project might be a help for school, university as a prototype project so that others generation can keep up and upgrade for their project ii TÓM TẮT Qua nhiều năm, cơng nghệ xử lý hình ảnh có nhiều cải tiến nhiều khía cạnh sống y tế, dịch vụ, mơi trường, khí, điện tử… Đặc biệt kết hợp khí, điện, chế biến hình ảnh vốn có tác động to lớn đến phát triển ngành công nghiệp ô tô Để áp dụng thị giác máy, chúng em muốn thiết kế thực hệ thống phát phân loại lỗi phốt bạc đạn Hơn nữa, với ý tưởng giảm số lao động phân loại sản phẩm cách vật lý tăng độ xác cơng nghiệp sản xuất phốt bạc đạn Sau dự án điện tử trước đó, chúng em thiết kế thực hệ thống để phát phân loại phốt bạc đạn đạt tiêu chuẩn không đủ tiêu chuẩn Trong luận án tốt nghiệp này, chúng em muốn cải thiện tất khía cạnh q trình khí, điện, điều khiển thị giác máy Thứ nhất, chất lượng phát phân loại xử lý mẫu khó khăn cách áp dụng khơng xử lý trực quan túy, mà áp dụng xử lí máy deep learning để xử lý lỗi nhỏ xác lỗi xảy bề mặt dấu mang Thứ hai, chúng em muốn cải thiện chế đóng gói cách thêm chế khí đóng gói Cuối cùng, mạch điện sử dụng lại bảng mà chúng em làm cho dự án trước thiếu thời gian tài sản Chúng em hy vọng dự án chúng em giúp ích cho trường học, trường đại học dự án nguyên mẫu để hệ khác theo kịp nâng cấp cho dự án họ iii - Real-time display of the operation of the defect detection system - The ability to control and manage the entire system through a user-friendly interface - Display of the processing results and a statistical table of classification results Figure 81 Design of GUI application 3.4 Implementation After designed the part, some part we converted it to 2D sketch or the dwg file type which needs for laser cutting We have cut and assembled one to two prototype to review and jot down if there are anything missing The final version is cut by laser with the clearance fit as we research for laser cutting is only +0.1 for details 74 Figure 82 Example of dwgfile converter view for laser cutting Figure 83 Parts view in laser cutting program There also some part that is needed to use 3D printed due to the complicated shaped of the parts or components Normal clearance fit for 3D printed is around 0.25 to 0.5 depending on the quality of the machine and the plastics which are used So that, after some tries on one particular 3D printer places, we will have to adjust clearance fit for +0.3 to have a better assemble Moreover, to make the shaft of printed 3D more tough and hardness, we increase the density of plastic up to 80% and 50% for other parts The part files converted to stl file for the printing program Ultimaker Cura to understand and adjust 75 76 Figure 84 3D printing file view in Cura There are many ways to create prototype but laser cutting and 3D printing are the two most common and flexible way for projects: Table 12 Product making methods Laser cutting machine 3D Printing Method Criteria Accuracy or synchronized Aesthetics Price Self-editing ability Time High High Flat, correct shape, and even Cheap No (base on design) Have unstable and not even corner, especially with shaft Expensive No (base on design) Less time and faster than 3D printing Take a lot of time if the parts are huge Due to the conditions and the requirements of the project We decide to choose these two methods only instead of using metal materials Below are some parts after laser cutting and 3D printing: 77 Figure 85 3D printed products Figure 86 Laser cutting products 78 As for the control circuit, the PCB design should also consider factors such as the routing of wires and trace width to prevent signal loss, ground planes for shielding, and placement of power supply components for efficient power management In addition, proper testing and debugging procedures should be in place to ensure the circuit is functioning as intended before final assembly Figure 87 PCB top view (Left) and bottom view (Right) Figure 88 PCB with headers, components and electric box 79 Figure 89 Final Prototype after assembly and attach Components, quantity, price for each component and total cost to complete our system are list below: Table 13 List of Price of Components COMPONENTS AND PARTS QUANTITY UNIT PRICE Mica (Laser cutting) 3D Printing Sensors Bevel Gears Clampings, Couplings And Flanges High Power LED Light Diffuser Screws, Bearing, Shaft, Wires PCB Electrical components Conveyor Camera 11 - TOTAL PRICE (VNĐ) 120.000 12 2 35.000 78.000 - 500.000 70.000 156.000 100.000 30.000 60.000 - 60.000 - 60.000 200.000 200.000 - 200.000 400.000 1 690.000 500.000 690.000 500.000 3.056.000 The table above does not include extra fee such as delivery fees and others that cost to make this system 80 CHAPTER 4: RESULT EVALUATION 4.1 Mechanical inspection Meet the requirements of the original design in terms of durability, size, and tolerance Bevel gear control powertrain meets the correct control position and the direct drive also work well However, the prototype needs to redo one to two times, due to some extra review and discover some missing factors 4.2 Electrical system, signal inspection The boards operate stably, without errors in the process of use and meet electrical safety requirements The power supply stable and flow through bucking circuit also work correctly and maintain the operation for the system However, the reason could be come from the largeness of the wired connect between power source and lighting high watt LED that cause the hot temperature of the light 4.3 Control application inspection Figure 90 Application interface image The interface is highly designed in a user-friendly and symmetric way which has some light basic color as it is an engineering application but still perform well at informing, interacting, processing and responding This application is directly used on host PC with a connection via UART protocol 81 and UART-to-USB model CP2102 Moreover, the application interface has some basic setting such as Camera Setting for choosing which camera that user want to use, Serial Port setting for choosing a correct Port which connects to CP2102, user can also set the compatible baud rate that use for the microcontroller Then we have Initialize System button which is necessary to initialize the home position also the first set up for the whole system In the below left corner of the application, user can start the process by clicking run or stop button Then the system will automatically run to classify and pack The interface also has some statistic number with help user easier to track the system, quality of the system and also the quality of bearing seals 4.4 Operation inspection To evaluation the system, the first test is checking the synchronization of the classification and packaging structure, we test whether the classification structure correctly transfers passed bearing seal to the correct position in the pack when receiving the classification command This is important because we use a sliding classify structure, which can be heavily influenced by the shape of the bearing seal and the initial velocity when falling into the classification structure Furthermore, we have programmed the number of bearing seals on each pack to be 6, when the quantity is enough, the packaging structure will rotate to a new position, which may cause inaccuracies when bearing seals fall We optimize the setup of a continuous test with all bearing seals being passed in to test the stability The number of samples executed is 30 times Table 14 Testing Results Result Result Result 1st 2nd 3rd 4th 5th 6th 7th 8th 9th 10th ✓ ✓ ✓ ✓ ✓ ✓  ✓  ✓ 11st 12nd 13rd 14th 15th 16th 17th 18th 19th 20th ✓ ✓ ✓ ✓  ✓ ✓ ✓  ✓ 21st 22nd 23rd 24th 25th 26th 27th 28th 29th 30th ✓ ✓  ✓ ✓ ✓ ✓    From the above result table, we can indicate that the classification and packaging structure is temporarily synchronized, however there are still quite a few times bearing seals fall out of the pack After 30 continuous tests, the error rate was around 27%, which can be seen mainly after the packaging structure rotates to a new position (at the 7th and 19th time) Additionally, at the 28th time, when the bearing seal did not fully fall into the pack, causing the bearing seals behind it to not be able to fall into the pack, causing a series of errors The second test is designed to verify the stability of the image processing section This is a crucial part of the evaluation, as if the machine vision system misclassifies, the system will not function properly Therefore, we decided to test on a dataset of 485 images to assess stability The accuracy was approximately 81% 82 Table 15 Confusion Matrix N = 485 Positive Negative - Predicted Positive 113 26 139 Predicted Negative 67 279 346 - 180 305 - - Accuracy: (113+279)/485 = 80,8% - True Positive Rate: 113/180 = 62.8% - False Negative Rate: 67/180 = 37.2% - False Positive rate: 26/305 = 8.5% - True Negative rate = 279/305 = 91.5% The results show that the image processing method works relatively well Although the error rate is still high, the False Positive rate is relatively low, indicating that the system is less likely to classify the wrong bearing seals into the pack Factors affecting the results can be mentioned such as: unstable lighting provided, non-totally flat conveyor surface, light reflecting on bearing seal and conveyor, the processing method depends heavily on colors and brightness 83 CHAPTER 5: CONCLUSION AND RECOMMMENDATIONS 5.1 Achievements After finishing the project, we can apply knowledge learned to improve mechanical design, design printed circuit boards, use communication protocols like UART, define control algorithms, and control applications We completed the most difficult steps in designing and displaying the system of classification and packaging bearing seals The implementation of the topic has systematically learned a lot of knowledge, from the background in mechanics, electronics, and programming Apply knowledge of electronics to design printed circuit boards, and programming knowledge to control the velocity and angle rotation of a stepper motor, how to calculate the angle for gravity to win the friction force In addition, the topic also applies the basic knowledge of C and Python programming language and also learned how to use Raspberry Pi Pico function and library From the above results, the project has successfully designed and built a system for classification and packaging bearing seals with high finishing ability, stability, and support for expanding the project model depending on the requirements 5.2 Limitation and future work 5.2.1 Limitation With research efforts, but because there is only a limited amount of time, the ability to complete the project will have some unexpected points Our group has identified the limitations of the topic as follows: • For the structure: We does not figure out a way to attach the two module packaging and classification to have a fix position Meanwhile, they still work consistently and synchronized • For the electronics: Sometimes, the power supply may have been used for too many device that when powered for the lighting box It’s still reached a hot temperature and also the current is not stable Due to the adapter is not stable as well which is a point that we need to improve on choosing component 5.2.2 Future work With the above shortcomings and limitations will be the future development orientation To troubleshoot and fix the problem, we would like to offer some solutions for the project as follows: • For the structure: We would like to improve an attachment of the classifying part and the packaging part Moreover, we want to recreate a mechanism of classifying which can slide smoother, reduce the friction more, such as using metal sheet or object • For the electronics: Research for a circuit that used for high watt lighting to not let the light reach a too hot temperature 84 REFERENCES [1] “Đặc điểm phớt chắn dầu” https://tdktech.vn/dac-diem-ky-thuat-cua-phot-chan-dau-sb-sa-sc-tc-tb-ta [2] “ Direct-drive mechanism” Available: Direct-drive mechanism - Wikipedia, 26 April 2022 [3] “C Introduction” Available: C Tutorial (w3schools.com) [4] Learning OpenCV3: Computer Vision in C++ with the OpenCV Library, Dr Adrian Kaehler & Gray Bradski Published by O’Reilly Media, Inc., 1005 Gravenstein Highway North, Sebastopol, CA 95472 [5] "Motor Sizing Calculations" [Online] Available: https://www.google.com/url?sa=t&rct=j&q=&esrc=s&source=web&cd=&cad=rja &uact=8&ved=2ahUKEwjog7wvfL4AhUJgGMGHdidBPkQFnoECA4QAQ&url=https%3A%2F%2Fwww.orie ntalmotor.com%2Fproducts%2Fpdfs%2FF_TecRef%2FTecMtrSiz.pdf&usg=AOv Vaw1jc_9w3IpLRs3D4AD4OBwq Trịnh Chất - Lê Văn Uyển Nhà xuất giáo dục 2005, "Tính tốn Thiết kế hệ dẫn động khí tập I," in 2.3.3, p 22 [6] [7] Friction calculator" [Online] Available: Friction - Friction Coefficients and Calculator (engineeringtoolbox.com) [8] "Changzhou Minsheng Electronics Co ,Ltd," [Online] Available: https://tutorials-raspberrypi.de/wp-content/uploads/2014/08/Stepper-Motor28BJY-48-Datasheet.pdf [9] “Mosiac Documentation Web” Stepper Motor Specification http://www.mosaic-industries.com/embedded-systems/microcontrollerprojects/stepper-motors/specifications “Determining the Coefficient of Friction - Succeed in Physical Science” by Ron Kurtus [10] https://nfsi.org/wp-content/uploads/2013/10/Determining.pdf [11] “Material Contact Properties Table” http://atc.sjf.stuba.sk/files/mechanika_vms_ADAMS/Contact_Table.pdf 85 [12] “LJ12A3-4-Z Datasheet, Equivalent, PROXIMITY SENSOR” https://datasheetspdf.com/pdf/1096182/ETT/LJ12A3-4-Z/1 [13] Raspberry Pi Pico wiki https://www.waveshare.com/wiki/Raspberry_Pi_Pico [14] DRV8825, 45-V, 2.5-A bipolar stepper motor driver with current regulation and 1/32 micro stepping , Texas Instruments https://m5.gs/QWllaE [15] https://www.omnicalculator.com/statistics/confusion-matrix [16] “Data Augmentation” TensorFlow Tutorial https://www.tensorflow.org/tutorials/images/data_augmentation?fbclid=IwAR1za p3uFjDdC3J6Az5RKtOrS3mZCeiwdGfMCPjesoTSBfe9WmvMLqVYp_E Raspberry Pico: Programming with PIO State Machines https://medium.com/geekculture/raspberry-pico-programming-with-pio-state-machinese4610e6b0f29 [17] [18] https://raspberrypi.github.io/pico-sdkdoxygen/?fbclid=IwAR3_M0QinkfEyC3nYcqilO90kLnPvu6P0D49qLdEmv1NOqei5D xCYI_jedY 86 APPENDIX 28BYJ-48 (12V) dimensions Figure 91 Stepper motor dimensions NEMA 17 dimension Figure 92 NEMA 17 stepper motor dimensions Link to Project Code https://github.com/thangktv2/GraduationThesis?fbclid=IwAR3BCxVpAXOyXwR7lCo Blc_EGi9_NThnMno9uFM91Z35-vvMNlufCYOa2aE Mechanical Drawings i S K L 0