(Đồ án hcmute) research and implement a model of mango softing machine with image processing applications

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION THESIS MECHATRONICS ENGINEERING TECHNOLOGY RESEARCH AND IMPLEMENT A MODEL OF MANGO SORTING MACHINE WITH IMAGE PROCESSING APPLICATIONS ADVISOR: PhD DANG TRI DUNG STUDENT: TRAN CHI THANH PHAM MINH HUNG SKL Ho Chi Minh City, Febuary, 2023 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING MAJOR OF MECHATRONICS ENGINEERING TECHNOLOGY GRADUATION THESIS RESEARCH AND IMPLEMENT A MODEL OF MANGO SORTING MACHINE WITH IMAGE PROCESSING APPLICATIONS ADVISOR: PhD DANG TRI DUNG STUDENTS: TRAN CHI THANH PHAM MINH HUNG  HO CHI MINH CITY – 02/2023  18146060 18146025 HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING Capstone Project Proposal Student name: Tran Chi Thanh Student ID: 18146060 : 0938 669 029 Student name: Pham Minh Hung Student ID: 18146025 : 033 376 4601 Advisor: : 0967 360 960 PhD Dang Tri Dung Title of Capstone Project: RESEARCH AND IMPLEMENT A MODEL OF MANGO SORTING MACHINE WITH IMAGE PROCESSING APPLICATIONS Initial materials: • • • • Calculation of data, design and construction of hardware Calculation and selection of suitable engines Control software design Apply the image processing for classification Main Content: The color of mangoes is based on Vietnamese national standard 9766:2013 from which we apply an image processing algorithm to be able to classify according to Expected Results • Model of classification of mango • Project report Delivery date: Submission date: Language: Dean Report: Oral presentation: English English Head of Department • • Vietnamese • Vietnamese • Advisor THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness Ho Chi Minh City, January, 2023 ADVISOR’S EVALUATION SHEET Student name: Student ID: Student name: Student ID: Major: Project title: Advisor: EVALUATION Content of the project: Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………/100(in words: ) Ho Chi Minh City, January, 2023 ADVISOR (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness Ho Chi Minh City, January, 2023 PRE-DEFENSE EVALUATION SHEET Student name: Student ID: Student name: Student ID: Major: Project title: Reviewer: EVALUATION Content and workload of the project Strengths: Weaknesses: Questions: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:…………/100(in words: ) Ho Chi Minh City, January, 2023 REVIEWER (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness Ho Chi Minh City, January, 2023 EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Student ID: Student name: Student ID: Major: Project title: Name of Defense Committee Member: EVALUATION Content and workload of the project Strengths: Weaknesses: Overall evaluation: (Excellent, Good, Fair, Poor) Mark:…………/100(in words: ) Ho Chi Minh City, January, 2023 COMMITTEE MEMBER (Sign with full name) PREFACE Students will use their knowledge of design, mechanics, computation, electrical and electronic circuits, and programming techniques to research and create models for their graduation project As a result, this is the point at which students have a more realistic view of the knowledge they have gained and can put it to use in the creation of products The group's assigned topic is to create a model for categorizing mangoes using camera operates on a miniature conveyor Our understanding of machine detail design is still limited due to our limited experience, so we have done our best to refer to relevant documents and lectures, but errors are unavoidable in design and construction calculations, design, etc Looking forward to the guidance and dedicated guidance of teachers to help students like us best complete this graduation project Our students, Pham Minh Hung and Tran Chi Thanh, are certain that this is a project that we will complete on our own with the assistance of our instructors and other dedicated teachers Team members rely solely on prior literature and online research, and we hereby declare that all figures are accurate and have not been plagiarized from any previous material or work Ho Chi Minh City, 02/2023 Student implementation Tran Chi Thanh Pham Minh Hung ACKNOWLEDGEMENTS The project team would like to especially thank PhD Dang Tri Dung was the one who guided and helped the group during the implementation of the project, gave research directions, answered questions and enthusiastically stood by the group During the implementation of the group, the teachers have absorbed practical knowledge and serious and effective working methods The group would like to thank the teachers of Mechatronics Department for their support students with solid basic knowledge and create the best conditions for students in the learning and research process The most important, unspoken support of family and friends is always the motivation for the whole team to work hard and complete the project in the best way possible Once again, the team is very honored to be a student of Ho Chi Minh City University of Technology and Education and a student of dedicated teachers, this thank you is also the team's deep gratitude to the team teachers, relatives and friends COMMENT OF THE ADVISOR …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… …………………………………………………………………………………… Ho Chi Minh City, 06/2022 Advisor PROJECT IMPLEMENTATION SCHEDULE Time Confirmed by Content Advisor Meet the lectures to listening about the request 10/09/2022 of topic to make project and approves topic which student has selected Student write out the project outline and schedule 11/09/2022 to Research the theoretical basis related to the 18/09/2022 topic 19/10/2022 to Design 02/10/2022 mechanical components, find construction solutions, draw of model 03/10/2022 to Circuit design, construction of electronic parts 16/10/2022 Mechanical and electronic connections, 17/10/2022 to hardware, testing electrical circuits, connection 06/11/2022 of sensor and camera modules with executive structure 07/11/2022 to Design software, conduct training model for 04/12/2022 image processing 05/12/2022 to Run the model with real mangoes, rework, 06/01/2023 adjust to real conditions, fix problems 07/01/2023 to Conduct report writing, video demo, prepare 30/01/2023 drawings as well as related documents Semi ripe mango Prediction value 0,997 0,996 0,997 0,998 0,998 10 0,997 Table 8: Prediction value of Semi ripe mango Unripe Prediction value 0,999 0,999 0,98 0,994 0,997 0,99 10 Table 9: Prediction value of Semi ripe mango 80 Figure 68: Prediction of 10 different ripe mango graph Figure 69: Prediction of 10 different semi ripe mango graph Figure 70: Prediction of 10 different Unripe mango graph 5) Compile and save model 81 4.5.5 Size of the mango and Experiment: Below are the step to measure the size (length and width) to make sure that between the size calculating and measure from the image processing should be same with measuring from the real size Step 1: Measuring the real size (length and weigth) of the mango In the fig ( & ), we measured the real length is 11cm and real width of the mango is 7cm Figure 71: Length of the mango (11cm) Figure 72: Width of the mango (7cm) 82 Step 2: Measuring the size (length and width) of the mango with image processing • By using the max function to find the contour with the largest area in the set of contours cnts The argument key=cv2.contourArea is a function that is used to extract the area of each contour The max function returns the contour with the largest area The result is stored in a variable named c • The line (x, y, w, h) = cv2.boundingRect(c) uses the boundingRect function from OpenCV to find the bounding rectangle for the contour c The function returns a tuple of four values (x, y, w, h), where x and y are the coordinates of the top-left corner of the rectangle, and w and h are the width and height of the rectangle, respectively • The result of the boundingRect function is unpacked and assigned to the variables x, y, w, and h, so that each variable holds the corresponding value from the tuple • The final result of the pipeline is the bounding rectangle for the contour with the largest area in the input set of contours The rectangle is represented by the four values x, y, w, and h, which define its position and size In this application we just only use the parameter below: h is length of the mango w is width of the mango Figure 73: Capture of code application length and width With the height between the camera and the surface of the mango is 35cm, we received the pixel value of width : 230 pixel and length: 435 pixel from image processin Figure 74: Pixel value of mango size 83 Figure 75: Bounding area of a mango Step 3: Calculate the size (length and width) of a mango with image processing: After we collect the value of mango size which measure from reality and image processing This first date will use for another case details in the table below: Camera Real Real Height Length Width [cm] [cm] [cm] 435 230 35 11 7,5 Table 10: Ripe Mango size data collection Length Width [Pixel] [Pixel] Camera Real Real Height Length Width [cm] [cm] [cm] 423 224 35 10,3 7,5 Table 11: Semi ripe Mango size data collection Length [Pixel] Width [Pixel] 84 Length [Pixel] Width [Pixel] Camera Height [cm] Real Length [cm] Real Width [cm] 402 198 35 10,4 7,3 Table 12: Unripe Mango size data collection How to calculate with the collected data above? From the table we can understand that at the height 35cm from camera: • Length of the mango is 11cm will equal to 435 pixel • Width of the mango is 7,5 cm will equal to 230 pixel So the length (11cm) and width (7.5cm) will the base to calculate for another case For example, in another mango the image processing return value : h = 405 pixel, w = 215 pixel Calculation ratio: 1) Length ratio: With 435 pixel = 11cm => x = (405*11)/435 = 10 cm 405 pixel = x cm 2) Width ratio With 230 pixel = 7.5 cm => y = (215*7,5)/230 = cm 215 pixel = y cm Finally, we want to make sure our calculation and process are correct We need to take experiment to verify if any size of mango can be measured correctly Proceed to measure size from 30 real mango of difference mango devide in group: 10 Ripe mango, 10 Semi ripe mango and 10 Unripe mango The value of h (length) and w (width) will return from image processing and comparing with real size 85 Table 13: Experiment in 10 ripe mango Table 14: Experiment in 10 semi ripe mango Table 15: Experiment in 10 unripe mango 86 Width error experiment in ripe mango 2,5 2,0 1,9 Width Error [%] 2,0 1,6 1,5 1,5 1,5 1,0 0,9 0,7 0,7 0,5 0,5 0,5 0,0 10 Number of mango Figure 76: Lenght error graph for ripe mango Length error experiment in ripe mango 2,5 2,1 Lenght Error [%] 2,0 1,8 1,6 1,5 1,4 1,4 1,2 1,2 1,0 0,8 0,8 0,5 0,1 0,0 10 Number of mango Figure 77: Lenght error graph forripe mango 87 Width error experiment in Semi ripe mango 3,0 2,5 Width Error [%] 2,5 2,0 1,9 2,0 1,9 2,2 2,2 10 2,0 1,4 1,5 1,2 1,0 0,7 0,5 0,0 Number of mango Figure 78: Width error graph for Semi ripe mango Length error experiment in ripe mango 3,0 2,7 2,7 2,5 Width Error [%] 2,5 2,0 1,9 2,0 1,5 1,4 1,4 1,4 1,2 0,9 1,0 0,5 0,0 10 Number of mango Figure 79: Length error graph for Semi ripe mango 88 Width error experiment in Unripe mango 2,5 2,4 2,1 2,1 1,9 Width Error [%] 2,0 1,9 1,8 1,6 1,5 1,2 1,0 1,0 0,9 0,5 0,0 Number of mango 10 Figure 80: Width error graph for unripe mango 3,0 Length error experiment in Unripe mango 2,5 2,5 2,3 Width Error [%] 2,0 1,5 1,4 1,5 1,5 1,4 1,1 0,9 1,0 0,5 0,3 0,0 0,0 10 Number of mango Figure 81: Length error graph for unripe mango 89 Show Result in monitor include tye of mango (Ripe mango, Semi ripe mango, Unripe mango) and record and show the lenght and width: Figure 82: Unripe Mango Results Figure 83: Ripe Mango Results 90 Figure 84: Semi Ripe Mango Results 91 CHAPTER 5: CONCLUSION AND FUTURE WORK 5.1 What we have done in this project: • Design a model with 3D with SolidWork • Processing the image of mangos fruit • Collecting data to retrain the mobilenet v2 model • Poster, video with details information 5.2 Difficulties: • The mobilenetV2 network required a large amount of data • The mobilenetV2 can be sensitive to overfitting, complexity and requiring a significant amount of computational resources • It is hard to interpret because of complex combination of weights and biases Challenging to understand why the model is making certain predictions 5.3 Future work: • Demo on real-time mode • Collect more data • Detect a variety of fruits • Calculate volume of the mangos • Classification of mangos based on Vietnam ISO standard • Add black box to reduce light noise • Add loadcell to determine the load of the mangos • Add control function in the interface display 92 REFERENCES [1] https://aiots.vn/phan-8-cac-khong-gian-mau-trong-opencv/ [2] https://maythietbivn.com/shop/thiet-bi-so-mau/he-mau-lab [3] https://adrecoplastics.co.uk/abs-plastic-properties/ [4] https://programmer.help/blogs/opencv-python-beginner-advanced-bit-operationsand- masks.html [5] http://vietnamtradeoffice.net/khai-quat-tinh-hinh-san-xuat-xoai-tren-the-gioi/ [6] https://www.tridge.com/intelligences/mango/import [7] http://hanoimoi.com.vn/tin-tuc/kinh-te/996759/viet-nam-la-quoc-gia-sanxuat-xoai-lon-thu-13-tren-the-gioi [8] https://vinbigdata.com/xu-ly-du-lieu-anh-mot-so-kien-thuc-can-ban/ [9] https://www.researchgate.net/figure/Applications-domains-for-embeddedsystems-in-the-Internet-of-Things_fig1_334646622 [10] https://www.researchgate.net/figure/Working-principle-diagram-of-beltconveyor_fig1_322215601 [11] https://vitutomedia.com/nguyen-ly-hoat-dong-cua-may-anh/ [12] https://uniduc.com/vi/blog/dong-co-servo-dc-la-gi-uu-nhuoc-diem-cuadong-co-servo-dc [13] https://phongvu.vn/may-quay-internet-facecam-1000x-genius s200400064 [14] https://dientutuonglai.com/gioi-thieu-arduino-nano.html [15] https://hshop.vn/products/cam-bien-vat-can-hong-ngoai-e18-d80nk-4 [16] https://www.thegioiic.com/gb37-545-dong-co-giam-toc-co-encoder-24vdc10-rpm-truc6mm?fbclid=IwAR27NVSv3Cp2vu4t0OuPQZ1PRlwiZKpXUL9SxX7Iev13Eu PeWzxQKccuYvM [17] https://icdayroi.com/servo-mg995 [18] https://www.thegioiic.com/1r5v-v2-module-1-relay-5v-kich-muc-cao [19] https://nshopvn.com/product/mach-giam-ap-dc-lm2596-3a/ [20] https://www.thegioiic.com/nguon-adapter-12v-5a-dau-dc-5-5x2-5mm-dai 93 S K L 0