Design and implementation of a baby monitoring system

MINISTRY OF EDUCATION AND TRAINING HO CHI MINH CITY UNIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING GRADUATION PROJECT COMPUTER ENGINGEERING TECHNOLOGY DESIGN AND IMPLEMENTATION OF A BABY MONITORING SYSTEM ADVISOR: TRƯƠNG NGỌC SƠN, Assoc Prof STUDENT: LE QUANG TRUNG MAI DUONG QUYEN SKL 0 Ho Chi Minh City, December, 2022 HO CHI MINH CITY UNVIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING  GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF A BABY MONITORING SYSTEM MAJOR: COMPUTER ENGINGEERING TECHNOLOGY Student name Lê Quang Trung ID Student 18119048 Student name Mai Dương Quyền ID Student 18119039 ADVISOR: Trương Ngọc Sơn, Assoc Prof HO CHI MINH CITY - 12/2022 Faculty for High Quality Training – HCMC University of Technology and Education HO CHI MINH CITY UNVIVERSITY OF TECHNOLOGY AND EDUCATION FACULTY FOR HIGH QUALITY TRAINING  GRADUATION PROJECT DESIGN AND IMPLEMENTATION OF A BABY MONITORING SYSTEM MAJOR: COMPUTER ENGINGEERING TECHNOLOGY Student name Lê Quang Trung ID Student 18119048 Student name Mai Dương Quyền ID Student 18119039 ADVISOR: Trương Ngọc Sơn, Assoc Prof HO CHI MINH CITY - 12/2022 ii Faculty for High Quality Training – HCMC University of Technology and Education THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, December 24, 2022 GRADUATION PROJECT ASSIGNMENT Student name: Mai Dương Quyền Student ID: 18119039 Student name: Lê Quang Trung Major: Computer Engineering Technology Student ID: 18119048 Class: 18119CLA Advisor: Trương Ngọc Sơn Phone number: 0837975783 Date of assignment: 1/10/2022 Date of submission: 25/10/2022 Project title: Design and implementation of a baby monitoring system Initial materials provided by the advisor: none Content of the project: - Analyzing and researching data related to behavioral behavior in young baby during sleep - Design and implement the LSTM network model in accordance with the monitoring system - Selective skeletal point computation using the Meadiapipe library and the EAR - Creating a telegram bot to send alerts to parents - Implement and test the system on Jetson Nano Final product: Implement a baby monitoring system through actions and behaviors during sleep The monitor system's primary operating environment is homes with infants older than six months The final product is tested for system performance in monitoring on Jetson Nano The results were transmitted for notification by telegram and spoken announcement by loudspeaker CHAIR OF THE PROGRAM ADVISOR (Sign with full name) (Sign with full name) iii Faculty for High Quality Training – HCMC University of Technology and Education THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, December 24, 2022 ADVISOR’S EVALUATION SHEET Student name: Mai Dương Quyền Student ID: 18119039 Student name: Lê Quang Trung Student ID: 18119048 Major: Computer Engineering Technology Project title: Design and implementation of a baby monitoring system Advisor: Trương Ngọc Sơn EVALUATION Content of the project: - Analyzing and researching data related to behavioral behavior in young baby during sleep - Design and implement the LSTM network model in accordance with the monitoring system - Selective skeletal point computation using the Meadiapipe library and the EAR - Creating a telegram bot to send alerts to parents - Implement and test the system on Jetson Nano Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, December 24, 2022 ADVISOR iv Faculty for High Quality Training – HCMC University of Technology and Education (Sign with full name) THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, January 1st, 2023 PRE-DEFENSE EVALUATION SHEET Student name: Mai Dương Quyền Student ID: 18119039 Student name: Lê Quang Trung Student ID: 18119040 Major: Computer Engineering Project title: Design and implementation of a baby monitoring system Name of Reviewer: Pham Van Khoa EVALUATION Content of the project: - Design and implementation of a baby monitoring system Strengths: - Apply artificial intelligence to the proposed product Weaknesses: - Realistic factors not be considered in this report Approval for oral defense? (Approved or denied) - Approved Comments and suggestions - Please refer the existing products and provide the proposed specification in detail Ho Chi Minh City, January 1st, 2023 REVIEWER (Sign with full name) Pham Van Khoa v Faculty for High Quality Training – HCMC University of Technology and Education HO CHI MINH CITY OF UNIVERSITY OF SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom – Happiness TECHNOLOGY AND EDUCATION FACULTY OF HIGH QUALITY TRAINING Ho Chi Minh City, January 10, 2023 MODIFYING EXPLANATION OF THE GRADUATION PROJECT MAJOR: COMPUTER TECHNOLOGY ENGINEERING Project title: Student name: Mai Dương Quyền ID: 18119039 Student name: Lê Quang Trung ID: 18119048 Advisor: Defending council: Council 2, Room: A3-404, 3rd January 2023 Modifying explaination of the graduation project: No Council comments Editing results Refer to the existing products and provide the proposed specification in detail Completed additional Note modifications to existing products and provided detailed recommended specifications in table 4.3 page 41 Head of department Advisor Students (Sign with full name) (Sign with full name) (Sign with full name) vi Faculty for High Quality Training – HCMC University of Technology and Education THE SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom– Happiness -Ho Chi Minh City, ., 2023 EVALUATION SHEET OF DEFENSE COMMITTEE MEMBER Student name: Mai Dương Quyền Student ID: 18119039 Student name: Lê Quang Trung Student ID: 18119040 Major: Computer Engineering Project title: Design and implementation of a baby monitoring system Advisor: Trương Ngọc Sơn EVALUATION Content of the project: Strengths: Weaknesses: Approval for oral defense? (Approved or denied) Overall evaluation: (Excellent, Good, Fair, Poor) Mark:……………….(in words: ) Ho Chi Minh City, …………… , 2023 COMMITTEE MEMBER (Sign with full name) vii Faculty for High Quality Training – HCMC University of Technology and Education DISCLAIMER Implementation group thus formally proclaim that the research and application that went into this thesis Without citing our study as the source, no published piece has been copied Any infractions that may have happened are fully our fault Students Le Quang Trung Mai Duong Quyen viii Faculty for High Quality Training – HCMC University of Technology and Education ACKNOWLEDGMENTS During the project implementation, implementation team received a lot of positive comments and support to be able to complete the project completely and successfully First of all, implementation team would like to express our sincere thanks to the team of teachers of HO CHI MINH University of Technical Education and the Department of High Quality Training for the graduation thesis Based on the knowledge throughout the four years of study, implementation group has been able to apply and orient themselves to their projects, thereby through the project to improve their understanding and help us determine our own direction to be more confident in other projects in the future In addition, implementation team would like to express our deep thanks to PhD Truong Ngoc Son who has oriented and dedicatedly supported the team during the implementation and completion of the project Implementation group also wants to express its gratitude to the 18119CLA students for their support, advise, and encouragement During the implementation of the project, the team has gained more knowledge from teachers and textbooks and reference materials However, due to the limited level of expertise and experience, the team could not avoid the shortcomings Implementation group expects to receive attention and input from teachers so that implementation group can improve our project better Finally, implementation team would like to wish all teachers of the Faculty of Electrical and Electronics Science and the Faculty of High Quality Training at Ho Chi Minh City University of Pedagogy and Engineering together with all students of the faculty a lot of health and a lot of success Sincerely! STUDENTS ix Faculty for High Quality Training – HCMC University of Technology and Education Figure 3.13: Flowchart of the training algorithm 35 Faculty for High Quality Training – HCMC University of Technology and Education 3.2.3.3 Flowchart of detection algorithm As mentioned above, the sleeping baby monitoring model has three main features: baby wake up detection, baby movement detection, baby outside detection The figure below is a flowchart of the wake-up detection algorithm based on the LSTM network model After much testing and finally the team decided to choose a time step of 30 for this function The system always calculates the EAR and after every 30 frames, the obtained output is compared with the trained weight file to give the prediction result 36 Faculty for High Quality Training – HCMC University of Technology and Education Figure 3.14: Flowchart of the algorithm to detect the baby waking up Similar to wake-up baby detection trained by LSTM network The system collects the coordinates of 33 points on the body to compare with the trained weight file to make predictions The baby motion detection flow chart is shown in Figure 3.15 below 37 Faculty for High Quality Training – HCMC University of Technology and Education Figure 3.15: Flowchart of algorithm to detect moving baby The final function of the model is to detect the baby outside shown in the Figure 3.16 below The system draws a polygon and it's like a baby's boundary line The implementation team selected points on the body namely: Nose, Left/Right Hand, 38 Faculty for High Quality Training – HCMC University of Technology and Education Left/Right Foot to use detection for this function Just at least point outside the polygon, the system emits the message “OUTSIDE” Figure 3.16: Flowchart of algorithm to detect baby outside 39 Faculty for High Quality Training – HCMC University of Technology and Education CHAPTER RESULTS AND DISCUSSIONS In this chapter, creating the system execution on the hardware, displaying the execution results, and building a system model 4.1 Results of the practical model The model of the sleeping baby monitoring system is depicted in the figure below The system includes the input device is camera, keyboard, mouse, output device is speaker, monitor, and the brain of the system is the Jetson Nano embedded computer Figure 4.1: Image depict the actual model 4.2 System results and evaluation 4.2.1 Results In order to have the most objective results and evaluate the performance of the system, the research team conducted a practical model test on functions of the system: "Detecting the baby waking up", "" Detect the baby moving", "Detect the baby out" However, the team did not have a baby for the real-time checks, so the idea was put forward by the team that a team member would the baby's duty during these checks Because this model mainly bases on the object's activity to make the final prediction, so replacing the baby with an adult does not affect the final prediction result The model's “wake-up detection” relies on EAR calculations to make predictions, shown in figures (a) and (b) below Figure (c) (d) depicts the predicted output of “Moving baby detection” The last one is “Detecting a baby out” depicted in figure (e) (f) 40 Faculty for High Quality Training – HCMC University of Technology and Education (a) (b) (c) (d) (e) (f) 41 Faculty for High Quality Training – HCMC University of Technology and Education Figure 4.2: The test result on three case (a-b) wake up, (c-d) moving , and (e-f) outside The system will notify the parent's phone when one of the three cases is identified and report the baby's status As a result, it is not necessary for caretakers to constantly watch over the baby Figure 4.3: The results of the notification are to be sent to the user's phone through the Telegram app 4.2.2 System evaluation Although using parallel LSTM networks at the same time, it ensures the simple, compact and scalable architecture of the system, because the LSTM network is a lightweight network However, the system still has some false positives because the trained data series is still not optimized Baby wake-up detection achieved 94.5 accuracy after 200 times of real-time testing The label "Sleep" has no error because when sleeping, the ratio of the eyes is almost zero, so the system easily detects and makes the main prediction The label "Wake up" has errors because the model is still noisy with "blink" Detecting baby moving has a fairly high accuracy of 93.5%, but there is still an error because the baby's limbs not move according to a certain rule Another subjective cause is that the data used for training is not optimized The table below describes the accuracy of the system in detecting the baby waking up and detecting the baby moving, respectively 42 Faculty for High Quality Training – HCMC University of Technology and Education Table 4.1: The table describes the accuracy of "Baby wake-up detection" True Label Wake Up Sleeping 89 100 Accuracy Predict Wake Up 94.5% Sleeping 11 Table 4.2: The table describes the accuracy of "Moving baby detection" True Label Moving No Moving Accuracy Predict Moving 94 93 93.5% No Moving To have a more objective view on the topic, below is a comparison table with scientific articles "Sleeping baby monitoring system" has been made Table 4.3: Comparison table with previous models [15] Work Motorola Infant Optics [10] Nanit [12] Lollipop Cubo Ai [13] [11] Our Implement [14] Live Video Yes Yes Yes Yes Yes Yes Boundary No No No Yes Yes Yes Cross Detection No No No Yes Yes No Cry detection No No Yes No No No Breathing No Monitoring No No No Yes No 43 Faculty for High Quality Training – HCMC University of Technology and Education Frequent Moving Detection No No Yes No No Yes Awake Detection from Eye No No No No No Yes The DXR Video Baby Monitor, an award-winning non-Wi-Fi baby monitor, is the first product [11] to use interchangeable lens technology The security camera's built-in zoom, normal, and wide-angle lenses can all be switched out using interchangeable lens technology, enabling users to clearly watch both day and night In addition, the system also incorporates two-way audio reception capabilities With a price tag that is $165.99 less than other baby monitors now on the market, this could be one of the DXR 8's advantages over competing products The technology has not been integrated very much in recognizing and monitoring baby's activities and behaviors, therefore there are still numerous restrictions in DXR 8, which solely monitors baby through Live Video In the device [12] - Nanit, one of the most well-liked monitoring systems in several nations nowadays, numerous capabilities are integrated to monitor baby's sleep One of the major benefits of the system is the monitoring of baby's breathing movements using the Breathing Band and the capability to precisely measure the baby's height The system always reaches 1080p HD video quality Additionally, Nanit is made available to users of the Nanit App, which was created and integrated concurrently with the monitoring system Using the application, parents may immediately examine the baby's sleep condition in realtime, in addition to other features The technology automatically monitors and provides guidance on when youngsters should go to sleep Finally, Breathing Wear offers consumers the chance to view their baby's breathing rate per minute in real time, which is another notable benefit over the system developed by implementation team The suggested retail price for the item is presently $322, which includes the equipment bundle (Nanit Pro Camera, Choice of Wall Mount, Travel Pack with Flex Stand and Case, One Breathing Band) Concerning the device [13], Lollipop is one of the analytical systems with a wealth of functions for monitoring and multi-action assistance in infants But unlike the gadget [12], here the system concentrates on monitoring and detecting the baby's behavior through the baby's crying and can separate it from other ambient noises like the sound open/closing door, or television Additionally, the system has the ability to detection of baby crossing like the concept used by the implementation team to identify baby kept outside The system 44 Faculty for High Quality Training – HCMC University of Technology and Education can be detected who is noise around the affected baby, and if there is a loud noise nearby, the system will send an alarm via app to parents The use of a smart camera that can monitor circumstances even in low-light areas is one of the device's notable benefits over other products now on the market Auto change the monitoring settings (morning to evening and vice versa) In addition, the system has a Lolipop Intelligent Air Quality Sensor, which measures temperature, humidity, and air quality all at once and outputs parameter findings via daily data logging charts The Lollipop Smart Baby Camera will cost $169, and the Lollipop Smart Air Quality Sensor will cost $55 As can be seen, the implementation group completed out of proposed items This is right line with the original goal of the implementation team was to focus on the field of “Computer Vision” In addition, the implementation team has further implemented and optimized the “Awake Detection from Eye” by training with the lightweight LSTM network 45 Faculty for High Quality Training – HCMC University of Technology and Education CHAPTER WORK CONCLUSION AND FURTER In last chapter, provides the outcomes obtained following the completion of the entire system, providing guidance for future system development and application expansion 5.1 Conclusion After a period of research, learn with the help of instructor Truong Ngoc Son Assoc Prof, project "Design and implementation of a baby monitoring system" that the team has completed and well met the requirements set forth The team has designed a sleeping baby monitoring system using parallel LSTM networks Function blocks work properly, ensuring correct data updates Objectively evaluating the image processing speed, the detection accuracy ratio of the system with the original goal achieved quite good results However, with the actual model that needs faster detection with higher accuracy, the Jetson Nano has not met the demand To meet the real demand of fast processing speed, the Jetson Nano kit central processor is not yet powerful enough to solve the problem of threaded processing speed typical of LSTM networks The team's system works best when the faces and bodies must be facing the camera position, the light is moderate, not bright or underexposed Because of the limited time to implement the project, as well as the cost limit, the implementation team has not been able to solve all the problems that arise or make further improvements so that the system can be more efficient and meet the needs of the people more realistic needs, but only stop at building a compact, simple model that solves the basic goal set out 5.2 Future work In the future, it is necessary to continue to improve and develop to improve the quality of the system both in terms of speed and accuracy The next development direction of the project is: Build and develop better hardware model with faster processing speed and more stability when the system has to complex calculations Combine more sensors, NLP, IOT to develop more tasks such as: "Detecting crying babies", "Breathing Monitoring" - Expand the data set of baby movements such as lying face down, crawling, rolling 46 Faculty for High Quality Training – HCMC University of Technology and Education REFERENCE [1] Paul J Werbos Backpropagation through time: What it does and how to it Proc of the IEEE, 78(10):1550–1560, 1990 [2] Ronald J Williams and David Zipser A learning algorithm for continually running fully recurrent neural networks Neural Computation, 1(2):270– 280, jun 1989 [3] Yong Yung, Xiaosheng Si, Changhua Hu, and Jia A Review of Recurrent Neraul Networks: LSTM Cells and Network Architecturecs, vol 31, isuue 7, July 2019 [4] Felix A Gers, Nicol N Schraudolph, and Jăurgen Schmidhuber Learning precise timing with LSTM recurrent networks Journal of Machine Learning Research (JMLR), 3(1):115–143, 2002 [5] Qi Lyu and Jun Zhu Revisit long short-term memory: an optimization perspective In Deep Learning and Representation Learning Workshop (NIPS 2014), pages 1–9, 2014 [6] R Szeliski Computer Vision: Algorithms and Applications Springer 2011 [7] R Laganière OpenCV Computer Vision Application Programming Cookbook Packt Publishing 2011 [8] Camillo Lugaresi, Jiuqiang Tang, Hadon Nash, Chris McClanahan, Esha Uboweja, Michael Hays, Fan Zhang, Chuo-Ling Chang, Ming Guang Yong, Juhyun Lee, Wan-Teh Chang, Wei Hua, Manfred Georg and Matthias Grundmann Google Research MediaPipe: A Framework for Building Perception Pipelines, Jun 2019 [9] S Hochreiter and J Schmidhuber, "Long Short-Term Memory," in Neural Computation, vol 9, no 8, pp 1735-1780, 15 Nov 1997, doi: 10.1162/neco.1997.9.8.1735 [10] MOTOROLA MBP36XL Baby Monitor Available online: https://www.motorola.com/us/motorola-mbp36xl-2-5-portablevideo-baby-monitor with2-cameras/p (accessed on June 2021) [11] Infant Optics DXR-8 Video Baby Monitor Available https://www.infantoptics.com/dxr-8/ (accessed on November 2022) online: [12] Nanit Pro Smart Baby Monitor Available online: https://www.nanit.com/products/nanit-pro-complete-monitoring-system?mount=wallmount (accessed on November 2022) [13] Lollipop Baby Monitor with True Crying Detection Available online: https://www.lollipop.camera/ (accessed on November 2022) 47 Faculty for High Quality Training – HCMC University of Technology and Education [14] Lienhart, R.; Maydt, J An extended set of Haar-like features for rapid object detection In Proceedings of the International Conference on Image Processing, Rochester, NY, USA, 22–25 September 2002 [15] Khan, T An Intelligent Baby Monitor with Automatic Sleeping Posture Detection and Notification AI 2021, 2, 290-306 https://doi.org/10.3390/ai2020018 [16] M -T Duong, T -D Do, M C Le, V -B Nguyen and M -H Le, "An Efficient Data Collecting Method for Enhanced Real-Time Drowsiness Detection Systems," 2021 International Conference on System Science and Engineering (ICSSE), 2021, pp 105-110, doi: 10.1109/ICSSE52999.2021.9538480 [17] Akihiro Kuwahara, Kazu Nishikawa, “Eye fatigue estimation using blink detection based on Eye Aspect Ratio Mapping” , doi: https://doi.org/10.1016/j.cogr.2022.01.003 [18] M Ramzan, H U Khan, S M Awan, A Ismail, M Ilyas and A Mahmood, "A Survey on State-of-the-Art Drowsiness Detection Techniques," in IEEE Access, vol.7, pp 61904-61919, 2019, doi: 10.1109/ACCESS.2019.2914373 48 S K L 0