final report title development of medical devices

Face Mesh when applied to Augmented Reality...15Figure 2.5 3D-Skeleton recognition of the hand...15Figure 2.6 Human Pose Tracking in 3D...16Figure 3.1 Video acquisition and processing sy

SCHOOL OF ELECTRICAL & ELECTRONIC ENGINEERING

ELECTRONICS DEPARTMENT

ADVANCED BIOMEDICAL ENGINEERING PROGRAM

FINAL REPORT

TITLE DEVELOPMENT OF MEDICAL DEVICES

A

HANOI, 2-2022

B

CHAPTER 1.STATE THE PROBLEM 1

1.1 Overview of human posture estimation 1

1.1.1 What is the human pose estimate? 1

1.1.2 The importance of estimating human posture 2

1.1.3 The estimate human pose in 2D and 3D 2

1.1.4 Models to estimate human posture 3

CHAPTER 2.THEORETICAL BASIS 4

2.1 OpenCV image processing library 4

3.2 Building processing algorithms 14

3.2.1 Put the video in Pycharm 14

3.2.2 Use Pose Estimation algorithm to determine the joint point 15

i

3.2.4 Output the results to Excel and compare graphically 17

3.2.4.1 Output the results to Excel 17

3.2.4.2 Graph comparison 18

3.3 The coding program for project 18

CHAPTER 4.EXPERIMENT AND RESULTS ASSESSMENT 25

4.1 Test of measuring method 25

4.2 Experiment results 27

ii

LIST OF FIGURES

Figure 1.1 Human body models 8

Figure 2.1 OpenCV library 11

Figure 2.2 Solutions offered and availability across platforms in MediaPipe 13

Figure 2.3 Face Detection on an Android device 14

Figure 2.4 Face Mesh when applied to Augmented Reality 15

Figure 2.5 3D-Skeleton recognition of the hand 15

Figure 2.6 Human Pose Tracking in 3D 16

Figure 3.1 Video acquisition and processing system 19

Figure 3.2 Diagram of processing algorithm 21

Figure 3.3 Detected motion on an image by Pose Landmark Model 23

Figure 4.1Camera setup for data collection 33

Figure 4.2 Joint points recognition 34

Figure 4.3 Line chart to compare joint points recognition in different resolutions 480P,720P and 1080P 35

iii

LIST OF TABLES

Table 3.1 20

iv

Recently, there are many different studies using solutions of OpenCV andMediapipe in the world such as Face Detection or Hands Detection and Face Mesh, etc.But in those studies, there has not been any study examining the impact of thesurrounding environment on the results of joint points detection of OpenCV andMediaPipe Therefore, in this research, we have studied the influence of the surroundingenvironment on the results of joint points recognition of OpenCV and Mediapipe We didthis research based on using the libraries of OpenCV in combination with the solution“Human Pose Detection and Tracking” of MediaPipe analyzes the moving videos ofhuman body parts Then, identify the joint points and compare the results obtained underdifferent environmental conditions Experimental results show that the accuracy of thejoint points recognition of OpenCV and MediaPipe depends on the resolution of thevideo Specifically, the higher the resolution of the video, the less noise the joint pointsrecognition results will have In this research, we examined the joint points detectionresults of three similar videos that had the same frames per second at 30FPS but variousresolutions of 480P, 720P, and 1080P Experimental results demonstrate that the jointpoints detection result of 480P video is the noisiest while the noise is significantlyreduced when increasing the resolution to 720P and 1080P In the future, we will continueto research to discover more factors that affect the joint points recognition results ofOpenCV and MediaPipe

v

OpenCV (stands for Open Computer Vision) is one of the most popular ComputerVision libraries in the world It is a set of software tools for real-time image processing,video, analytics, and Machine Learning MediaPipe is a tool designed by google, itprovides many features for AI and Machine Learning problems running on manydifferent platforms It brings together a series of cross-platform, interoperable andextremely lightweight Machine Learning solutions

The basic goal of a computer vision system is to detect moving things Motiondetection is a well-known computer technology related to computer vision and imageprocessing that focuses on detecting objects or instances of a particular class in digitalphotos and videos (e.g., people, flowers, and animals) Face detection, characterrecognition, and vehicle computation are just some of the well-studied applications ofobject motion detection Recognition of joint points is also one of the importantapplications in sketching the seams between joints from which the parameters of angleand speed can be obtained However, one of the main reasons for the difficulty ofaccurate detection, there are many limitations such as environmental fluctuations thatmake the joint points detection process more difficult Currently, in the world there aremany papers on applications of vision special computers in using open CV andMediaPipe to detect joint points, however, there are very few articles mentioning theinfluence of environmental factors on detection In this project, we will introduceenvironmental factors that affect and have solutions to increase the ability to identify jointpoints more accuracy

vi

CHAPTER 1 STATE THE PROBLEM

This chapter provides the background on human posture estimation including what ispose estimation, the importance of human posture estimation, the difference between 2Dand 3D human posture estimation, Types of models to estimate human posture andenvironmental factors affecting joint point recognition And from there, come up with asolution to limit the factors of the environment to identify the joint point, especially theinfluence of the camera.

1.1 Overview of human posture estimation

1.1.1 What is the human pose estimate? [1]

Human posture estimation is a computer vision task that represents a person's orientationin a graphical format This technique is widely applied to predict a person's body parts orjoint position This is one of the most exciting areas of research in computer vision thathas gained a lot of traction because of the abundance of applications that can benefit fromsuch a technology Human Pose Estimator identifies and classifies the posture of humanbody parts and joints in images or videos In general, model-based techniques are used torepresent and infer human body postures in 2D and 3Dspace.

Figure 1.1 Human body models

It is basically a way to capture a set of coordinates by defining human body joints likewrist, shoulder, knee, eye, ear, ankle, and arm, this is the point key in images and videosthat can depict a person's posture.

1

Then, when an image or video is fed to the posture estimation model as input, itdetermines the coordinates of these detected body parts and joints as output and theconfidence score shows accuracy of the estimates.

1.1.2 The importance of estimating human posture

Person detection has long been the main discussion center for various applications intraditional object detection With recent developments in machine learning algorithms,computers can now understand human body language by performing posture detectionand posture tracking The accuracy of these findings and the hardware requirements torun them have now reached commercially viable levels.

In addition, the evolution of technology is also profoundly transformative amid thecoronavirus pandemic, where high-performance real-time posture detection and trackingwill bring about some of the most impactful trends in machine vision count Forexample, it can be used to create social distancing by combining human postureestimation and distance prediction experience It helps people maintain physical distancefrom each other in a crowded place.

Human posture estimates will significantly impact various industries, including security,business intelligence, health and safety, and entertainment One of the areas where thistechnique has proven its existence is in autonomous driving With the help of real-timehuman posture detection and tracking, computers can radically sense and predictpedestrian behavior - enabling more stable driving.

1.1.3 The estimate human pose in 2D and 3D

There are two main techniques by which posture estimation models can detect humanposture.

2D Posture Estimation: In this type of pose estimation, you only need to estimate theposition of the body joints in 2D space relative to the input data (i.e., image or videoframes) The location is represented by the X and Y coordinates for each key point 3D Pose Estimation: In this type of pose estimation, you turn a 2D image into a 3Dobject by estimating an extra Z dimension to the prediction 3D pose estimation allows usto predict the exact spatial position of a represented person or thing.

Estimating 3D poses is a significant challenge faced by machine learning engineersbecause of the complexity involved in constructing datasets and algorithms that estimate

2

several factors, such as the background scene of an image or video, lighting conditions,etc

1.1.4 Models to estimate human posture [2]

a) Skeleton-based model also known as kinematic model, this representation includes aset of key points (joints) such as ankles, knees, shoulders, elbows, wrists, and major limbdirections are used to estimate 3D and 2D poses.

This flexible and intuitive human body model covers the skeletal structure of the humanbody and is often applied to capture the relationships between different body parts.b) Contour-based model also known as flat model, it is used to estimate 2D posture andincludes rough contour and width of the body, trunk and limbs It basically represents theappearance and shape of the human body, where body parts are shown by the boundariesand rectangles of a person's contour.

A well-known example is Active Shape Modeling (ASM), which records the entirehuman body graph and silhouette deformations using principal component analysis(PCA).

c) Mass-based model also known as volumetric model, used to estimate 3D posture Itincludes many popular 3D human body models and poses represented by meshes andgeometric human shapes, often captured for deep learning-based 3D human poseestimation.

3

CHAPTER 2 THEORETICAL BASIS

The content of this chapter presents the theoretical basis to solve the problemposed in the previous chapter: "Analysis of factors affecting the identification of jointpoints" The content of this chapter is to introduce the tools used such as: Open CV,Mediapipe, Pycharm and some of the main methods used:

2.1 OpenCV image processing library

OpenCV (Open-Source Computer Vision Library) library is an open-source librarydeveloped by Intel starting in 2000 OpenCV is one of the perfect tools for computervision development and learning machine learning, especially real-time relatedapplications This library is written in C/C++ programming language, can beimplemented on Windows, Linux, MacOS, Android, iOS operating systems.

Figure 2.2 OpenCV library

The OpenCV library has about 500+ functions, divided into modules corresponding toeach function Some of the main modules in the OpenCV library include:

▪ Core: contains the basic structures and classes that OpenCV uses to store and processimage manipulation such as: Mat, Scale, Point, Vec, etc; and basic methods to use forother modules.

4

▪ Imgproc: is OpenCV's image processing module, including linear filters andnon-linear, geometric transformations, color space transformations, andalgorithms related to the histogram of the image.

▪ Highgui: allows user interaction on user interface (UI) such asDisplay pictures, display videos.

▪ Feature2d: has the function of finding features in the image, performing algorithmsimage feature extraction algorithms such as PCA video: used for analysis video data,including motion estimation, background subtraction, and tracking algorithms objecttracking

• Objdetect: used to detect objects such as faces, eyes, people, cars in the image.Algorithms used in this module are Haar-like Features.

• Ml: contains machine learning algorithms, serving classification problems and otheralgorithms clustering problem Some of the algorithms used in this module are SVM(Support Vector Machine), ANN.

The OpenCV library has been used to build applications in many different fields Inrobotics, the OpenCV library is used to control direction, obstacle avoidance, and human-machine interaction In medicine, libraries are used to classify and detect cancer cells,segment 2D, 3D, reproduce 3D images of organs In the field of industrial automation,the OpenCV library is used to build applications for fault identification, barcodechecking, product sorting In the field of information security and safety, libraries.OpenCV is widely used in the fields of camera surveillance, biometric image processing.

2.2 MediaPipe

In a nutshell, Mediapipe is a collection of a series of cross-platform, interoperable andextremely lightweight Machine Learning solutions Some of the advantages of thissolution include:

• Provides a quick inference solution: Google claims that this toolkit can run stably onmost common hardware configurations.

Easy to install and deploy: The installation is extremely easy and convenient, can bedeployed on many different platforms such as Mobile (Android/iOS), Desktop/Cloud,Web and IoT devices.

5

• Free and open source: The entire source code is publicly available on MediaPipe, userscan completely use and customize it directly to suit their problems.

Figure 2.3 Solutions offered and availability across platforms in MediaPipeMost of the outstanding problems in the field of Computer Vision - Computer Vision, areinstalled by Google in MediaPipe We will go through the solutions provided to betterunderstand the diversity of MediaPipe.

2.2.1 Face Detection

This is a problem familiar to everyone With an image or a video as input, our task is tofind the position and bounding box of the human faces appearing on it, as well as mark theimportant points (MediaPipe uses 5 -landmarks) on that face MediaPipe Face Detectionuses the BlazeFace network as the foundation but changes the backbones In addition, theNMS (non-maximum suppression) algorithm has also been replaced by another tactic,greatl reducing the processing time.

6

2.2.2 Face Mesh

Instead of finding the bounding box surrounding the face, Face Mesh is a problem thatidentifies a series of points on the face, thereby forming a mesh of the face This mesh willbe applied to 3D face image editing problems or tasks related to 3D Alignment and Anti-spoofing The tool in MediaPipe will generate a total of 468 points on the face and create amesh without requiring too much computing power as well as the number of cameras (just1 front camera).

7

2.2.3 Hands detection

Hands Detection, also known as hand recognition, is the problem that we will test in thisarticle The output of this solution is a skeleton model of the hand, which includes thelocations of landmarks on the hand and is joined together to form a complete hand frame.

Figure 2.6 3D-Skeleton recognition of the hand

8

Expanding on the Hands Detection problem, Human Pose Estimation provides a 3Dskeleton model of the whole body, with predefined key joints and joined together to formthe human frame The strategy used for this problem is similar to Hands Detection andFace Mesh BlazeFace, again, is used as the main idea for this card processing algorithm.

Figure 2.7 Human Pose Tracking in 3D

In addition, there are many more solutions provided by Google in this kit, includingSegmentation problems (Selfie, Hair, ), Object Detection, Motion Tracking, 3D ObjectDetection,

2.3 Pycharm

PyCharm software provides a complete set of tools for professional Python developers.PyCharm is built around an editor that understands code deeply, and a debugger thatgives a clear view of how the code works PyCharm provides integration withcollaboration tools such as version control systems and trackers The professional editorextends the essentials by seamlessly integrating with web frameworks, JavaScriptengines, virtualization, and containerization support.

9

An important aspect of programming is understanding the code platform you're puttingon PyCharm ensures you can explore your project with just a few taps, it gives you anoverview of the project structure and gives you instant access to relevant documents.from the editor Understanding a coding platform faster means speeding up yourdevelopment.

Key features of Pycharm software:• Supports Windows, macOS and Linux

• Smart completion code support, one-click navigation and PEP8 style checking• Safe and automated refactoring of your project

• Automatic detection of code problems: e.g unused code analysis• High performance debugger

•Vim simulation modePython debuggingVersion controlUnit testing &Code coverageProfilingDatabase ToolsWeb frameworkDjangoFlaskPyramidWeb2pyWeb developmentJavaScriptHTML / CSSAngularJSReact

10

Node.jsVue.jsVirtualization:Remote serversVagrant boxesDocker containers

11