THÔNG TIN TÀI LIỆU
MINISTRY OF EDUCATION AND TRAINING MINISTRY OF TRANSPORT HO CHI MINH CITY UNIVERSITY OF TRANSPORT LÂM QUANG CHUYÊN NEURAL NETWORK IN THE WHEELCHAIR CONTROL SYSTEM FOR SEVERE DISABILIED PEOPLE USING EEG SIGNAL AND CAMERA FIELD OF STUDY TECHNIQUE OF CONTROL AND AUTOMATION CODE: 9520216 SUPERVISORS: Assoc Prof., Dr NGUYỄN HỮU KHƯƠNG Assoc Prof., Dr VÕ CÔNG PHƯƠNG HO CHI MINH CITY– 03/2020 ABSTRACT Nowadays, EEG signal, one of the most important field was interested by science researchers, the main purpose of research is to support application devlepments, diagnose and find out pathological of human as stress, depression, epileptic, alzheimer, brain trauma…, however, in the field of automatic control serving for human, especially for disabilities people, has not been studied so much For long time ago, recording and processing the EEGs or ECGs signal was the work of neurologists or cardiologists, nowadays, with the development of modern signal processing and analysis tools such as neural networks and AI systems, such signals can be processed to meet the other needs, such as the control system support human acitivites.The goal of this thesis is to build a control system, which support some basic human activities through EEG signal For example, wheelchair equipement control for disabled people, meet today’s pressing social needs In this thesis, author researched and analyzed three EEG signal pre-processing methods as Fourier transform, Wavelet transform and HHT transform, converting EEG signal to basic EEG waves (Delta, Theta, Alpha, Beta, Gamma), and then using data clustering technical before put them into input layer of multilayer neural network The neural network was test from singlelayer to multilayer (3 layer) Author combined the EEG signal processing system with HHT pre-processing and image processing using multi neural network to control the wheelchair model with accurate rate 92.4% for group 20 students, this shows the successful in the practical of the thesis Trang THE THESIS STRUCTURE Chapter 1: Overview – Presenting of the research situation of EEG signal in domestic and internation, presenting of results have been researched and published, anayzing unsolved and limited problems that thesis needs to solve, in addition, the thesis also presentate the aimed and scope research, the contributions of thesis in science and reality life Chapter 2: Theoretical basis – Presenting of basic knowledge relatived to thesis as: Fourier Transform, STFT, Wavelet Transform, HHT…, clustering data and then classifying EEG signal partterns by multilayer neural network Chapter 3: Model Contruction – Presenting how to construct multilayer neural network, this process was conducted from singlelayer (detect signal partterns) to multilayer (detect signal partterns), in addition, author also discusses between EEG signals and image processing via camera in combination Each result has published on international journals or international conference Chapter 4: Constructing software and hardware to control the wheelchair model – This chapter introduced the functions of sotfware, the image processing to detect the eye direction in combined with EEG signals processing to achived the final result This chapter also present the experimental results had been performed by student in Ho Chi Minh City Industry and Trade College (HITU), comparison the experimental results between processes (image and EEG signal processing), and then the results has been combined with processes Chapter 5: Conclusions and recommendations – this chapter present the results have been achived compared to thesis requirements and offering the solutions to develop EEG field more and more completely Trang CHAPTER OVERVIEW OF EEG SIGNAL 1.1 Researching situation in domestic In present, the previous researchs in EEG field mainly focused on motor activities, eye blink and movement of head… to detecting and classifying these activities could be performed by amplitude threshold method There aren’t topics now related in processing and classifying the difference EEG signal partterns corresponding to difference image types observation yet, this couldn’t be resolved by amplitude threshold method but must be resolved by extracted these signal partterns into specific features via multilayer neural network 1.2 International research situation The research works were published at international paper mainly focused on diagnosing epilepsy sleep disorders, coma and brain death, stress, depresssion pathological… in automation field as spelling, eye blink, head movement, mental arithmetic… this were performed by offline, no mainly focus on resolving in realtime and in control automation field 1.3 The content of thesis At first, author constructed the multilayer neural network base on raw EEG data from San Diego University (UCSD), to identify and classify EEG signal partterns corresponding to difference image types (human, city, landscape, flower and animal), after determining the feasibility of the multilayer neural network, author conducted on realtime EEG data combined with camera to increase efficiency, this has been performed by student of HITU, in the implementation process, the thesis extracted the feature signal by Hibert Huang Transform (HHT), clustering data and then using multilayer neural network, making system work more efficienty and avoiding “overfitting” problem Trang 1.4 The purpose of the research Finding out the way collect EEG signal partterns easily and effectively Using math tools to transform the EEG signal partterns into specified extract features, using K-Mean technique to cluster EEG signal data before putting in multilayer neural network to classify these partterns Combining EEG signals and camera processing aim to identified and classified process more easily and effectively 1.5 Object and scope of the research The main object in thesis is to use multilayer neural network to classify EEG signal partterns into control commands corresponding to commands to control the wheelchair as: Forward, Turn right, Turn left, Reverse and Stop In thesis also mention to image processing to detect the eye direction aim to help the system work more easily and effectivly However, the thesis don’t focus more on image processing, but on EEG signal processing 1.6 The contribution of thesis 1.6.1 The contribution about theory Find out the observation board which easy to use to collect data, combine scientifically between feature extraction algorithm and cluster data before putting into neural network to classify data partterns 1.6.2 Practical contribution The experimental result of thesis show that the EEG signal partterns classification through eye observation (with differnce image partterns), for people who has mind and eyes as normal people could absolutely performance CHAPTER - THEORETICAL BASIS 2.1 EEG signal and its characterizations Delta wave (0 – Hz), the highest amplitude as figure 2.1, it often appear at the child up to year old and adult when sleeping, well sleep It represents the grey matter of the brain This wave usually appears everywhere on the scalp Trang Figure 2.1 The Delta wave The Theta wave (3 – Hz), it appear when the eye are closed and the mind is in a relax state as Figure 2.2, it apperas in adult or when awake in the elderly and often appears in the temples Figure 2.2 The Theta wave The Alpha wave (7 – 13 Hz), it appear often in elder as Figure 2.2, Alpha wave usually appear on both sides of scalp but having an uneven amplitude, this waves appear when the eye are closed (a state of relaxation) and often disappear when the eyes are opened or under stress Figure 2.3 The Alpha wave The Beta waves (13 – 30 Hz), with low amplitute as Figure 2.4, This waves usually appears in patients who are often in a state of alertness, prevention, and anxiety… this waves are distributed symmetrically on both sides and most clearly at the front, it usually appear in front and at the top of the cerebral cortex, the amplitude is less than 30uV Figure 2.4 The Beta wave Trang The Gamma wave (30 – 45 Hz), it often referred to as fast Beta wave This wave usually has low amplitude and rarely appears, but the discovery of this wave plays a important role in identifying neurological diseases that occur in the center of the cerebral cortex Figure 2.5 The Gamma wave 2.2 The electrode positions on the scalp The brain is one of the largest and most complex organs in the human body, it is made up of over 100 billion nerves, communicating with 1,000 billion synapses The electrode positions are mounted on the scalp according to international standard 10/20 as Figure 2.6 Figure 2.6 Electrode positions according to international standard 10/20 CHAPTER CONTROL MODEL CONTRUCTION To begin the research process, author used the database provided on the prestigious University of San Diego (UCSD) website of the USA, ranked 38 in the world in 2018, this data was obtained from participants when looked at different image objects (human, city, landscape, flower and animal), with color bits and size (256 pixels Trang wide and 384 pixels high), the total number of samples is 21,000 In this chapter, author present to built the model from simple to complex step by step, and then evaluate the experimental results on database provided by the University of San Diego (UCSD), and from 80 students from HITU to make clarify the contribution and scientific meaning of the thesis 3.1 Single-Layer neural network model At first, author built a single neural network to separate two parttern of signals (animals and not animals), the purpose of this study is to evaluate whether the neural network meets the classification requirements, thesis used Matlab software for this experimental process The features of EEG database is extracted by Wavelet transform (Mexico hat) and used a single neural network to identify The system model is shown in Figure 3.1, this model included of stages: stage 1: preprocessing raw data signals and then synthesize into basic EEG signals Delta, Theta, Alpha, Beta and Gamma Stage 2: builting a single neural network with inputs corresponding to basic EEG signals: Delta, Theta, Alpha, Beta, Gamma and one output to determine the clasified results Raw data Pre-processing Single-Layer Neural Network Classified result Figure 3.1 Single-Layer neural network model The training process is performed on the database with the following parameters: Learning rate: 0.7 Trang Initial random weight in the range from -0.5 to 0.5 The error threshold is 1x10-5 based on MSE (Mean Square Error) The maximun number loop: 5.000 Experimental results of identification on the database are shown in Table 3.1 Table 3.1 Experimental results on database Image types Animal/Landscape Identification Rate 99,13% France Landscape Wild sheep Animal 98,67% Wild cats Animal 99,28% Bali, Indonesia Landscape 62,44% Wild animals Animal 99,64% California Coasts Landscape 56,89% Wolves Animal 98,64% Mushrooms Landscape 95,16% Kenya Animal 99,76% The big Apple Landscape 98,79% Snakes, lizards Animal 98,32% Caves Animal 67,18% Polar bears Animal 99,03% Exotic Hong Kong Landscape 98,72% Images of France Landscape 99,37% Fabulous fruit Landscape 98,25% Wild animals Animal 93,97% Sand & solitude Animal 98,42% Lions Animal 62,78% Trang Image types Great Silk Road Animal/Landscape Landscape Identification Rate 98,47% From the experimental results in Table 3.1, author found that the average accuracy rate of the identification results on the database was 91.15% 3.2 Multi-Layer neural network model 3.2.1 System Model Based on the results achieved from the single neural network model, author continued to develop a multi neural network model with the results of classifying EEG signal partterns corresponding to control signals with accurate rate 93.57% Table 3.2 describes the result of 05 control commands corresponding to the equivatent image types This model uses Wavelet transform to noise signals and extract features, then using K-mean algorithm to cluster the characteristics of the signal partterns and then put into the multi-layer neural network to classify, in this model, author chooses 10 channels to reduce processing time and enhance performane System model is shown in Figure 3.2 EEG Signal Selecting Channel Wavelet Transform Clustering Muli-Layer Neural Network Classified Result Figure 3.2 Multi-Layer neural network system model Trang 1 2 3 4 5 6 7 8 Output Layer 10 Input Layer 11 Hidden Layer Figure 3.11 Multi-layer neural network model The first layer contains nodes like Delta, Theta, Alpha, Beta, Gamma, d1, d2, d3 and d4 This class is called the input class The second layer is the hidden layer, the number of nodes in the hidden layer is 11 nodes The output layer contains nodes, the result of this node is used to classify EEG signals Because the activation function is hyperbolic tangent, the value of the output node is between [-1, 1] Since the output has nodes, which one has the largest value, it will be selected and that is the control signal 3.3.4 Select the data set and experimental results Experimental data was collected from 80 volunteering students of Ho Chi Minh City College of Industry and Trade (HTIU) Students wear the Emotiv EEG device and sit 120 cm away from the observation board Experimental data is divided into data sets as follows: Training data set was collected from 70% data of 60 students Trang 18 The first test data set was collected from 30% remaining data of 60 student The second set of test data was collected from the remaining 20 students After training the neural network from the training database, experimental results of the first test data set are shown in the confusion matrix, shown in Table 3.5 and identification rate in Table 3.6 Table 3.5 The confusion matrix of the result from the first database classification Actual classification Predic tive Classi ficatio n Huma n Anim al Flowe r City Lands cape Human 91,2% 1,6% 3,1% 0,9% 2,7% Animal 1,9% 91,1% 1,2% 3,2% 2,0% Flower 2,8% 2,5% 92,8% 1,7% 1,5% City 2,4% 2,1% 0,7% 92,1% 1,9% Landscape 1,7% 2,7% 2,2% 2,1% 91,9% Table 3.6 Experimental results on the first database TP TN FP FN AC P Human 91.2% 91.3% 8.7% 8.8% 91.3% 91.3% Animal 91.1% 91.1% 8.9% 8.9% 91.1% 91.1% Flower 92.8% 92.6% 7.4% 7.2% 92.7% 92.6% City 92.1% 92.2% 7.8% 7.9% 92.2% 92.2% Landscape 91.9% 92.1% 7.9% 8.1% 92.0% 92.1% The experimental results of the second test database are shown in the confusion matrix in Table 3.7 and the identification rate in Table 3.8 Trang 19 Table 3.7 Confusion matrix of classification result for the second database Predic tive Classi ficatio n Human Animal Flower City Landscape Actual classification Anim Flowe Human al r 90.7% 1.9% 2.7% 1.4% 90.8% 2.1% 2.3% 2.3% 92.3% 3.1% 2.6% 1.2% 2.5% 2.4% 1.7% City 1.1% 2.6% 3.2% 91.6% 1.5% Lands cape 3.1% 2.4% 1.7% 1.3% 91.5% Table 3.8 Experimental results on the second test dataset TP TN FP FN AC P Human 91.2% 91.3% 8.7% 8.8% 91.3% 91.3% Animal 91.1% 91.1% 8.9% 8.9% 91.1% 91.1% Flower 92.8% 92.6% 7.4% 7.2% 92.7% 92.6% City 92.1% 92.2% 7.8% 7.9% 92.2% 92.2% Landscape 91.9% 92.1% 7.9% 8.1% 92.0% 92.1% Experimental results for eye direction signals are shown in Table 3.9 and for EEG signals shown in Table 3.10 Table 3.11 compares experimental results The graph in Figure 3.12 shows the chart of two signals when identifying them separately Table 3.9 Experimental results on eye direction signals TP TN FP FN AC P Human 85.1% 85.7% 14.3% 14.9% 85.4% 85.6% Animal 84.5% 84.1% 15.9% 15.5% 84.3% 84.2% Flower 87.3% 86.3% 13.7% 12.7% 86.8% 86.4% City 83.6% 84.0% 16.0% 16.4% 83.8% 83.9% Landscape 84.2% 83.2% 16.8% 15.8% 83.7% 83.4% Trang 20 Table 3.10 Experimental results on EEG signals TP TN FP FN AC P Human 90.2% 89.9% 10.1% 9.8% 90.1% 89.9% Animal 90.3% 90.0% 10.0% 9.7% 90.2% 90.0% Flower 92.3% 91.8% 8.2% 7.7% 92.1% 91.8% City 90.7% 90.4% 9.6% 9.3% 90.6% 90.4% Landscape 90.4% 90.5% 9.5% 9.6% 90.5% 90.5% Table 3.11 Experimental results of identification methods EEG and Camera EEG Camera Human 90,8% 90,1% 85,4% Animal 90,9% 90,0% 84,3% Flower 92,2% 91,8% 86,8% City 91,7% 90,4% 83,8% Landscape 91,7% 90,5% 83,7% Figure 3.12 Experimental results on separate methods Trang 21 CHAPTER CONSTRUCTING SOFTWARE AND HARDWARE TO CONTROL THE WHEELCHAIR MODEL In this chapter, author built the wheelchair hardware and software system based on the models presented in Chapter Then evaluate the experimental results to clarify the practical contribution of the thesis 4.1 Wheelchair control software system Wheelchair control software is designed on Visual Studio C# 2015, the software interface includes functions as follows: 4.1.1 Login to the system In this section, it is mandatory for anyone who uses software to have an account to log in to the system, personal account used to manage the database of brain signals, training time, accurate rate during the control, the interface of the login part in Figure 4.1 Figure 4.1 System login interface 4.1.2 Wheelchair control training For a person who has never participated in control, this step must be done, as well as someone who has never drive, they had to learn to drive In order to control the wheelchair, the participants have to control commands with the acuary rate of more 90% for each command The purpose of this training is to help participants become familiar with wheelchair control and concentrate in control The training software interface in Figure 4.2 Trang 22 Figure 4.2 Training software interface 4.1.3 View the EEG signal via graph The software also has the function of reviewing the graph of EEG signals for every channel, depending on the purpose of the research, just click on the electrode channel position as shown in Figure 4.3 Figure 4.3 Viewing EEG signals via graph 4.1.4 Extracting the feature of EEG signals To extract the feature of EEG signals, author used the HHT algorithm as presented in the theoretical basis, total number of analyzied channels is 10, each channel is analyzed into 12 IMF (intrinsic functions), so we have all 120 IMFs for each processing The program extracted signal channel into IMFs, and then from these IMFs extracted in basic waves as Figure 4.4 Trang 23 Figure 4.4 An EEG signal channel is transformed into the basic wave 4.2 Hardware system The hardware system includes devices such as observation board, wheelchair model, Emotiv equipment and computers with control software, the hardware system is shown as Figure 4.5 Figure 4.5 Sitting posture and hardware devices 4.2.1 Observation board The observation board is 46x42 cm in size, can adjust the tilting direction like a laptop screen to suit each participants, on the observation board there are images of 8x12 cm each placed at even intervals at a distance of cm, a camera is placed between the image of the person and the flower to record the direction of the eye as Figure 4.6 Trang 24 Figure 4.6 Observation board Camera is mounted on the observation board to detect eye direction corresponding to 05 image types on the board, camera of Logitech Co used with model C615, the specifications as follows: resolution HD 1080, 30fps, field of view 78o, connect to computer via USB, the camera is shown in Figure 4.7 Figure 4.7 Camera Logitech C615 4.2.2 Wheelchair model Wheelchairs used to simulate the process of commands from computer, the wheelchair with compact size can run forward, backward, right turn, left turn and stop in accordance with the commans form computer, the schematic diagram of the circuit is shown in Figure 4.9, specifications of wheelchair is shown in Table 4.1 Trang 25 Figure 4.8 Model of Wheelchair Table 4.1 Specifications of model wheelchairs No Dicriptions size (length – width – high) Motor Number of motor Battery 9VDC Weight Specifications 35x30x35 cm 200 rpm, VDC 02 2000 mA 0.8 kg Wheelchair control system principle schematic includes the arduino UNO3 processor, driver LM298 for two motors, and bluetooth module HC-05 to receive control commands from computer Figure 4.9 Principle schematic of control board 4.2.3 Emotiv device Trang 26 An indispensable device in experimental control is the Emotiv EPOC + device (EPOC+ head), this device record EEG signals and send it to the computer via bluetooth, EPOC + device is shown in Figure 4.10 Figure 4.10 EPOC+ device EPOC + has the specifications as follows: Number of channels: 14 channels + reference channels Sampling frequency: 128 SPS / 256 SPS Data resolution 14 bits, LSB = 0.51 uV Connect by bluetooth, 2.4GHz band Battery using time 12 hours The computer connect to devices, EPOC + head and wheelchair model as Figure 4.11 Figure 4.11 The computer connect to devices via bluetooth 4.3 Selecting a group to participate in the system evaluation After finishing the above steps, author conducted the final experimental step to re-evaluate the entire research results During this period, author selected groups of participant as follows: Group 1: Selecting 20 from 60 students in the first phase of the best experimental results Group 2: Choosing 20 from 60 remaining student Group 3: Choosing 20 student never participated in the experiment test Trang 27 In groups and 2, student not have to go through training steps, but group 3, student must be trained on the manipulation steps of using hardware, data acquistion and how to use the control software, experimental time is described in Table 4.2 Table 4.2 Experimental time to evaluation system Decription Time Take part in experiment 04/03/2019 – already, and get the best 08/03/2019 experiment result Group Take part in experiment 11/03/2019 – already 15/03/2019 Never take part in 18/03/2019 – Group experiment 22/03/2019 4.4 Experimental results for options using EEG and Camera The experimental results of these groups are as follows: Group Table 4.3 Experimental results of groups Combining EEG EEG signal Camera and Camera Group 92,4% 92,4% 86,7% Group 88,6% 86,2% 84,9% Group 86,4% 72,3% 83,5% From the experimental results of Table 4.3, author show that who have used to a lot about operation system will have more better results on EEG, and the results on the camera are relatively equal, thus to use the system effectively, the first requirement is that the participant must be trained of control system CHAPTER CONCLUSIONS AND RECOMMENDATIONS 5.1 Conclusion Through theoretical research, algorithm construction, builting the model and conducting experiments were presented in the thesis, the thesis show that the problem of using EEG signals combine with supporting cameras for people with disabilities was solved and Trang 28 achieved the initial goals Some scientific contributions and new things in the thesis can be summarized as follows: Using HHT method to extract EEG signals feature, these signals are synthesized into the sum of the basic signals before putting into the neural network, which is a new proposal of the thesis as well as helps the classification of signal samples to be fast and accurate rate 92.4% Reducing the number of channels that help reduce processing time, reducing the number of signal channels base on the characteristics of each electrode position in the scalp as well as experiments The combination of camera to detect the eye direction helps the system run stably and helps the trainees quickly become more proficient with wheelchair control Using EEG signals in the field of control is also a new proposal in the thesis because at present, serveral publics in the country only research the theory of EEG signals, filter noise, and use blink in identifying and detecting Building an EEG signals acquisition software and experiment on wheelchair control model The author has built a wheelchair control system through EEG signals processing that control the wheelchair physical model as Figure 5.1 Combined on controlling the wheelchair model and processing of continuous signals 5.2 Recommendation After the research, author is looking forward to receiving the attention of friends and colleagues, scientific researchers, social activists are interested in spiritual support as well as sharing relevant knowledge to build a more complete and useful project for the society in general as well as those people with disabilities in particular Trang 29 Figure 5.1 Synthetic model The acquisition and processing of EEG signals are being strongly researched in the world… in order to solve various medical field and automation control problems in Figure 5.2, production, business in Figure 5.3 and entertainment… so in our country in general and the University of Transport of Ho Chi Minh City in particular should continue research to create products that are truly useful to society Figure 5.2 Wheelchairs controlled by EEG signals Beside of studying EEG signaling in control automation field, EEG signals can also be applied in other areas such as biomedical psychology to detect cases of depression, stress to make recommendations suitable for each person, in business field, in the product exhibition area, brands, EEG signals helps traders know which products customers are most interested in, thereby helping planners to come up with appropriate and effective business strategies Trang 30 Figure 5.3 Psychological survey of customers by EEG signals Finally, in order for this field to grow and help the passionate students to inherit and develop more and more, author suggested the Graduate Institute and the HCMC University of Transport should equip a laboratory room for EEG signal, so that the student can easily exchange their mutual knowleges Trang 31 THE RESEARCH WORKS ARE PUBLISHED [1] Quang Chuyen Lam, Luong Anh Tuan Nguyen, Huu Khuong Nguyen A Novel Approach for Classifying EEG Signal with Multi-Layer Neural Network ICRAI 2017 Proceedings of the 2017 International Conference on Robotics and Artificial Intelligence (ISBN:978-1-4503- 5358-8), (ACM Digital Library), Pages 79-83, December, 2017 (Scopus) [2] Lâm Quang Chuyên, Nguyễn Hữu Khương “Identifying EEG signals with Emotiv devices via multilayer neural networks” Fourth Transport Science and Technology Conference (ISBN: 978-604-76-1578-0), pages 150-156 May 2018 [3] Quang Chuyen Lam, Luong Anh Tuan Nguyen, Huu Khuong Nguyen Build Control Command Set Based on EEG Signals via Clustering Algorithm and Multi-Layer Neural Network Journal of Communications (ISSN: 1796-2021), pages 406-413, Vol 13, No 7, July, 2018 (Scopus) [4] Luong Anh Tuan Nguyen, Quang Chuyen Lam and Huu Khuong Nguyen Developing a Wheelchair System Controlled Based on EEG Signal and Eye-Direction International Journal of Computer Science and Network Security (ISSN: 1738-7906), Pages 115-122, VOL.19 No.3, March 2019 (ISI – ESCI) [5] Nguyễn Hữu Khương, Lâm Quang Chuyên, Phạm Thúy Ngọc, School-level scientific research project “Building wheelchair control system based on EEG signal with the help of Camera”, Code: KH1818, Aug 2019 Trang 32 ... partterns into specified extract features, using K-Mean technique to cluster EEG signal data before putting in multilayer neural network to classify these partterns Combining EEG signals and camera... partterns by multilayer neural network Chapter 3: Model Contruction – Presenting how to construct multilayer neural network, this process was conducted from singlelayer (detect signal partterns)... and cluster data before putting into neural network to classify data partterns 1.6.2 Practical contribution The experimental result of thesis show that the EEG signal partterns classification through
Ngày đăng: 14/04/2020, 06:29
Xem thêm: