Đang tải... (xem toàn văn)
NATIONAL AND KAPODISTRIAN UNIVERSITY OF ATHENS SCHOOL OF SCIENCE DEPARTMENT OF INFORMATICS AND TELECOMMUNICATION M Sc THESIS Tiny ML in Microcontroller to Classify EEG Signal into Three States Thuy T Pham Supervisor ( or supervisors ): Van - Tam Nguyen , Professor at Telecom ParisTech, France ATHENS SEPTEMBER 2022 M Sc THESIS Tin y ML in Microcontroller to Classify EEG Signal into Three States Thuy T Pham S N : 7115192100007 Supervisor (or supervisors) : Van - Tam Nguyen , Professor at Telecom ParisTech, France ABSTRACT This thesis investigates how to implement an own - built neural network for electroencephalography signals classification on a n STM32L475VG microcontroller unit The original dataset is analyzed and processed to better understan d the brain signals Th ere is a comparison between three mach ine learning algorithms (linear support vector machine, extreme gradient boosting, and deep neural network) in three testing paradigms: specific - subject , all - subject , and adaptable to select the most appropriate approach for deploying on the microco ntroller The implementation procedure with detailed notation is presented , and the inference is also performed to feasible observation Finally, possible improvement solutions are proposed with in a clear demonstration SUBJECT AREA : Signal processing, machine learning, embedded system KEYWORDS : electroencephalography, artificial neural network, STM32 microcontroller, SVM, XGBoost Erasmus Mundus Joint Master’s Degree “SMART Telecom and Sensing NETworks” (SMARTNET) ( 2019/2021 intake ) Aston University, Triangle, B4 7ET / Birmingham, UK Email: aipt_smartnet@aston ac uk / Web - site: smartnet astonphotonics uk/ Acknowledgement This Master Thesis has been accomplished in the framework of the European Funded Project: SMART Teleco m and Sensing Networks (SMARTNET) - Erasmus+ Programme Key Action 1: Erasmus Mundus Joint Master Degrees – Ref Number 2017 – 2734/001 – 001, Project number - 586686 - EPP - 1 - 2017 - 1 - UK - EPPKA1 - JMD - MOB, coordinated by Aston University, and with the participatio n of Télécom SudParis, member of IP Paris and National and Kapodistrian University of Athens ACKNOWLEDGEMENT Khi ng ồ i vi ế t nh ữ ng dòng này, em bi ế t r ằ ng hành trình hai năm c ủ a mình v ớ i SMARTNET s ắ p đi t ớ i h ồ i k ế t Th ờ i gian trôi qua ch ẳ ng ch ờ đ ợ i ai, nhưng k ỷ ni ệ m là th ứ s ẽ còn t ồ n t ạ i mãi mãi ! C ả m ơn SMARTNET đã tin tư ở ng và cho em cơ h ộ i đ ể có th ể th ự c hi ệ n hóa ư ớ c mơ c ủ a mình Em không nghĩ r ằ ng mình l ạ i có th ể “s ố ng sót” qua nh ữ ng th ử thách kh ắ c nghi ệ t trong su ố t khó a h ọ c này: nh ữ ng khó khăn lúc làm visa, lo ch ỗ ăn ở ; g ầ n m ộ t năm tr ờ i ch ỉ quanh qu ẩ n v ớ i căn phòng vì d ị ch Covid, nh ữ ng môn h ọ c mà em chưa bao gi ờ đư ợ c ti ế p xúc, nh ữ ng đ ề thi dài 10, 11 trang gi ấ y… ^_^ Nhưng có l ẽ , chính nh ữ ng th ử thách ấ y đã rèn luy ệ n em t rư ở ng thành và t ự tin hơn trong cu ộ c s ố ng SMARTNET không ch ỉ là m ộ t chương trình h ọ c đơn thu ầ n mà v ớ i em đó là m ộ t gia đình, nơi có các b ạ n t ừ kh ắ p nơi trên th ế gi ớ i cùng nhau chia s ẻ nh ữ ng kho ả nh kh ắ c, có nh ữ ng th ầ y cô thân thi ệ n c ở i m ở và nhi ệ t tình v ớ i sinh viên C ả m ơn t ấ t c ả nh ữ ng gì thu ộ c v ề SMARTNET! Em cũng mu ố n g ử i l ờ i c ả m ơn chân thành nh ấ t t ớ i th ầ y Văn - Tâm Nguy ễ n, ngư ờ i đã hư ớ ng d ẫ n cũng như đ ị nh hư ớ ng cho em trong su ố t quá trình th ự c hi ệ n th ự c t ậ p Không ch ỉ là ki ế n th ứ c h ọ c thu ậ t mà nh ữ ng ki ế n th ứ c đ ờ i s ố ng, th ầ y đ ề u chia s ẻ nhi ệ t tình v ớ i em, giúp em không c ả m th ấ y t ự ti trong nghiên c ứ u Bên c ạ nh đó, em cũng xin c ả m ơn s ự h ỗ tr ợ t ậ n tình t ừ các th ầ y cô trong b ộ môn COMELEC, Telecom Paris; đ ặ c bi ệ t là th ầ y Germain Pham – ngư ờ i luôn luôn truy ề n nh ữ ng năng lư ợ ng tích c ự c , d ạ y em nh ữ ng ki ế n th ứ c r ấ t h ữ u ích trong đ ề tài này Nh ữ ng l ờ i tri ân n ày em cũng xin g ử i t ớ i t ấ t c ả các th ầ y cô, nh ữ ng đi ề u ph ố i chương trình trong SMARTNET, đ ặ c bi ệ t là t ạ i Telecom SudParis và NKUA Ch ặ ng đư ờ ng này c ủ a em s ẽ không th ể nào hoàn thi ệ n đư ợ c n ế u không có s ự gi ả ng d ạ y và h ỗ tr ợ t ừ m ọ i ngư ờ i C ả m ơn nh ữ ng ngư ờ i b ạ n luôn sát cánh, đ ộ ng viên và “lôi kéo” mình đi ăn trong nh ữ ng lúc stress n ặ ng n ề C ả m ơn Juhyun Kim và Rilwanu Kasno – đư ợ c làm b ạ n v ớ i hai ngư ờ i là m ộ t trong nh ữ ng thành công l ớ n c ủ a mình trong chương trình th ạ c s ỹ này :D Cu ố i cùng, con xin c ả m ơn gi a đình ở Vi ệ t Nam, c ả m ơn b ố m ẹ và em trai đã luôn ủ ng h ộ s ự l ự a ch ọ n c ủ a con, luôn quan tâm, lo l ắ ng và giúp con có m ộ t tinh th ầ n m ạ nh m ẽ trong l ầ n đ ầ u tiên xa nhà Con s ắ p hoàn thành ch ặ ng đư ờ ng này r ồ i đ ể v ề thăm gia đình mình r ồ i ạ ! Thank you so much ! Merci beaucoup! Ευχαριστώ από τα βάθη της καρδιάς μου! SMARTNET PH Ạ M TR Ọ NG Th ủ y CONTENTS 1 INTRODUCTION 9 1 1 Related work 9 1 2 Project duration 10 2 BACKGROUND 11 2 1 Electroencephalography 11 2 2 Machine learning classifiers 12 2 2 1 Linear Kernel Support Vector Machine (SVM) 13 2 2 2 Extreme Gradient Boost (XGBoost) 14 2 3 Artificial neural networks 14 2 3 1 The perceptron 14 2 3 2 Ac tivation functions 15 2 3 3 Training the network 16 2 4 Machine learning on microcontrollers 16 2 4 1 Fixed - point quantization 17 2 4 2 The STM32L475 discovery kit 17 3 IMPLEMENTATION 19 3 1 Dataset 19 3 2 Data processing 20 3 3 Model training 21 3 4 Neural network deployment and inference 22 4 RESULTS 23 4 1 Specific - subject paradigm 23 4 2 All - subject paradigm 24 4 3 Adaptable paradigm 25 4 4 In ference 26 5 CONCLUSION AND FUTURE WORK 28 ABBREVIATIONS - ACRONYMS 29 ANNEX Ι 30 REFERENCES 35 LIST OF FIGURES Figure 2 - 1: 4 typical dominant brain normal rhythms [13] 11 Figure 2 - 2: EEG signal processing pipeline [15] 12 Figure 2 - 3: Confusion matrix for 3 classes 13 Figure 2 - 4 : SVM mechanism illustration [19] 13 Figure 2 - 5: An ANN with multiple hidden layers 14 Figure 2 - 6: A perceptron schematic 15 Figure 2 - 7: Activation functions: Sigmoid (left), ReLu (right) 16 Figure 2 - 8: Post - quantization in an ANN layer 17 Figure 2 - 9: The B - L475E - IOT01A discovery kit [35] 18 Figure 3 - 1: The general work flow of implementing neural network on microcontroller 20 Figure 3 - 2: Feature extraction step 20 Figure 3 - 3: Neural network diagram 21 Figure 3 - 4: Validation flow overview [33] 22 Figure 3 - 5: Flowchart of implemented AI model on STM32L475VG 22 Figure 4 - 1: Accuracy in SVM method 23 Figure 4 - 2: Accuracy in XGBoost method 23 Figure 4 - 3: Accuracy varies with number of electrodes in SVM 24 Figure 4 - 4: Accuracy varies with number of electrodes in XGBoost 24 Figure 4 - 5: Accuracy of NN over each record of 1 st subject 25 Fig ure 4 - 6: Accuracy of NN over each record of 2 nd subject 25 Figure 4 - 7: Accuracy results of XGBoost in adaptable paradigm 26 Figure 4 - 8: Testing inference 27 Figure 5 - 1: Transformer model’s performance 28 LIST OF TABLES Table 2 - 1: Board specification 18 Table 3 - 1: Sample data 19 Table 4 - 1: The accuracy results for all - subject paradigm 24 Table 4 - 2: The cross - accuracy report 27 Table 4 - 3: Execution time per layer 27 Deep Learning in FPGA or Microcontroller to classify EEG/ECG signals for a brain training application T Pham 9 1 INTRODUCTION Using electroencephalography (EEG) signals in neuroscience and brain disease diagnosis was started in the first half of the twentieth century Even today, the principles of operation have unchanged , but the field of study for electroencephalography signals has now considerably broadened al ong with the development of science and technology Many patterns were found between electroencephalography signals and studies of motor activity, mental state , and brain activity However, the valuable information extracted from the electroencephalography is still limited Science in this area is still in the early stages of development The evolution of machine learning has made significant strides in the previous ten years, influencing several industries, including signal processing for EEG Among them , neural network (NN) is a flourish ing tool for working with EEG signals Thousands of publications about NN applications for EEG signals were published in various a reas, such as diagnosing diseases, lie recognition, research ing the physiology process, image classification, control artifact s , etc However, the research on implementing NN for EEG application on embedded systems is limited This thesis will deal with designing NN for EEG classification and implementing NN on an emb edded system as the STM32 microcontroller 1 1 Related work F or the comfort of understanding, the related work will be divided into two domains: the first is works about neural networks in EEG signal recognition/classification, and the second is works about ma chine learning on microcontrollers In [1] , the authors used machine learning to automatically detect alertness/drowsiness from the combination of EEG and electrooculography signals An efficient extremely learning machine (ELM) was employed for stat e classification The proposed algorithm performed a high accuracy and also computed in a fast speed The best state - detection accuracy when using ELM within radial basis function is 97 3% Xiaojun Bi et al [2] applied deep learning for EEG spectral images to detect the early Alzheimer’s disease EEG data was collected from 12 Alzheimer and Mild Cognitive Impairment patients to build a whole dataset with 12000 EEG spectral images with 32 × 32 reso lution The outcome was impressive with 95 04% accuracy, and it had a better performance compare d with SVM method Another work on proposing a Brain - Computer Interface system for mental sta te recognition based on real time EEG signals was introduced by L i et al in [3] A k - NN classifier built using the Self - Assessment Manikin ( SAM) model identified three different degrees of attent iveness Although the average accuracy peak is 57 03% but the method’s advanced aspects are low latency in computation and real - time EEG data recorded from six people on the cognitive tasks was analyzed and classified by the SVM algorithm in [4] The multiclass SVM classifier s w ere designed to detect five cognitive activities for each participant The average accuracy estimated for all candidates was 93 33 ± 8 16% However, there was not a standard paradigm for all participants, and this work is only used for studying with the less practical contr ibution Aci et al [5] developed a passive brain - computer interface using machine learning approaches for observing the attention states o f human being They designed the SVM model to classify three attention levels (focused, unfocused, and drowsy) , then made a comparison with two other methods as k - Nearest Neighbor and Adaptive Neuro - Fuzzy System The results were promising for the future w ork when the individual’s attention identification reached 96 7% (best), and 91 72% (average) accuracy Deep Learning in FPGA or Microcontroller to classify EEG/ECG signals for a brain training application T Pham 10 The volume of Internet of Things (IoT) devices is growing explosively with more than 75 billion connections to the Internet by 2025 as estimated [6] It makes the trending of shifting computation t o the edge devices is becoming reasonable Additionally, this pattern is applicable to machine learning methods, particularly for inference runs, which requires significantly less processing power than the earlier training phase [7] Several industry giants released the platforms to support implementing machine learning on embedded systems such as: Google has the Tensoreflow Lite, which supplies the powerful engines to convert the original models into the simplified and lighter version ; ARM also released a free library that is only compatible with their Cortex - M processors; even STMicroelectronics introduced the X - CUBE - AI extension for STM32CubeIDE software to deploy deep neural networks on STM 32 - bit microcontrollers feasibly A conv olutional neural network on STM Nucleo - L476RG for human presence detection was presented by Cerutti et al [8] They used the Cortex Microcontroller Software Interface Standard Neural Network (CMSIS - NN) library for maximizing the NN efficiency The network perfo r med 76 7% of accuracy while solely used 6 kB of RAM, and consumed 16 5 mW in steady mode A substance detector named MobileNet - Single Shot Detector (SSD) was introduced by Zhang et al [9] , and used the well - known Caffe framework in a deep convolutional NN That model was implemented on NanoPi2, using Samsung Cortex - A9 Quad - Core 1 4GHz , and 1 GB DDR3 RAM Emotion detection by a brac elet which could run multilayer NN was published by Magno et al [10] The power measurement proved that application could fit the mW power ARM Cortex M4F microcontroller The emotion was detected with 100% of accuracy surprisingly while using only 2% of available memory Several examples were presented in [ 11] to give the fundamental knowledge of tiny machine learning These projects were deployed on Arduino Nano 33 BLE Sense board, STM32F746G Discovery kit, and SparkFun Edge board There are a bunch of works on implementing NN on embedded systems, howev er most of them used the robust edge devices (e g , Cortex - A9, Raspberry PI, Cortex - A53) A great work in [7] showed the effort to compensate the l ack of works on mainstream microcontrollers 1 2 Project duration The time duration for this project is from 2 nd May, 2022 to 31 st August, 2022 (including holidays and weekends ) The time estimation is an approximate evaluation and could be affected by exteri ors ( hardware and software resources such as boards, devices for collecting the own dataset; or project scope , etc ) Deep Learning in FPGA or Microcontroller to classify EEG/ECG signals for a brain training application T Pham 11 2 BACKGROUND This chapter will supply the basic concepts about electroencephalography signal s , machine learning classifiers , neural networks , an d edge device computation A section on electroencephalography which includes definition and fundamental features will be presented Furthermore, there are parts describing the conventional classifying methods in machine learning Lastly , embedded deployment contexts such as number representations and quantization will be invest igated 2 1 Electroencephalograp h y Electroencephalography (EEG) is the study of capturing and figuring out the electrical activity generat ed from the brain’s surface When the brain receives the impact from senses such as sight, hearing, taste, etc it will produce the biological electrical signals which was transmitted through the neural system Electrical activity can be recorded with electrodes placed on the scalp; each electrode as considered as a channel will capture the electrical pulse in each specific area Depending on the application, the number of electrodes can be used in range of 2 to 512 The EEG is recorded and displayed as waveforms of varying frequenc y and amplitude measured in voltage [12] EEG consists of mainly 4 st andard pattern s: delta (0 5 - 4 Hz), theta (4 - 8 Hz), alpha (8 - 12 Hz), beta (13 - 30 Hz) as shown in Figure 2 - 1 Delta and theta signals are often monitored while human is in drowsy or asleep state Graphic chart shows the amplitudes varies from 0 5 – 1 5 mV and reaches several millivolts at peaks However, these values on scalp are within 10 – 100 μV regularly Figure 2 - 1 : 4 typical dominant brain normal rhythms [13] As proven, each frequency band for brain activity is identified with particular cognitive function There is a considerable quantity of information in EEG signal s which indicates the spatial, temporal, and spectral aspects These benefits make EEG an alternative option to consider in not only neur oscience but also clinical treatments and disease diagnosis However, in contrast, the EEG method also pays the cost of data processing complexity due to high dimension ality, non - stationary, and a low signa l - to - noise ratio Following the technology tendency, machine learning has been considered as a sufficient solution to engage in natural challenges of EEG approach Deep Learning in FPGA or Microcontroller to classify EEG/ECG signals for a brain training application T Pham 12 In o rder to determine the user’s mental state, raw EEG signals must be processed into the group of these signals The two basic steps of a pattern recogni tion approach that is typically used to accomplish this translation are as follows: • Feature extraction: th e initial signal processing phase, tries to characterize the EEG signals by a small number of relevant values referred to “features” [14] Such features should exclude noise and other irrelevant information when capturing the information included in EEG signals that is pertinent to describing the mental states to be identified The arrangement of all extracted feature s int o a single vector is called a feature vector • Classification: the second stage is where a class is assigned to a set of features derived from the signals This class is appropriate for the type of identified mental state Each classification algorithm is n amed as “classifier” For example, the imagined le ft - right hand movement labeled process is shown in Figure 2 - 2 There are two mental states (imagined right hand and imagined left - hand movements) are distinguished Band power features, or the strength of the EEG signal in a particular frequency range , are common characteristics that can be used to distinguish them from EEG signals The f ollowing step is using a Linear Discriminant Analysis (LDA) classifier to detect the states Figure 2 - 2 : EEG signal p rocessing pipeline [15] 2 2 Machine learning classifiers Machine learning frequently encounters categorization problems, where the model must give anticipated class labels to a set of input data Binary classification is used when there are only two classes fro m which to choose, and multi - class classification is used when there are more than two groups [16] The classification accuracy is good parameter for evaluating performance and one can get additional details through monitor a confusion matrix The confusion matrix plays a role as highlight potential issues such as whether the model frequently conflates t wo classes Each row of the matrix corresponds to the instances in an actual class while the columns represent the classes that the model predicted [17] The illustration of a confusion matrix is shown in Figure 2 - 3 It is crucial that the input data must be balanced for categorization accuracy to be significant and pertinent It means that each class shou ld appear in the training dataset equally about time, number of samples, etc A bias toward one class maybe occurred through an imbalanced class distribution In the wor st scenario, the accuracy of model will only perform the underlying class distribution [18] There are some alternative indices using for measuring model’s performance as precision and recall Along with accuracy, precision and recall can also be estimated through true positives (TP), true negatives (TN), false positives (F P) and false negatives (FN) quantities Deep Learning in FPGA or Microcontroller to classify EEG/ECG signals for a brain training application T Pham 13 Figure 2 - 3 : Confusion matrix for 3 c lasses Th e following equations will be used for calculating each type of metric:
NATIONAL AND KAPODISTRIAN UNIVERSITY OF ATHENS SCHOOL OF SCIENCE DEPARTMENT OF INFORMATICS AND TELECOMMUNICATION MSc THESIS Tiny ML in Microcontroller to Classify EEG Signal into Three States Thuy T Pham Supervisor (or supervisors): Van-Tam Nguyen, Professor at Telecom ParisTech, France ATHENS SEPTEMBER 2022 MSc THESIS Tiny ML in Microcontroller to Classify EEG Signal into Three States Thuy T Pham S.N.: 7115192100007 Supervisor (or supervisors): Van-Tam Nguyen, Professor at Telecom ParisTech, France ABSTRACT This thesis investigates how to implement an own-built neural network for electroencephalography signals classification on an STM32L475VG microcontroller unit The original dataset is analyzed and processed to better understand the brain signals There is a comparison between three machine learning algorithms (linear support vector machine, extreme gradient boosting, and deep neural network) in three testing paradigms: specific-subject, all-subject, and adaptable to select the most appropriate approach for deploying on the microcontroller The implementation procedure with detailed notation is presented, and the inference is also performed to feasible observation Finally, possible improvement solutions are proposed within a clear demonstration SUBJECT AREA: Signal processing, machine learning, embedded system KEYWORDS: electroencephalography, artificial neural network, STM32 microcontroller, SVM, XGBoost Erasmus Mundus Joint Master’s Degree “SMART Telecom and Sensing NETworks” (SMARTNET) (2019/2021 intake) Aston University, Triangle, B4 7ET / Birmingham, UK Email: aipt_smartnet@aston.ac.uk / Web-site: smartnet.astonphotonics.uk/ Acknowledgement This Master Thesis has been accomplished in the framework of the European Funded Project: SMART Telecom and Sensing Networks (SMARTNET) - Erasmus+ Programme Key Action 1: Erasmus Mundus Joint Master Degrees – Ref Number 2017 – 2734/001 – 001, Project number - 586686-EPP-1-2017-1-UK-EPPKA1-JMD-MOB, coordinated by Aston University, and with the participation of Télécom SudParis, member of IP Paris and National and Kapodistrian University of Athens ACKNOWLEDGEMENT Khi ngồi viết dịng này, em biết hành trình hai năm với SMARTNET tới hồi kết Thời gian trôi qua chẳng chờ đợi ai, kỷ niệm thứ tồn mãi! Cảm ơn SMARTNET tin tưởng cho em hội để thực hóa ước mơ Em khơng nghĩ lại “sống sót” qua thử thách khắc nghiệt suốt khóa học này: khó khăn lúc làm visa, lo chỗ ăn ở; gần năm trời quanh quẩn với phịng dịch Covid, môn học mà em chưa tiếp xúc, đề thi dài 10, 11 trang giấy…^_^ Nhưng có lẽ, thử thách rèn luyện em trưởng thành tự tin sống SMARTNET khơng chương trình học đơn mà với em gia đình, nơi có bạn từ khắp nơi giới chia sẻ khoảnh khắc, có thầy thân thiện cởi mở nhiệt tình với sinh viên Cảm ơn tất thuộc SMARTNET! Em muốn gửi lời cảm ơn chân thành tới thầy Văn-Tâm Nguyễn, người hướng dẫn định hướng cho em suốt trình thực thực tập Không kiến thức học thuật mà kiến thức đời sống, thầy chia sẻ nhiệt tình với em, giúp em khơng cảm thấy tự ti nghiên cứu Bên cạnh đó, em xin cảm ơn hỗ trợ tận tình từ thầy môn COMELEC, Telecom Paris; đặc biệt thầy Germain Pham – người luôn truyền lượng tích cực, dạy em kiến thức hữu ích đề tài Những lời tri ân em xin gửi tới tất thầy cô, điều phối chương trình SMARTNET, đặc biệt Telecom SudParis NKUA Chặng đường em khơng thể hồn thiện khơng có giảng dạy hỗ trợ từ người Cảm ơn người bạn sát cánh, động viên “lôi kéo” ăn lúc stress nặng nề Cảm ơn Juhyun Kim Rilwanu Kasno – làm bạn với hai người thành công lớn chương trình thạc sỹ :D Cuối cùng, xin cảm ơn gia đình Việt Nam, cảm ơn bố mẹ em trai ủng hộ lựa chọn con, quan tâm, lo lắng giúp có tinh thần mạnh mẽ lần xa nhà Con hoàn thành chặng đường để thăm gia đình ạ! Thank you so much! Merci beaucoup! Ευχαριστώ από τα βάθη της καρδιάς μου! SMARTNET PHẠM TRỌNG Thủy CONTENTS INTRODUCTION 1.1 Related work 1.2 Project duration 10 BACKGROUND .11 2.1 Electroencephalography 11 2.2 Machine learning classifiers 12 2.2.1 Linear Kernel Support Vector Machine (SVM) 13 2.2.2 Extreme Gradient Boost (XGBoost) 14 2.3 Artificial neural networks 14 2.3.1 The perceptron 14 2.3.2 Activation functions 15 2.3.3 Training the network 16 2.4 Machine learning on microcontrollers 16 2.4.1 Fixed-point quantization 17 2.4.2 The STM32L475 discovery kit 17 IMPLEMENTATION .19 3.1 Dataset 19 3.2 Data processing 20 3.3 Model training 21 3.4 Neural network deployment and inference 22 RESULTS .23 4.1 Specific-subject paradigm 23 4.2 All-subject paradigm 24 4.3 Adaptable paradigm 25 4.4 Inference 26 CONCLUSION AND FUTURE WORK 28 ABBREVIATIONS - ACRONYMS .29 ANNEX Ι 30 REFERENCES 35 LIST OF FIGURES Figure 2-1: typical dominant brain normal rhythms [13] 11 Figure 2-2: EEG signal processing pipeline [15] 12 Figure 2-3: Confusion matrix for classes 13 Figure 2-4: SVM mechanism illustration [19] 13 Figure 2-5: An ANN with multiple hidden layers 14 Figure 2-6: A perceptron schematic .15 Figure 2-7: Activation functions: Sigmoid (left), ReLu (right) 16 Figure 2-8: Post-quantization in an ANN layer .17 Figure 2-9: The B-L475E-IOT01A discovery kit [35] 18 Figure 3-1: The general work flow of implementing neural network on microcontroller 20 Figure 3-2: Feature extraction step 20 Figure 3-3: Neural network diagram 21 Figure 3-4: Validation flow overview [33] .22 Figure 3-5: Flowchart of implemented AI model on STM32L475VG 22 Figure 4-1: Accuracy in SVM method 23 Figure 4-2: Accuracy in XGBoost method .23 Figure 4-3: Accuracy varies with number of electrodes in SVM 24 Figure 4-4: Accuracy varies with number of electrodes in XGBoost 24 Figure 4-5: Accuracy of NN over each record of 1st subject 25 Figure 4-6: Accuracy of NN over each record of 2nd subject 25 Figure 4-7: Accuracy results of XGBoost in adaptable paradigm 26 Figure 4-8: Testing inference .27 Figure 5-1: Transformer model’s performance 28 LIST OF TABLES Table 2-1: Board specification .18 Table 3-1: Sample data 19 Table 4-1: The accuracy results for all-subject paradigm 24 Table 4-2: The cross-accuracy report 27 Table 4-3: Execution time per layer .27 Deep Learning in FPGA or Microcontroller to classify EEG/ECG signals for a brain training application INTRODUCTION Using electroencephalography (EEG) signals in neuroscience and brain disease diagnosis was started in the first half of the twentieth century Even today, the principles of operation have unchanged, but the field of study for electroencephalography signals has now considerably broadened along with the development of science and technology Many patterns were found between electroencephalography signals and studies of motor activity, mental state, and brain activity However, the valuable information extracted from the electroencephalography is still limited Science in this area is still in the early stages of development The evolution of machine learning has made significant strides in the previous ten years, influencing several industries, including signal processing for EEG Among them, neural network (NN) is a flourishing tool for working with EEG signals Thousands of publications about NN applications for EEG signals were published in various areas, such as diagnosing diseases, lie recognition, researching the physiology process, image classification, control artifacts, etc However, the research on implementing NN for EEG application on embedded systems is limited This thesis will deal with designing NN for EEG classification and implementing NN on an embedded system as the STM32 microcontroller 1.1 Related work For the comfort of understanding, the related work will be divided into two domains: the first is works about neural networks in EEG signal recognition/classification, and the second is works about machine learning on microcontrollers In [1], the authors used machine learning to automatically detect alertness/drowsiness from the combination of EEG and electrooculography signals An efficient extremely learning machine (ELM) was employed for state classification The proposed algorithm performed a high accuracy and also computed in a fast speed The best state-detection accuracy when using ELM within radial basis function is 97.3% Xiaojun Bi et al [2] applied deep learning for EEG spectral images to detect the early Alzheimer’s disease EEG data was collected from 12 Alzheimer and Mild Cognitive Impairment patients to build a whole dataset with 12000 EEG spectral images with 32 × 32 resolution The outcome was impressive with 95.04% accuracy, and it had a better performance compared with SVM method Another work on proposing a Brain-Computer Interface system for mental state recognition based on real time EEG signals was introduced by Li et al in [3] A k-NN classifier built using the Self-Assessment Manikin (SAM) model identified three different degrees of attentiveness Although the average accuracy peak is 57.03% but the method’s advanced aspects are low latency in computation and real-time EEG data recorded from six people on the cognitive tasks was analyzed and classified by the SVM algorithm in [4] The multiclass SVM classifiers were designed to detect five cognitive activities for each participant The average accuracy estimated for all candidates was 93.33±8.16% However, there was not a standard paradigm for all participants, and this work is only used for studying with the less practical contribution Aci et al [5] developed a passive brain-computer interface using machine learning approaches for observing the attention states of human being They designed the SVM model to classify three attention levels (focused, unfocused, and drowsy), then made a comparison with two other methods as k-Nearest Neighbor and Adaptive Neuro-Fuzzy System The results were promising for the future work when the individual’s attention identification reached 96.7% (best), and 91.72% (average) accuracy T.Pham Deep Learning in FPGA or Microcontroller to classify EEG/ECG signals for a brain training application The volume of Internet of Things (IoT) devices is growing explosively with more than 75 billion connections to the Internet by 2025 as estimated [6] It makes the trending of shifting computation to the edge devices is becoming reasonable Additionally, this pattern is applicable to machine learning methods, particularly for inference runs, which requires significantly less processing power than the earlier training phase [7] Several industry giants released the platforms to support implementing machine learning on embedded systems such as: Google has the Tensoreflow Lite, which supplies the powerful engines to convert the original models into the simplified and lighter version; ARM also released a free library that is only compatible with their Cortex-M processors; even STMicroelectronics introduced the X-CUBE-AI extension for STM32CubeIDE software to deploy deep neural networks on STM 32-bit microcontrollers feasibly A convolutional neural network on STM Nucleo-L476RG for human presence detection was presented by Cerutti et al [8] They used the Cortex Microcontroller Software Interface Standard Neural Network (CMSIS-NN) library for maximizing the NN efficiency The network performed 76.7% of accuracy while solely used kB of RAM, and consumed 16.5 mW in steady mode A substance detector named MobileNet-Single Shot Detector (SSD) was introduced by Zhang et al [9], and used the well-known Caffe framework in a deep convolutional NN That model was implemented on NanoPi2, using Samsung Cortex-A9 Quad-Core 1.4GHz, and GB DDR3 RAM Emotion detection by a bracelet which could run multilayer NN was published by Magno et al [10] The power measurement proved that application could fit the mW power ARM Cortex M4F microcontroller The emotion was detected with 100% of accuracy surprisingly while using only 2% of available memory Several examples were presented in [11] to give the fundamental knowledge of tiny machine learning These projects were deployed on Arduino Nano 33 BLE Sense board, STM32F746G Discovery kit, and SparkFun Edge board There are a bunch of works on implementing NN on embedded systems, however most of them used the robust edge devices (e.g., Cortex-A9, Raspberry PI, Cortex-A53) A great work in [7] showed the effort to compensate the lack of works on mainstream microcontrollers 1.2 Project duration The time duration for this project is from 2nd May, 2022 to 31st August, 2022 (including holidays and weekends) The time estimation is an approximate evaluation and could be affected by exteriors (hardware and software resources such as boards, devices for collecting the own dataset; or project scope, etc.) T.Pham 10