Thông tin tài liệu
UTILIZING EEG SIGNAL IN MUSIC
INFORMATION RETRIEVAL
ZHAO WEI
B.Sc. OF ENGINEERING
UNIVERSITY OF ELECTRONIC SCIENCE AND TECHNOLOGY OF CHINA
2006
A THESIS SUBMITTED
FOR THE DEGREE OF MASTER OF SCIENCE
DEPARTMENT OF COMPUTER SCIENCE
SCHOOL OF COMPUTING
NATIONAL UNIVERSITY OF SINGAPORE
2010
Abstract
Despite significant progresses in the field of music information retrieval (MIR),
grand challenges such as the intention gap and the semantic gap still exist. Inspired
by the current successes in the Brain Computer Interface (BCI), how to utilize
electroencephalography (EEG) signal to solve the problems of MIR is investigated
in this thesis. Two scenarios are discussed respectively: EEG-based music emotion
annotation and EEG-based domain specific music recommendation. The former
project addresses the problem that how to classify music clips to different emotion
categories based on audiences’ EEG signal when they listen to the music. The
latter project presents an approach to analysis sleep quality from EEG signal as
a component of an EEG-based music recommendation system which recommends
music according to the user’s sleep quality.
i
Acknowledgement
This thesis would not have been possible without the support of many people.
I wish to express my greatest gratitude to my supervisor, Dr. Wang Ye who offered
valuable support and guidance since I started my study in School of Computing. I
also owe my gratitude to Dr. Tan from Singapore General Hospital for her professional suggestions about music therapy, to Ms. Shi Dongxia of National University
Hospital for her generous help in annotating the sleep EEG data.
I would like to thanks Wang Xinxi, Li Bo and Anuja for their assistance and help
in the system implementation of my work. Special thanks also to all participants
involved in the EEG experiments: Ye Ning, Zhang Binjun, Lu Huanhuan, Zhao
Yang, Zhou Yinsheng, Shen Zhijie, Xiang Qiaoliang, Ai Zhongkai, et al.
I am deeply grateful to my beloved families, for their consistent support and endless love. To support my research, my wife even has attached electrodes on her
scalp during sleep for a week.
Without the support of those people, I would not be able to finish this thesis.
Thanks you so much!
ii
Contents
Abstract
i
Acknowledgement
ii
Contents
iii
List of Publications
v
List of Figures
vi
List of Tables
vii
1 Introduction
1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Organization of the Thesis . . . . . . . . . . . . . . . . . . . . . . .
2 EEG-based Music Emotion Annotation System
2.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Emotion Recognition in Affective Computing . . . . . .
2.3 Physiology-based Emotion Recognition . . . . . . . . .
2.3.1 General Structure . . . . . . . . . . . . . . . . .
2.3.2 Emotion Induction . . . . . . . . . . . . . . . .
2.3.3 Data Acquisition . . . . . . . . . . . . . . . . .
2.3.4 Feature Extraction and Classification . . . . . .
2.4 A Real-Time Music-evoked Emotion Detection System
2.4.1 Introduction . . . . . . . . . . . . . . . . . . . .
2.4.2 System Architecture . . . . . . . . . . . . . . .
2.4.3 Demonstration . . . . . . . . . . . . . . . . . .
2.5 Current Challenges and Perspective . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
1
1
3
4
4
6
7
8
9
10
14
16
16
18
24
26
3 Automatic Sleep Scoring using EEG Signal-First Step Towards a
Domain Specific Music Recommendation System
29
iii
CONTENTS
3.1
3.2
3.3
3.4
3.5
Introduction . . . . . . . . . . . . . . . .
3.1.1 Music Recommendation according
3.1.2 Normal Sleep Physiology . . . . .
3.1.3 Paper Objectives . . . . . . . . .
3.1.4 Organization of the Thesis . . . .
Literature review . . . . . . . . . . . . .
3.2.1 Manual PSG analysis . . . . . . .
3.2.2 Computerized PSG Analysis . . .
Methodology . . . . . . . . . . . . . . .
3.3.1 Feature Extraction . . . . . . . .
3.3.2 Classification . . . . . . . . . . .
3.3.3 Post Processing . . . . . . . . . .
Experiment Results . . . . . . . . . . . .
Conclusions . . . . . . . . . . . . . . . .
iv
. . . . .
to Sleep
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
Quality
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
. . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
31
32
33
34
35
37
37
37
39
40
41
42
45
50
4 Conclusion and Future work
51
4.1 Content-based Music Similarity Measurement . . . . . . . . . . . . 53
Bibliography
58
List of Publications
Automated Sleep Quality Measurement using EEG signal-First Step
Towards a Domain Specific Music Recommendation System,
Wei Zhao, Xinxi Wang and Ye Wang, ACM Multimedia International Conference
(ACM MM), 25-29th October 2010, Firenze, Italy.
v
List of Figures
2.1
2.2
2.3
2.4
2.5
2.6
2.7
2.8
2.9
2.10
Recognize Musical Emotion from Acoustic Features of Music
Recognize Musical Emotion from Audience’s EEG Signal . .
System Architecture . . . . . . . . . . . . . . . . . . . . . .
Human Nervous System . . . . . . . . . . . . . . . . . . . .
EEG signal Acquisition Experiments . . . . . . . . . . . . .
Physiology-based Music-evoked Emotion Detection System .
Electrode Position in the 10/20 International System . . . .
Feature Extraction and Classification Module . . . . . . . .
Music Game Module . . . . . . . . . . . . . . . . . . . . . .
3D Visualization Module . . . . . . . . . . . . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
5
5
8
12
15
19
21
22
23
25
3.1
3.2
3.3
3.4
3.5
3.6
Physiology-based Music Rating Component . . . . . . .
Typical Sleep Cycles . . . . . . . . . . . . . . . . . . . .
Traditional PSG system with Three Physiological Signals
Band Power Features and Sleep Stages . . . . . . . . . .
Position of Fpz and Cz in the 10/20 System . . . . . . .
Experiment Over the Recording, st7052j0 . . . . . . . . .
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
32
34
36
41
46
49
4.1
Content-based Music Recommendation Component . . . . . . . . .
54
vi
.
.
.
.
.
.
.
.
.
.
.
.
List of Tables
2.1
2.2
2.3
Targeted Emotion and Associated Stimuli . . . . . . . . . . . . . .
Physiological Signals related to Emotion . . . . . . . . . . . . . . .
Extracted Feature and Classification Algorithm . . . . . . . . . . .
11
14
17
3.1
3.2
3.3
3.4
3.5
3.6
Accuracy of SVM Classifier in 10-fold Cross-validation
Confusion Matrix on st7022j0 . . . . . . . . . . . . . .
Confusion Matrix on st7052j0 . . . . . . . . . . . . . .
Confusion Matrix on st7121j0 . . . . . . . . . . . . . .
Confusion Matrix on st7132j0 . . . . . . . . . . . . . .
Accuracy of SVM and SVM with Post-processing . . .
47
47
47
48
48
48
vii
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
.
Chapter 1
Introduction
1.1
Motivation
With the rapid development of digital music industry, music information retrieval
(MIR) has received much attention in last decades. Over years of development,
however, critical problems still remain, such as the intention gap between users
and systems and the semantic gap between low-level features and high-level music
semantics. These problems significantly influence the performance of current MIR
systems.
User feedback plays a important role in Information Retrieval (IR) systems.
It has been presented as an efficient method to improve the performance of IR
system by conducting relevance assessment [1]. This technique is also useful for
MIR systems. Recently, physiological signal was presented to be new approach
1
Chapter 1 Introduction
2
to continuously collects reliable information from users without interrupt towards
them [2]. However, physiological signals have received few attentions in the MIR
community.
For the last two years, I have been conducting research about Electroencephalography (EEG) signal analysis and its applications in MIR. My decision to choose
this topic is also inspired by the successful stories of Brain Computer Interface
(BCI) [3]. Two years ago, I was surprised by the amazing applications of BCI
technology such as P300 speller [4], and motor-imaginary-controlled robot [5], etc.
At that time I came up the idea that utilizing EEG signal in traditional MIR systems. I have been trying to find a scenario where EEG signal can be integrated
in a MIR system. So far two projects have been conducted: EEG-based musical
emotion recognition and EEG-assisted music recommendation system.
The first project is musical emotion recognition from audience’s EEG feedback.
Music emotion recognition is an important but challenging task in music information retrieval. Due to the well-known semantic gap problem, musical emotion
cannot be accurately recognized from the low level features extracted from music
items. Consequently I try to recognize musical emotion from Audience’s EEG signal instead of music item. An online system was built to demonstrate this concept.
Audience’s EEG signal is captured when s/he listens to the music items. Then
Alpha frontal power feature is extracted from EEG signal. A SVM classifier is used
to classify each music items in three musical emotions: happy, sad, and peaceful.
In the second project, an EEG-assisted music recommendation system is proposed. This work addresses a healthcare scenario, music therapy, that utilizing
Chapter 1 Introduction
3
music to heal people who suffer from sleep disorders. Music therapy research has
indicated that music does have beneficial effects on sleep. During the process of
music therapy, people are asked to listen to a list of music, which is pre-selected
by music therapist. In spite of its clear benefits towards sleep quality, current
approach is difficult to be widely used because it is a time consuming task for
music therapist to produce a personalized music list. Based on this observation,
an EEG-assisted music recommendation system was proposed, which automatically recommends music for user according to his sleep quality estimated from
EEG signal. As the first attempt, how to measure sleep quality from EEG signal
is investigated. This work was recently selected for poster presentation in ACM
Multimedia 2010.
1.2
Organization of the Thesis
The thesis is organized as follows. EEG-based Music Emotion Annotation System
is presented in detail in Chapter 2. Chapter 3 discuss the EEG-assisted music recommendation system. Future work and Perspective are summarized in Chapter 4.
Chapter 2
EEG-based Music Emotion
Annotation System
2.1
Introduction
Like genre and culture, emotion is one important factor of music which has attracted much attention in MIR community. Musical emotion recognition is usually
regarded as a classification problem in earlier studies. To recognize the emotion
of one music clip, low-level features are extracted and fed into a classifier which
is trained based on labeled music clips [6], as presented in Figure 2.1. Due to the
semantic gap problem, low-level features, such as MFCC, cannot reliably describe
the high level factors of music. In this chapter I explore an alternative approach
which recognizes music emotion from human’s physiological signal instead of low-
4
Chapter 2 EEG-based Music Emotion Annotation System
5
Figure 2.1: Recognize Musical Emotion from Acoustic Features of Music
Figure 2.2: Recognize Musical Emotion from Audience’s EEG Signal
level feature of music item, as described in Figure 2.2.
A physiology-based music emotion annotation approach is investigated in this
part. The research problem is how to recognize human’s perceptive emotion from
physiology signal while he or she listen to emotional music. As human emotion
detection was first emphasized in the affective computing community [7], we briefly
introduce affective computing in Chapter 2.2. A survey about emotion detection
from physiology signal is given in Chapter 2.3. Our research prototype, a online
music-evoked emotion detection system, is presented in Chapter 2.4. The challenge
and perspective are discussed in Chapter 2.5.
Chapter 2 EEG-based Music Emotion Annotation System
2.2
6
Emotion Recognition in Affective Computing
Emotion is regarded as a complex mental and physiological state associated with a
large amount of feeling and thought. When human communicate with each other,
their behavior considerably depends on their emotion state. Different emotion
states, happy, sad and disgust always influence the decision of human and the
efficiency of the communication. To efficiently cooperate with others, people need
to take account of this subjective factor of human, the emotion. For example, a
salesman talks with many people every day. To promote his product, he has to
adjust his communication strategy in accordance with the respondent emotion of
consumers. The implication is clear to all of us: emotion plays a key role in our
daily communication.
Since human is subject to their emotion states, the efficiency of communication
between human and machine is also affected by the user’s emotion. Obviously it
is beneficial that if the machine can response differently according to the user’s
emotion, as what a salesman has to do. There is no doubt that taking account
of human emotion can considerably improve the performance for human machine
interaction [7, 8, 9]. But so far few emotion-sensitive systems have been built. The
problem behind this is that emotion is generated by the mental activity which is
hidden in our brain. Because of the ambiguous definition of emotion, it is difficult
to recognize emotion fluctuation accurately. Since automated recognition of human
emotion has a big impact and implies a lot of applications in Human Computer
Chapter 2 EEG-based Music Emotion Annotation System
7
Interaction, it has attracted a large body of attention from researcher in computer
science, psychology, and neuroscience.
There are two main approaches to recognize the emotion: Physiology-based
Emotion Recognition and Facial&Vocal-based Emotion Recognition. On
the one hand, some researchers have obtained many results to detect emotion from
facial image, human voice [10]. These face and voice signals, however, depends on
human explicit and deliberately expression of emotion [11]. With the advances
in sensor technology, on the other hand, physiological signal are introduced to
recognize the emotion. Since emotion is results of human intelligence, it is believed
that emotion can be recognized from physiology signal, which is generated by
human nervous system, the source of human Intelligence [12]. In contrast with
face and voice, the main advantage of physiology approach is that emotion can
be analyzed from physiological signal without subject’s deliberately expression of
emotion.
2.3
Physiology-based Emotion Recognition
Current approaches of physiology-based emotion detection are investigated in this
part. As discussed in Chapter 2.3.1, an typical emotion detection system consists
of four components, emotion induction, data acquisition, feature extraction, and
classification. The methods and algorithms employed in these components are
respectively summarized in Chapter 2.3.2, Chapter 2.3.3, and Chapter 2.3.4.
Chapter 2 EEG-based Music Emotion Annotation System
8
Figure 2.3: System Architecture
2.3.1
General Structure
To detect emotion states from physiological signal, the general approach can be
summarized as the answers for four questions as follow:
a. What emotion states are going to be detected?
b. What stimuli are used to evoke the specific emotion states?
c. What physiological signals are collected while the subject obtains the
stimuli?
d. Given the signals, how to extract feature vector and do the classification?
As described in Figure 2.3, a typical physiology-based emotion recognition system
consists of four components: emotion induction module; data acquisition module;
feature extraction & classification module. Each component addresses one question
as given above.
Emotion induction component is responsible to evoke the specific emotion by
Chapter 2 EEG-based Music Emotion Annotation System
9
using the emotional stimuli. For example, emotion induction components may play
back peaceful music or display a picture of traffic accident to help the subject to
reach the specific emotion stage.
While the subjects obtain the stimuli, data acquisition model keep on collecting
the signal from subject. The sensors attached on subject’s body are used to collect
physiological signal during the experiment. Different kind of sensor is used to
collect the specific physiological signals such as Electroencephalography (EEG),
Electromyogram (EMG), Skin conductivity response (SCR), and Blood Volume
Pressure (BVP) etc. For example to collect the EEG signal, the subject is usually
required to wear an electrode caps in the experiment.
After several runs of experiment, many physiological signal fragments can be
collected to build a signal data set. Given such a data set, the feature extraction
and classification component is applied to classify EEG segment into different
emotion categories. First the data set is divided into two parts: training set and
testing set. Then the classifier is built based on the training data set.
2.3.2
Emotion Induction
Emotion can be categorized into several basic states such as fearful, angry, sad,
disgust, happy, and surprise [13]. To recognize emotion states, the emotion have
to be define clearly in the beginning. The categorization of emotion varies in
different papers. In our system, we recognize three emotion stages: sad, happy,
and peaceful.
Chapter 2 EEG-based Music Emotion Annotation System
10
Once the emotion categorization is defined, another problem arises: how to
induce the subject to obtain the specific emotion states. Currently, the popular
solution is to provide some emotional cues to help the subject experience the emotion. Many stimuli are presented for such purpose, such as sound clips, music
item, picture and even movie clips. These stimuli can be categorized into four
main types:
a. Subject obtain the emotion by imagine.
b. Visual stimuli.
c. Audition stimuli.
d. Combination of visual and audition stimuli.
The emotion and stimuli presented in earlier papers are summarized in Table
2.1.
2.3.3
Data Acquisition
The human nervous system can be divided into two parts: the Central Nervous
System (CNS) and the Peripheral Nervous System (PNS). As described in
Figure 2.4, CNS contains the majority of the nervous system and consists of brain
and spinal cord. PNS extends the CNS and connect the CNS to the limbs and
other organs. Human nervous system is the source of physiological signal, and thus
physiology signals can be categorized into two categories: CNS-generated signals
and PNS-generated signals. The details of those two kinds of physiology signals
are discussed in the following part.
Chapter 2 EEG-based Music Emotion Annotation System
11
Table 2.1: Targeted Emotion and Associated Stimuli
Categorization of Emotion Stimuli to Evoke Emotion
Authors
Disgust, Happiness, Neutral Images from International Affec- [14], [15]
tive Picture System (IAPS)
Disgust, Happiness, Neutral (1)Images (2)Self-induced Emo- [16]
tion (3)Computer Game
Positive Valence v.s. High (1)Self-induced Emotion by imag- [17]
Arousal; Positive Valence ining past experience (2)Images
v.s. Low Arousal; Negative from IAPS (3)Sound clips from
Valence v.s. High Arousal; IADS (4)the Combination of
Negative Valence v.s. Low above Stimuli
Arousal;
Joy, Anger, Sadness, Plea- Music selected from Oscar’s [18], [19], [20]
sure
movie soundtracks
No Emotion, Anger, Hate, Self-induced Emotion
[21], [22], [23],
Grief, Rove, Romantic
[24]
Love, Joy, Reverence
Joy, Anger, Sadness, Plea- Music selected by subjects
[25], [26], [27],
sure
[28], [29]
Amusement, Contentment, Images from IAPS
[30]
Disgust, Fear, No emotion
(Neutrality), Sadness
5 emotions on two emo- Images from IAPS
[31]
tional dimensions, valence
and arousal
Chapter 2 EEG-based Music Emotion Annotation System
Figure 2.4: Human Nervous System
[32]
12
Chapter 2 EEG-based Music Emotion Annotation System
13
Electromyogram (EMG) is the electric signal generated by muscle cells when
these cells are active or at rest. The EMG potential usually ranges from 50 mV to
30 mV. The typical frequency of EMG is about 7-20 Hz. Because face activity is
abundant and indicates human emotion, some researchers capture the EMG signal
from farcical muscle, and employ it in the emotion detection system [33].
Skin conductivity response (SCR) / Galvanic Skin Response (GSR) is
one of the most well studied physiological signals. It describes the change of levels
of sweat in the sweat glands. SCR is generated by sympathetic nervous system
(SNS) which is part of peripheral nervous system. Since SNS always becomes
active while the human feel stress, SCR is also related with the emotion.
Blood Volume Pressure (BVP) is the indicator of blood flaw, it measure the
force of blood pushing against blood vessel. BVP is measured by a unit called
Mm Hg (millimeters of mercury). Each time the heart pumps blood in the blood
vessel, resulting in a peak in BVP signal. Heart rate (HR) signal can be extracted
from BVP easily. BVP is also influenced by emotions and stress. Active feeling
such as anger, fear or happiness always increases the value of BVP signal.
Electroencephalography (EEG), the electric signal generated by neuron cells
can be captured by placing electrodes on the scalp, as described in Figure 2.5.
It has been proven that the difference of spectral power between left and right
brain hemispheres is an indicator of the fluctuations in emotions [34]. Specifically,
pleasant music causes a decrease in left frontal alpha power, whereas unpleasant
music elicits a decline of right frontal alpha power. Based on this phenomenon,
one feature called Asymmetric Frontal Alpha Power is extracted from EEG to
Chapter 2 EEG-based Music Emotion Annotation System
14
Table 2.2: Physiological Signals related to Emotion
Physiological signal
Authors
EEG
[17], [18],
[19], [20],
[31]
(1)EMG (2)GSR (3)Respiration (4)Blood [21], [22],
volume pressure
[23], [24]
(1)EMG (2)ECG/EKG (3)skin conductivity [25], [26],
(4)Respiration
[27], [28],
[29]
(1)Blood volume pulse (2)EMG (3)Skin [30]
Conductance Response (4)Skin Temperature
(5)Respiration
(1)EEG (2)GSR (3)Blood pressure (4)Respi- [15]
ration (5)Temperature
(1)Video recording (2)fNIRS (3)EEG [14]
(4)GSR (5)Blood pressure (6)Respiration
(1)EEG (2)GSR (3)Respiration (4)BVP [16]
(5)Finger temperature
recognize the emotion [35, 36, 37].
In spite of the physiological signals discussed above, Temperature of skin,
Respiration, and functional Near-Infrared Spectroscopy (fNIRS) are also
used to detect the emotion. The varieties of physiological signals, which are employed to detect emotion states in earlier works, are summarized in Table 2.2.
2.3.4
Feature Extraction and Classification
To decode the emotion from physiological signals, many features have been presented. Two popular features are spectral density in frequency domain and statistical information in time domain.
Chapter 2 EEG-based Music Emotion Annotation System
(a) EEG Electrode Cap and EEG Amplifier
(b) Experiment conducted on Zhao (c) Experiment conducted on Yi Yu
Wei
(d) Experiment conducted on Zhao Yang
Figure 2.5: EEG signal Acquisition Experiments
15
Chapter 2 EEG-based Music Emotion Annotation System
16
EEG signals are usually divided into 5 frequency bands: delta (1-3 Hz), theta
(4-7 Hz), alpha (8-13 Hz), beta (14-30 Hz), and gamma (31-50 Hz). One common
feature of EEG is the average spectral density on specific frequency band. Furthermore, subtract between different channel and ratio of different bands is also
used as feature vectors.
Besides, the signals generated by PNS just cover a small range of frequency
band. Consequently, the signal such as blood pressure, respiration, and skin conductivity cannot be divided into several frequency bands. Usually time-domainbased features are extracted from these signals, such as peak rate, statistical mean,
and variance.
The extracted feature and classification algorithms used in previous papers are
summarized in Table 2.3.
2.4
A Real-Time Music-evoked Emotion Detection System
2.4.1
Introduction
Advances in sensor and computing technologies have made it possible to capture
and analyze human physiological signals in different applications. These capabilities unpack a new scenario wherein the subject’s emotions evoked by external
stimuli such as music can be detected and visualized in real-time. Two approaches
Chapter 2 EEG-based Music Emotion Annotation System
Table 2.3: Extracted Feature and Classification Algorithm
Features
Classification
(1)averaged spectral power of 6 frequency (1)Native Bayesian classifier
bands (2)wavelet coefficients (CWT) of (2)Fisher Discriminant Analyheart rate (3)the mean, variance, mini- sis
mum and maximum of peripheral signals
To select the best feature, several meth- (1)Naive Bayesian classifier
ods are applied: (1)filter (ANOVA, Fisher (2)Discriminant
Analysis
based and FCBF) (2)wrapper (SFFS) fea- (3)SVM (4)Relevance Vector
ture selection algorithms
Machines (RVM)
based on three EEG channels, Fpz and binary linear FDA (Fisher’s
F3/F4, following feature are extracted: Discriminant Analysis) classi(1)alpha, beta, alpha and beta, beta/al- fier
pha power (2)beta power / alpha power
(1)the means of raw signal; (2)the stan- Sequential Floating Forward
dard deviation of the raw signal; (3)the Search (SFFS) is used to select
means of the absolute values of the first the best features from feature
differences of the raw signals; (4)the space. Finally, three strategies
means of the absolute values of the first are presented to do the clasdifferences of the normalized signal; (5)the sification task: (1)Sequential
means of the absolute values of the second floating forward search (SFFS)
differences of the raw signal; (6)the means feature selection with K-NN;
of the absolute value of the second differ- (2)Fisher Projection (FP) with
ence of the normalized signal.
MAP classification; (3)A hybrid SFFS-FP method.
Eleven feature extracted from signal
(1)Fisher projection matrix
(2)SFFS (3)K-NN
Many methods are investigated to find the (1)linear discriminant function
best features from feature space: (1)analy- (LDF) (2)k-nearest neighbors
sis of variance (ANOVA) (2)sequential for- (KNN) (3)multiplayer percepward selection (SFS) (3)sequential back- tron (MLP)
ward selection (SBS) (4)PCA (5)fisher
projection
30 feature values are extracted from five (1)Fisher discriminant classisignals
fier (2)SVM
asymmetry frontal alpha power over 12 multiplayer perceptron classielectrode pairs
fier (MLP)
(1)asymmetry frontal alpha power over SVM
12 EEG electrode pairs (2)spectral power
density of 24 EEG channels
(1)asymmetry frontal alpha power over hierarchical SVM
12 EEG electrode pairs (2)spectral power
density of 24 EEG channels
17
Authors
[15]
[16]
[17]
[21],
[24]
[22],
[23]
[25],
[27],
[29]
[30]
[18]
[19]
[20]
[26],
[28],
Chapter 2 EEG-based Music Emotion Annotation System
18
have been identified to make use of these physiological signals in multimedia systems. First, physiological signals can be visualized continuously while the subject
interacts with a multimedia system. Second, the physiological signals can be used
as a control message in applications such as game. Our system is designed to combine these two approaches and to demonstrate an application scenario of real-time
emotion detection. EEG signals generated as a response to the musical stimuli are
captured to detect the subject emotion states. This information is then used to
control a simple emotion-based music game. While the subject plays the music
game, his EEG is visualized on a 3D head model which serves as a synchronized
feedback for monitoring the subject. In such a case, our system provides a real-time
tool to monitor the subject and can serve as a useful input for music therapists,
for example.
2.4.2
System Architecture
The proposed system is shown in Figure 2.6 which shows how the four modules
are connected. The data acquisition and analysis modules together constitute the
music-evoked emotion detection subsystem.
Before providing the stimuli to evoke the subject’s emotion, the subject is asked
to wear an electrode cap which consists of 40 electrode channels. Each channel
captures EEG signals continuously. To perform real-time analysis, the EEG signals
are collected by the signal acquisition module. The module buffers the continuous
signals and feeds them as smaller EEG segments of 1s in duration, into the analysis
module. The analysis module calculates the spectral power density and the frontal
Chapter 2 EEG-based Music Emotion Annotation System
19
Figure 2.6: Physiology-based Music-evoked Emotion Detection System
alpha power feature from each EEG segment. The frontal alpha power feature is
discussed in details in following paragraphs.
Using these features followed by a SVM classifier, subject emotions are classified
into three states: happy, sad and peaceful. Finally, the classification result is sent
to the music game module to drive the game and the spectral powers of each
channel and emotions are fed into the 3D module for visualization.
Data Acquisition Module
To capture EEG signals, we have used products of NeuroScan, Quik-Caps and
NuAmps. Quik-Caps consist of 40 electrodes located on the head of the subject,
in accordance with the 10-20 system standards [38]. The electrical signals captured
Chapter 2 EEG-based Music Emotion Annotation System
20
by the electrodes are amplified by NuAmps. The sampling rate of the EEG signal
is 500 Hz. Since the effective frequency range of EEG is from 1 to 50 Hz, the
EEG signals are first band-pass filtered to retain only components between 1 and
200 Hz. The filtered EEG signals are continuously sent from the data acquisition
module to the analysis module.
Frontal Alpha Power Feature
The analysis module consists of two main components: feature extraction and
classification. To detect the music-evoked emotion from EEG, we have used the
asymmetry features commonly used in the physiological community [39]. It has
been proven that the difference of spectral power between left and right brain
hemispheres is an indicator of the fluctuations in emotions. Specifically, pleasant
music causes a decrease in left frontal alpha power, whereas unpleasant music elicits a decline of right frontal alpha power [40]. In comparison to most existing BCI
systems, we have not used any artifact rejection/removal method in our system.
The rationale is that artifacts usually have very similar effects on both the electrodes, which are symmetrically located on two hemispheres. Asymmetric features
are subtractions between symmetric electrode pairs thus compensating artifacts
caused by eye blinking for example [40]. Since 8 selected electrodes are symmetrically located on frontal lobe in our electrode cap, 4 pairs of electrodes can be
used to calculate the asymmetry features: Fp1-Fp2, F7-F8, F3-F4 and FC3-FC4.
The position of these 8 electrodes is illustrated in the Figure 2.7. EEG signals are
usually divided into 5 frequency bands: delta (1-3 Hz), theta (4-7 Hz), alpha (8-13
Hz), beta (14-30 Hz) and gamma (31-50 Hz). The averaged differential spectral
Chapter 2 EEG-based Music Emotion Annotation System
21
Figure 2.7: Electrode Position in the 10/20 International System
power over the alpha band is calculated as a feature from each electrode pair.
The dimension of the resulting feature vector is 4. Using this asymmetric feature
vector, emotion detection becomes a multi-class classification problem.
SVM Classifier
In the beginning of the music game, the subject is required to listen to 3 music
clips associated with 3 emotion states (happy, sad and peaceful). The evoked EEG
features are used as training data to build an SVM model. This model is then used
to predict the emotion state of the incoming EEG features in real-time. Libsvm
Chapter 2 EEG-based Music Emotion Annotation System
22
Figure 2.8: Feature Extraction and Classification Module
was used to implement the training and prediction [41]. Four-dimension feature
is extracted from each 1-second EEG segment. Existing kernels in the Libsvm
package are used to conduct the experiment. The classifier is trained based on a
6-minute EEG recording, which covers each emotion for 2 minutes.
We noticed other offline emotion classification systems (e.g. [42]) using much
higher dimension of feature vectors. Unfortunately, those offline systems cannot
be simply extended to a real-time system with acceptable performance. To reduce
the duration between training data collection and real-time prediction, we have
implemented a simple GUI (see Figure 2.8) to make the training process more
convenient and efficient. This has mitigated the performance degradation of the
real-time system, although it cannot solve the problem completely.
Chapter 2 EEG-based Music Emotion Annotation System
23
Figure 2.9: Music Game Module
Music Game Module
As in Figure 2.9, the game module has two main functions: to playback music
for evoking the required emotion state and to visualize emotion states transition
in real-time. The interface of the game is simple yet functional. This module,
however, needs to be improved for real life applications, such as music therapy.
3D Visualization Module
As Figure 2.10, the 3D visualization module displays the spectral power of each
EEG channel with different colors on a 3D head model which adopted and modified
Chapter 2 EEG-based Music Emotion Annotation System
24
from an open source project, Tempo [43].
The spectral energy changes of each EEG channels are displayed with different
colors on a 3D head model. We believe that 3D visualization is more intuitive to
human beings and this could be a useful feedback to experiment conductors. Since
visual patterns are friendlier to human eyes than the decimal numbers, an intuitive
illustration of the EEG changes can also be gained.
By observing the classification results and the EEG energy visualized in the
3D module, we can monitor the performance of the proposed approach during
the whole experiment. For example, the classifier might produce a wrong label
after the subject move head slightly. Therefore, which events might influence the
accuracy of proposed system can be distinguished. This kind of information could
be useful to improve the proposed system in the future work.
2.4.3
Demonstration
We proposed a research prototype to detect music-evoked emotion states in realtime using EEG signals synchronized with two visualization modules. We also
show its potential applications such as music therapy. As a start of the project,
we have re-implemented an offline system described in [42] and have achieved an
accuracy of up to 93% based on k-fold cross-validation, which is similar to the
reported performance.
However, the accuracy drops to randomly guess (about 35% in 3-class classifi-
Chapter 2 EEG-based Music Emotion Annotation System
Figure 2.10: 3D Visualization Module
25
Chapter 2 EEG-based Music Emotion Annotation System
26
cation) in online prediction.
The original offline system employs a 60-dimension feature extracted from whole
head area. We have then modified the approach by extracting features only from
the frontal lobe. Therefore, a 4-dimentions feature is extracted from 8 EEG channels of frontal lobe, as described in the Figure 2.7.
With the reduced features, we have managed to improve the prediction accuracy
based on our preliminary evaluations, which is discussed in details in Chapter 2.5.
2.5
Current Challenges and Perspective
Many papers have been published to recognize emotion from physiological signals.
To the best of our knowledge, no one has succeeded to extend their algorithm
into practical application. Although the accuracy of emotion recognition comes
up to 90% in the experiment of cross-validation, few works can obtain acceptable
accuracy in prediction. Based on the results in our experiment, the accuracy of
emotion recognition varies considerable under different validation strategy such as
prediction, k-fold cross-validation, and one-leave-out cross-validation.
In our preliminary work described in Chapter 2.4, to detect emotion from EEG,
asymmetric frontal alpha power features are extracted from 8 EEG channel.
These feature vectors are fed into a SVM classifier to do 3-class classification.
Under cross-validation, the accuracy can come up to 90%, however the accuracy
drops to randomly guess (35%) in the prediction. The exceedingly difference be-
Chapter 2 EEG-based Music Emotion Annotation System
27
tween accuracy associated with prediction and cross-validation implies that the
cross-validation might not be used correctly.
EEG signal has consistency in a short time period, thus results in the high
similarity between the feature vectors extracted from neighboring EEG segments.
Meanwhile, the soundness of k-fold cross-validation is partially based on the independency between feature vectors. Consequently, the dependency between frontal
alpha power feature results in a considerable distortion in the k-fold cross-validation
(randomly select feature vector) where training feature vectors and testing feature
vectors are extracted from neighboring EEG segments.
Another issue which might influence the accuracy is the ground truth problem.
This is equivalent to how to guarantee the subject obtain the specific emotion
during experiment. Actually, many stimuli are introduced to help the subject to
experience the emotion. For example, music from terrible movie, sound clips such
as baby laughing, and picture of car accident are used as stimuli. However, no
matter how strong stimuli we used to evoke the emotion, it is still impossible to
verify the subject obtain the emotion indeed.
In addition, current systems do not considerate the difference caused by stimuli.
Many researchers use the similar methods to detect the emotion which is evoked by
different stimuli. Different stimulus, audition or visual will induce signal in different
brain area, and evoke the emotion in different way. More attention should be focus
on how the motion generated in our brain, and what is the difference between the
emotion stages evoked by different stimuli such as happy image and happy music.
Further effort need to employ this knowledge to improve the accuracy of emotion
Chapter 2 EEG-based Music Emotion Annotation System
28
detection from physiological signal.
Furthermore, since physiological signal is quite ambiguous, more attention needs
to be focused on how to extract feature from these ambiguous signal. Unfortunately, unlike facial image or human voice, there is no a gold standard to verify
which pattern of physiological signal is good or bad.
To conclude, how to accurately recognize human emotion from physiology signals and employ this technique to annotate music emotion is still an open problem.
Chapter 3
Automatic Sleep Scoring using
EEG Signal-First Step Towards a
Domain Specific Music
Recommendation System
With the rapid pace of modern life, millions of people suffer from sleep problems.
Music therapy, as a non-medication approach to mitigating sleep problems, has
attracted increasing attention recently. However the adaptability of music therapy is limited by the time-consuming task of choosing suitable music for users.
Inspired by this observation, we discuss the concept of a domain specific music
recommendation system, which automatically recommends music for users according to their sleep quality. The proposed system requires multidisciplinary efforts
29
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
30
including automated sleep quality measurement and content-based music similarity measure. As a first step towards the proposed recommendation system, I focus
on the automated sleep quality measurement in this Chapter. A literature survey
about content-based music similarity measurement is given in Chapter 4.1.
To measure sleep quality, standard Polysomnography (PSG) approach require
various signals such as EEG, ECG and EMG etc. Although satisfactory accuracy
can be obtained based on analysis of multiple signals, those systems might make
subjects feel uncomfortable because they are asked to wear many sensors to collect
the signals during sleep. The situation become evenly worse especially for the
home-based health-care applications in which users’ experience is relatively more
important than systems’ accuracy. To improve users’ experience, it become a
important problem how to measure sleep quality from few signals while reasonable
accuracy still can be achieved.
Motivated by this, an EEG-based approach is proposed to measure user’s sleep
quality. The advantages of the proposed approach over standard PSG method are:
1) it measures sleep quality by recognizing three sleep categories rather than six
sleep stages, thus higher accuracy can be expected; 2) three sleep categories are
recognized by analyzing Electroencephalography (EEG) signal only, so the user
experience is improved because he is attached with fewer sensors during sleep.
Based on the experiments conducted over standard data set, the proposed approach achieves high accuracy and hence shows promising potential for the music
recommendation system.
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
3.1
31
Introduction
People spend about one-third of their whole life on sleep. Although it is still an
open question why human need sleep, there is no doubt that sleep is necessary to
maintain our overall health. Deficiency of good sleep could result in severe physical
and mental effects such as fatigue, tiredness, and depression.
Nowadays millions of people are affected by sleep problems, many of whom
remain undiagnosed. Although various medicines were developed to cure sleep
problems, medicines are not recommended to use because of their negative side
effects. At present music therapy offers an alternative healing method, which
improves sleep quality by playing back music at bed time. With its development
in 2000s, music therapy research has indicated that music does have beneficial
effects on sleep for children [44], young people [45] and older adults [46].
During the process of music therapy, people are asked to listen to a list of music,
which is pre-selected by a music therapist. In spite of its clear benefits towards
sleep quality, current approach is difficult to be widely used because it is a time
consuming task for music therapist to produce a personalized music list. Based on
this observation, a domain specific music recommendation system is introduced to
automatically recommends music for user according to his sleep quality.
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
32
Figure 3.1: Physiology-based Music Rating Component
3.1.1
Music Recommendation according to Sleep Quality
The proposed recommendation system consists of two main components: EEGbased music rating and content-based music recommendation. After music
is played back at bed time, the former component will monitor user’s sleep quality.
Music items will be rated according to user’s sleep quality, as figure 3.1 shows.
It is our contention that similar music pieces share the similar influence towards
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
33
sleep quality. The music items associated with good sleep quality are used as
queries in the music recommendation component. The music which are similar to
the query items will be recommended to the user by conducting music similarity
analysis, as illustrated in figure 4.1. As the first step towards the proposed recommendation system, in this thesis, I focus on automated sleep quality measurement
which is the major task in the physiology-based music rating component.
3.1.2
Normal Sleep Physiology
According to current sleep analysis standard [47], normal sleep physiology consists
of Non Rapid Eye Movement (NREM) and Rapid Eye Movement (REM). NREM
is subdivided into four stages: stages 1 (S1), stage 2 (S2), stage 3 (S3) and stage 4
(S4). S3 and S4 are also called deep sleep. A healthy person will experience about
5 complete sleep cycles per night. [48]. S1 is the first stage in a sleep cycle. S2 is
the second stage. Then S3 and S4 will occur in the cycle consecutively. After the
completion of four stages of NREM, these four stages will reverse rapidly and then
followed by REM, as Figure 3.2 shows.
Given the sleep cycles over night, three main parameters can be calculated to
measure sleep quality: sleep latency, sleep efficiency and percentage of deep sleep.
Specifically, sleep latency is the time that it takes to finish the transition from
wakefulness to the first sleep stage. Sleep efficiency is the ratio of time spent
asleep to the time spent in bed. Percentage of deep sleep is the ratio of deep
sleep to the all sleep stages. To calculate these parameters, we do not need to
recognize every sleep stages in sleep cycles. It is enough if three sleep categories
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
34
Figure 3.2: Typical Sleep Cycles
can be distinguished: wakefulness, deep sleep (S3,S4), and other sleep stages
(S1,S2,REM). Consequently, the problem of sleep quality measurement is converted
into how to recognize these three sleep categories.
3.1.3
Paper Objectives
Now Polysomnography (PSG) technique is widely used in hospital to monitor
patients’ sleep cycles. First developed in 1960s [49], this method, also known
as sleep scoring, has become a golden standard in sleep studies. While users
are sleeping, three physiological signals are monitored: Electroencephalography
(EEG), Electrooculography (EOG), and Electromyography (EMG). Based on the
analysis of these three signals, six sleep stages can be recognized by human expert:
wakefulness, S1, S2, S3, S4 and RAM [50]. These stages are scored based on
30-second epoch.
The standard PSG approach utilizes 9 sensors to monitor the required signals:
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
35
EEG, EOG and EMG [51]. Although satisfactory accuracy can be obtained based
on analysis of multiple signals, standard PSG systems might make subjects feel
uncomfortable because they are asked to wear many sensors to collect the signals
during sleep, as described in Figure 3.3. The situation become evenly worse especially for the home-based health-care applications in which users’ experience is
relatively more important than systems’ accuracy. To improve the user experience, we use fewer signal channels to recognize sleep stages. As discussed above,
to calculate sleep quality parameters, we just need to recognize three categories:
wakefulness, deep sleep and other stages. On the account of the fact that EOG
and EMG are mainly used to distinguish between REM and S1, these two signals
may not contribute much in the calculation of sleep quality. Consequently, we
recognize the three sleep categories from EEG signal only.
3.1.4
Organization of the Thesis
Based on the analysis of current PSG techniques in Section 3.2, we proposed to
recognize the three sleep categories from EEG signal only. The details of our
approach is presented in Section 3.3. Section 3.4 discuses the experiment results.
Conclusions are given in Section 3.5.
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
(a) PSG experiment for Adults [52]
(b) PSG experiment for Children [53]
Figure 3.3: Traditional PSG system with Three Physiological Signals
36
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
3.2
3.2.1
37
Literature review
Manual PSG analysis
In 1968, Rechtschaffen and Kales first proposed a standard method, which called
R&K scoring system [49], to recognize human’s sleep stages by manual. Over 40
years of development, the manual scoring system has been modified and improved
[54, 55, 56], especially for the recording procedure [51], signal amplifier [57] and
artifact issue [58]. Actually R&K method is still the most popular approach in
clinic practice, nowadays.
3.2.2
Computerized PSG Analysis
Intense research has been conducted on computerized PSG analysis, that is how
to automated recognize sleep stages from physiological signals. Two nice survey
papers were published about computerized PSG analysis [59, 60]. Based on the
literature, automated PSG analysis can achieve reasonable accuracy, compared
with the accuracy of human scoring. However its performance is influenced by
many factors, such as subject selection, electrode application and recording quality.
Current automated scoring systems can be mainly categorized into two families:
rule-based expert system and classifier-based system. Intuitively, rule-based expert
system imitates the process of human scoring by considering domain knowledge
such as R&K scoring method. Thus sleep stages is recognized by the expert system
in the similar way as it is done by human expert. Alternatively, classifier-based
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
38
system treats the sleep scoring as a normal classification problem. The typical
works of computerized PSG system are discussed in detail as below.
Based on the survey in literature, most of existing works intend to provide a
service of sleep scoring for clinic application. The main differences between our
approach and these works are that our approach is specifically designed for a
multimedia application rather than clinic one. In particular, we balance the user
experience and system accuracy, a EEG-based approach is proposed to recognize
three sleep categories.
Rule-based Expert System
Peter Anderer et al. [61, 62] developed an expert system, which is called Somnolyzer
24 X 7, to recognize six sleep stages (W,S1,S2,S3,S4 and REM) from 30-second
epoch of three signals (EEG, EMG and EOG). The Somnolyzer 24 X 7 system
consists of two main components: feature generator and rule-based expert system.
First, feature vector are extracted from EEG, EMG and EOG respectively. EEG
signal is filtered into five frequency bands: delta (0.5-2 Hz), theta (2-7 Hz), alpha
(7-12 Hz), beta (12-20 Hz) and fast beta(20-40 Hz). Then density, mean amplitude, mean frequency and variability are calculated from each 30-second epoch in
different frequency bands as well as the full-frequency band (0.5-40 Hz). For the
EMG signal, squared amplitude is calculated from each 1-second epoch, then the
minimum, the maximum and the mean of squared amplitude are determined for
30-second epoch of EMG.
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
39
A decision tree is built based on 10 linear discriminant analysis classifiers. The
80% agreement between expert system scoring and human scoring was achieved.
classifier-based system
The commercial package, BioSleep of Oxford BioSignals Company, is a typical
classifier-based system [63]. Auto regressive coefficients are extracted as feature
vectors from EEG segment, then a neural network classifier is used to classify each
EEG segment into different sleep stages. The BioSleep package obtains reasonable
results with the comparison of human scoring in the third-part evaluation [64].
Other methods
In spite of expert system and classifier approaches, additional solutions have been
proposed also. such as, fuzzy reasoning [65], higher order statistical analysis [66],
hidden markov model [67].
3.3
Methodology
To measure sleep quality, we need to recognize three sleep stage categories: wakefulness, deep sleep, and other sleep stages. To recognize these three sleep categories,
we extract spectral power feature from each 30-seconds EEG epoch. LibSVM
package [41] is used to build a SVM classifier to classify each EEG epoch into one
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
40
of the three categories. Additionally, as discussed in Section 3.1, the transition of
sleep stages follows the trend of sleep cycles. For example, if current epoch belongs
to deep sleep, then the next epoch probably also belongs to deep sleep. However,
SVM classifier treats each 30-second epoch independently, thus it does not take the
advantage of correlations between nearby epochs. Consequently, we further model
the sleep transition as a Markov chain. A matrix will be learned from training data
to indicate the transition probability between different sleep stages, as Figure 3.6
illustrates. Based on this matrix the classification results are further refined. The
probability estimated by SVM classifier for each epoch is processed by a dynamic
programming algorithm.
3.3.1
Feature Extraction
Band power features
EEG is the summation of electrical signal generated by millions of neurons in
the brain. It was first recorded by Richard in 1875 [68], now has been widely
used in PSG studies. EEG signal is usually divided into five different frequency
bands: delta (0.5-2 Hz), theta (2-7 Hz), alpha (8-12 Hz), beta (12-20 Hz) and
fast beta (20-40 Hz). According to previous studies, spectral powers of EEG in
different frequency bands highly relate to sleep stages. In particular, some sleep
stages are recognized by the presence of EEG signal in the specific frequency band.
Spectral power in specific frequency band of EEG is a well-known pattern in PSG
studies [50]. For example, deep sleep is recognized by the present of high amplitude
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
41
(a) Band Power Features extracted from 30-second EEG epoch
(b) Sleep Stages annotated by Human Expert
Figure 3.4: Band Power Features and Sleep Stages
delta wave of EEG. One epoch will be scored as wakefulness when alpha wave
presents in EEG. Consequently, we use spectral power extracted from five frequency
bands as the feature vector. As sleep stage is usually scored based on 30-second
epoch, feature vector is generated for each 30-second EEG epoch.
3.3.2
Classification
Support Vector Machine
Platt et al. [69] presented a work to convert the results of binary SVM into posterior
probabilities. Later this work is extended to estimate multi-class SVM probabil-
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
42
ities by combining the output of multiple one-against-one binary classifiers [70].
With this technique, we use a multi-class SVM classifier to estimate P (x|Ft ), the
probability of epoch t belonging to sleep stage x. Ft is the feature vector extracted from epoch t. We also define C(t, x) = P (x|Ft ). C(t, x) will be used in the
following section.
In SVM classification, epoch t is scored as the stage which could maximize the
probability C(t, x), as following equation shows:
ψ(t) = arg max C(t, x),
x
where ψ(t) is the label generated by classifier for epoch t.
3.3.3
Post Processing
Sleep Stage Transition Modeling
As SVM classifier treats each epoch independently, it does not utilize the correlation between epochs. To take the advantage of sequential information, we model
the transition of sleep stages as a discrete time Markov chain. The effectiveness of
this modeling has been proved by Gibellato [71]. The property of Markov chain
can be formalized in equation 3.1. It is also called first-order Markov assumption,
the stage of current epoch just depends on the stage of previous epoch.
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
43
P r(Xt = xt |Xt−1 = xt−1 , Xt−2 = xt−2 , . . . , X1 = x1 )
= P r(Xt = xt |Xt−1 = xt−1 ) = Pxt−1 xt , (3.1)
where Xt indicates the sleep stage of epoch t; Pxt−1 xt indicates the transition probability from stage xt−1 to stage xt .
As x has three possible stages, there are nine possible values for Pxt−1 xt . These
nine values construct a 3 by 3 matrix which indicates the transition probability
between three stages. This matrix is learned from training data as follows:
Count(i, j)
Pij = ∑3
k=1 Count(i, k)
where Count(i, j) counts the transitions from stage i to stage j. The calculation
of this matrix is also illustrated in Figure 3.6.
Dynamic Programming for Post-processing
Based on the probability generated by SVM and the transition matrix learned from
training data, a dynamic programming (DP) algorithm is designed to score each
epoch in the way that optimum overall posterior probability can be obtained. The
subproblem of dynamic programming is defined in Equation 3.2, which presents
the maximum overall probability from the first epoch to current epoch t, where
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
44
current epoch is scored as stage v.
L(t, v) =
max
x1 ,··· ,xt−1
{
P r(X1 = x1 , · · · , Xt−1 = xt−1 , Xt = v)
t
∏
}
C(i, Xi )
(3.2)
i=1
Because first-order Markov assumption is hold, Equation 3.2 can be simplified as
follows:
L(t, v) = max {L(t − 1, u)P r(Xt = v|Xt−1 = u)} C(t, v)
u
Thus L(t, v) can be calculated on top of L(t − 1, ·), and this forms an optimal
substructure for the subproblem. Therefore, a dynamic programming algorithm
can be designed to find the optimum solution.
To eliminate the float point overflow problem, logarithm probability is used:
S(t, v) = ln L(t, v). For the first epoch, we define S(1, v) = ln C(1, v).
ln C(1, v)
t=1
S(t, v) =
maxu {S(t − 1, u) + ln Puv } + ln C(t, v) t > 1
In backtracing, variable α(t, v) is defined to record the optimum sleep stage of
previous epoch.
α(t, v) = arg max{S(t − 1, u) + ln Puv + ln C(t, v)}
u
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
45
The process of backtracing is shown in the following formulation:
α(t + 1, ϕ(t + 1)) t < n
ϕ(t) =
arg maxv S(n, v) t = n
where ϕ is the refined classification labels which result in the maximum overall
posterior probability.
3.4
Experiment Results
The automated sleep quality measurement consists of two phases. First, SVM
classifier categorizes EEG epoch into one of three classes: wakefulness, deep sleep,
and other sleep stages. Second, a dynamic programming algorithm is used to refine
the results of SVM by utilizing the sequential information. Two experiments are
respectively conducted to evaluate the classifier and the post-process algorithm.
Experiments are conducted based on the Sleep EDF Database, which is selected
from the PhysioBank that is a large achieve of digital recording for biomedical research community [72]. Four recordings which were obtained from healthy subjects
in the hospital are used: st7022j0, st7052j0, st7121j0, and st7132j0. Three signals
(EEG, EOG and EMG) were monitored during night with good signal quality.
Only the EEG channel (Fpz-Cz) is utilized in our experiment. The channels position of Fpz and Cz is illustrated in the Figure 3.5. This data set is annotated by
human expert according to PSG standard [49]. Human annotation is used as the
ground truth in our experiments.
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
Figure 3.5: Position of Fpz and Cz in the 10/20 System
46
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
47
Table 3.1: Accuracy of SVM Classifier in 10-fold Cross-validation
Recording st7022j0 st7052j0 st7121j0 st7132j0
SVM
88.4%
93.9%
90.9%
92.7%
Table 3.2: Confusion Matrix on st7022j0
Deep Others Wake
Deep
238
45
1
Others
26
545
15
Wake
0
22
53
Sleep Stage Classification
In the first experiment, we evaluate the accuracy of SVM classifier. Four sleep
recordings of Sleep EDF Database are divided into 30-second epochs, and 3874
epochs are generated in total. The spectral power feature is extracted from each
epoch. 10-fold cross-validation is conducted based on the band power feature
vectors of each recording. The classifier achieves the average accuracy up to 93.9%
at frame level, as described in Table 3.1. The confusion matrix of SVM are showed
in table 3.2, table 3.3, table 3.4 and table 3.5.
Post Processing
In the second experiment, we evaluate the performance of DP algorithm. As
DP algorithm utilizes sequential information, it processes on continuous epochs.
Table 3.3: Confusion Matrix on st7052j0
Deep Others Wake
Deep
158
18
4
5
724
14
Others
Wake
0
22
105
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
48
Table 3.4: Confusion Matrix on st7121j0
Deep Others Wake
Deep
172
30
1
Others
26
718
9
Wake
1
26
43
Table 3.5: Confusion Matrix on st7132j0
Deep Others Wake
Deep
91
10
2
Others
22
664
3
Wake
0
25
35
Consequently, each recording is divided into two continuous parts, one for training
and one for testing. The first 400 epochs of each recording are used to train the
SVM model and learn the transition matrix. The rest epochs are testing data.
The experiment conducted on one recording is demonstrated in Figure 3.6. Based
on the experiment results showed in Table 3.6, DP algorithm stably improves
the accuracy in each recording. Comparing the sleep cycle generated by classifier
and DP algorithm, in spite of the improvement in accuracy, DP algorithm also
generates a more smooth sleep cycle than the SVM classifier, as Figure 3.6 shows.
In the early stage of this work, the proposed approach is evaluated based on a
data set of four EEG recordings. Due to limitation of the size of data set, training
and testing are conducted on the same subject in the same session. To evaluate
the generalization of the proposed approach, more recording should be collected
and further evaluation should be done in the future work.
Table 3.6: Accuracy of SVM and SVM with Post-processing
Recording st7022j0 st7052j0 st7121j0 st7132j0
SVM
87.5%
92.7%
93.4%
94.2%
SVM+DP
89.3%
95.8%
96.8%
99.3%
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
Figure 3.6: Experiment Over the Recording, st7052j0
49
Chapter 3 Automatic Sleep Scoring using EEG Signal-First Step
Towards a Domain Specific Music Recommendation System
3.5
50
Conclusions
In this thesis, we discussed the concept of a domain specific music recommendation
system, which automatically recommends music for users according to their sleep
quality. One important problem in the proposed recommendation system is how
to automatically monitor users’ sleep quality at night. To address this problem,
we first investigate sleep physiology and traditional PSG approach in literature.
Considering that standard PSG system may make users feel uncomfortable, we
specifically design an approach to recognize three sleep categories from EEG signal
only. Then three parameters, sleep latency, sleep efficiency and percentage of
deep sleep, can be calculated to measure sleep quality. The experiment results
demonstrates that our approach can achieve high accuracy though only the EEG
channel is used. These results motivate us to further develop the content-based
music recommendation components and evaluate the full functions of EEG-based
music recommendation system in the future. As the future work, content-based
music similarity, which is the major task in content-based music recommendation
system, is discussed in Chapter 4.
Chapter 4
Conclusion and Future work
How to utilize EEG signal in music retrieval systems is investigated in this thesis.
Two projects are conducted: EEG-based music emotion annotation and EEGbased music recommendation.
In the first project, an online system is built to recognize audience’s musicevoked emotion from EEG signal. EEG signal is recorded and processed while
subject listens to music clips. Frontal alpha power feature is extracted from 4
pairs of EEG channels: Fp1-Fp2, F7-F8, F3-F4, and FC3-FC4. A SVM classifier
is built to classify the feature vector into three emotion categories: happy, sad, and
peaceful. The proposed approach achieves accuracy around 90% in 10-fold crossvalidation (randomly select feature vector), and 35% in the prediction. To the best
of our knowledge, a few works also present the accuracy up to 90% in the crossvalidation, but nobody achieve reasonable accuracy in the prediction. As discussed
in Chapter 2.5, two conclusions can be derived from this observation. First, special
51
Chapter 4 Conclusion and Future work
52
attention need to be paid on the evaluation of EEG-related system, because the
dependency between feature vectors extracted from neighboring EEG segments
can result in considerable distortion in the k-fold cross-validation. Second, how
to recognize human emotion from EEG signal is still an open question because
nobody can achieve reasonable accuracy in the prediction.
In the second project, a technique is presented to analysis sleep qualify from
EEG signal, as a component of an EEG-based music recommendation system.
The system consists of two components, EEG-based music rating and contentbased music recommendation. Band Power Feature is extracted from one EEG
channel Fpz-Cz to recognize three sleep categories, wakefulness, deep sleep, and
other stages. The proposed approach achieves accuracy up to 90% and shows huge
potential for the EEG-based music rating component. The next step is to implement the content-based music recommendation component and to evaluate the
full function of whole system. In the initial stage of content-based music recommendation component, a literature survey about music similarity measurement is
given in Chapter 4.1. Three approaches are discussed: distance-based approaches,
cluster-based approaches and model-based approaches. Based on the results reported in literatures, model-based approach produces best results and is the most
competent solution for the content-based music recommendation system.
Chapter 4 Conclusion and Future work
4.1
53
Content-based Music Similarity Measurement
As showed in the Figure 4.1, music similarity is the major task in the content-based
music recommendation component. As one of the most fundamental problems in
MIR, how to measure the similarity between music pieces by analyzing content information is still a challenging research topic. Intense works have been conducted
in this field. On the account of the fact that the semantics of music can be ambiguous and complex, to measure their similarity, music is first represented as a set
of feature vector. Thus the music similarity measurement is converted to be how
to measure the similarity of two set of feature vector in a multi-dimensional space.
Therefore, music similarity measurement can be mainly divided to be two phases:
feature extraction from music and similarity measure in a multi-dimensional space.
In the phase of feature extraction, feature vectors are extracted from music
pieces and then placed in a multi-dimensional space. Then music piece is represented by points in high-dimension space. How to extract reliable and effective
feature from music piece is a traditional pattern recognition problem. So far a
large amount of features were presented to describe the semantic of music or audio
piece, such as temporal feature, spectral feature, rhythmic feature, and timbral
feature [73]. Over years of development, many convenient tools are implemented
to effectively extract those features from music such as the Marsyas package [74].
Given the feature vectors of music, many methods were proposed to measure
their similarity. Those methods can be mainly categorized into three families:
distance-based approach, cluster-based approach and model-based approach.
Chapter 4 Conclusion and Future work
Figure 4.1: Content-based Music Recommendation Component
54
Chapter 4 Conclusion and Future work
55
The most generic approach is to calculate the Euclidean distance in the raw
feature space. Therefore the similarity between two music pieces is measured by
the Euclidean distance between feature vectors. This approach implies that feature
value in different dimension is equally important. Thus one unit of distance in X
direction is equal to one unit of distance in Y direction. However, this is not always
the case. Addressing on this problem, Malcolm etc. described and evaluated
five approaches (whitening, LDA, NCA, LMNN and RCA) to rotate and scale
the raw feature space with a linear transform [75]. To evaluate the performance
of the proposed approaches, a straight-forward distance-based classifier, kNN, is
employed to process a classification task in the modified space.
A cluster-based approach, which regards the set of feature vectors as a entity,
was proposed by Logan and Salomn [76]. The proposed method consists of two
steps: frame clustering and cluster model similarity. First, feature vectors are
clustered into different groups by using K-means. Then the distribution of feature
vectors is regarded as a signature of music pieces. Second, the similarity is calculated by comparing the signature using the Earth Movers Distance (EMD) [77].
This approach was improved by Aucouturier and Pachet later [78, 79]. Gaussian
mixture models (GMM) was introduced to cluster the feature vectors. Monte Carlo
(MC) sampling is employed to measure the similarity of cluster model, which is
the likelihood of samples from model A is generated by model B. This approached
was implemented by Elias Pampalk [80] in an open source Package and used in a
genre classification task by Pampalk etc. [81].
In spite of the distance-based approach and cluster-based approach, West and
Chapter 4 Conclusion and Future work
56
Lamere [82] proposed a model-based approach to constructing music similarity.
Authors stated that high-level music factors, such as genre, are also important to
measure the similarity. Instead of directly calculating the distance between feature
vectors, feature vectors are first classified into different categories, and then an
internal profile is build to present the music. Suppose that Px = {cx0 , cx1 , ..., cxn } is
the profile of music pieces x in genre dimension, where cxi indicate the probability
that x is belong to genre i. The similarity Sx,y between music x and y is calculated
based on the Euclidean distance of Px and Py as follows:
Sx,y = 1 −
n
∑
(cxi − xyi )2
(4.1)
i=1
Different classification method can be used such as, LDA or SVM. The genre
label can also be augmented with other music semantic factors such as mood,
tempo or melody.
Zhang etc. [83] extended this approach to represent one music piece as a fuzzy
music semantic vectors by considering the profiles of different music dimensions
(genre, mood, tempo, etc.) together. They also evaluate the extended approach
in a large scale data set. The distance in different dimensions is first calculated as
follows:
1∑ x
(c − xyi )2
n i=1 i
n
Disjx,y
=
(4.2)
Chapter 4 Conclusion and Future work
57
The distance values are scaled by a sigmoid function, and multiplied by different
weight factors, wpi , then they are summed together and generalized by factors a
and b, as following equations:
Sx,y = a
e+1
Where a = 2 e−1
and b =
2
,
e−1
Np
∑
wpi
j=1
1 + eDisx,y
j
and
∑Np
j=1
−b
(4.3)
wpi = 1. Hence the value of Sx,y is in
the range of [0,1].
As above discussion, current approaches of music similarity measurement can
be mainly categorized into three families: distance-based approaches, cluster-based
approaches, and model-based approaches. Based on the evaluation results reported
in the literature [82, 79], model-based approaches likely produce the best performance. Consequently, model-based approach is one competent option for our
content-based music recommendation system. The similar methods reported in
[82, 83] can be employed in the implementation of content-based music recommendation component.
Bibliography
[1] Donna Harman. Relevance feedback revisited. In Proceedings of the 15th
annual international ACM SIGIR conference on Research and development
in information retrieval, pages 1–10, New York, NY, USA, 1992. ACM.
[2] Ioannis K. Ioannis Arapakis. Using facial expressions and peripheral physiological signals as implicit indicators of topical relevance. In ACM Multimedia,
2009.
[3] Mikhail A. Lebedev and Miguel A. L. Nicolelis. Brain-machine interfaces:
past, present and future. Trends in NeurosciencesVolume 29, Issue 9, pages
536–546, September 2006.
[4] Onathan R. Wolpaw, Niels Birbaumer, Dennis J. McFarland, Gert
Pfurtscheller, and Theresa M. Vaughan. Braincomputer interfaces for communication and control. Clinical Neurophysiology, Volume 113, Issue 6:767–791,
2002.
[5] Leeb. Navigation in virtual environments through motor imagery. Proc. of
9th Computer Vision Winter Workshop, , eds. D. Skocaj, publ.: Slovenian
58
BIBLIOGRAPHY
59
Pattern Recognition Society, Piran, Slowenien, pages 99–108, 2004.
[6] Lie L. Liu; and Hong-Jiang Zhang. Automatic mood detection and tracking
of music audio signals. IEEE Transactions on Audio, Speech, and Language
Processing, 14:5–19, 2006.
[7] Rosalind W. Picard.
Affective computing.
The MIT Press, Cambridge,
September 1997.
[8] Yashar Moshfeghi. Affective adaptive retrieval: Study of emotion in adaptive
retrieval. In SIGIR 2008.
[9] Joemon M. Ioannis Arapakis. Affective feedback: An investigation in the role
of emotions in the information seeking process. In ACM SIGIR, 2008.
[10] Zhihong Zeng, Jilin Tu, Ming Liu, Thomas S. Huang, Brian Pianfetti, Dan
Roth, and Stephen Levinson. Audio-visual affect recognition. Multimedia,
IEEE Transactions on, 9(2):424–428, 2007.
[11] Maja P. Zhihong Zeng. A survey of affect recognition methods: Audio, visual,
and spontaneous expressions. IEEE Transactions on Pattern Analysis and
Machine Interlligence.
[12] Elisabeth A. Jonghwa Kim.
Emotion recognition based on physiological
changes in music listening. IEEE Transactions on Pattern Analysis and Machine Interlligence, 30:2067–2083, 2008.
[13] Ekman, Levenson, and Friesen. Autonomic nervous system activity distinguishing among emotions. Science, 221, 1983.
BIBLIOGRAPHY
60
[14] Guillame Chanel, Koray Ciftci, Javier C. Mota, Arman Savran, Luong H.
Viet, Lale Akarun, Alice Caplier, Michele Rombaut, and Bulent Sankur. Emotion detection in the loop from brain signals and facial images. Workshop on
Emotion-Based Agent Architectures, Third International Conference on Autonomous Agents, 2006.
[15] Guillaume Chanel, Julien Kronegg, Didier Grandjean, and Thierry Pun. Emotion assessment: Arousal evaluation using eeg’s and peripheral physiological
signals. Lecture Notes in Computer Science, Multimedia Content Representation, Classification and Security, 2006.
[16] Guillaume Chanel. Emotion assessment for affective-computing based on brain
and peripheral signals. PhD thesis, University of Geneva, Switzerland, 2009.
[17] Danny O. Bos. Eeg-based emotion recognition. 2007.
[18] Yuan-Pin Lin, Chi-Hong Wang, Tien-Lin Wu, Shyh-Kang Jeng, and JyhHorng Chen. Multilayer perceptron for eeg signal classification during listening
to emotional music. TENCON - IEEE Region 10 Conference, 2007.
[19] Yuan-Pin Lin, Chi-Hong Wang, Tien-Lin Wu, Shyh-Kang Jeng, and JyhHorng Chen. Support vector machine for eeg signal classification during listening to emotional music. In IEEE 10th Workshop on Multimedia Signal
Processing, 2008.
[20] Yuan-Pin Lin, Chi-Hong Wang, Tien-Lin Wu, Shyh-Kang Jeng, and JyhHorng Chen. Eeg-based emotion recognition in music listening: A comparison
of schemes for multiclass support vector machine. In IEEE International
Conference on Acoustics, Speech, and Signal Processing (ICASSP’09), 2009.
BIBLIOGRAPHY
61
[21] Elias Vyzas. Recognition of emotional and cognitive states using physiological
data. Master’s thesis, MIT, 1999.
[22] Elias Vyzas and Rosalind W. Picard. Offline and online recognition of emotion expression from physiological data. Workshop on Emotion-Based Agent
Architectures, Third International Conference on Autonomous Agents, 1999.
[23] Jennifer Healey. Wearable and Automotive Systems for the Recognition of
Affect from Physiology. PhD thesis, MIT, 2000.
[24] Rosalind W. Picard, Vyzas, and Healey. Toward machine emotional intelligence: analysis of affective physiological state. IEEE Transactions on Pattern
Analysis and Machine Intelligence, 23, Issue 10:1175–1191, 2001.
[25] Jonghwa Kim. From physiological signals to emotions: Implementing and
comparing selected methods for feature extraction and classification. IEEE
International Conference on Multimedia & Expo (ICME 2005), pages 940–
943, 2005.
[26] Jonghwa Kim and E. Andre. Emotion-specific dichotomous classification and
feature-level fusion of multichannel biosignals for automatic emotion recognition. Multisensor Fusion and Integration for Intelligent Systems, 2008. MFI
2008. IEEE International Conference on, 2008.
[27] Jonghwa Kim and Elisabeth Andre. Emotion recognition based on physiological changes in music listening. IEEE Transactions on Pattern Analysis and
Machine Interlligence, 30:2067–2083, 2008.
BIBLIOGRAPHY
62
[28] Jonghwa Kim and Elisabeth Andre. Biomedical Engineering Systems and
Technologies–Four-Channel Biosignal Analysis and Feature Extraction for Automatic Emotion Recognition. Springer Berlin Heidelberg, 2009.
[29] Jonghwa Kim and Elisabeth Andr´e. Multisensor Fusion and Integration for
Intelligent –Fusion of Multichannel Biosignals Towards Automatic Emotion
Recognition. Springer Berlin Heidelberg, 2009.
[30] C. Maaoui, A. Pruski, and F. Abdat. Emotion recognition for human-machine
communication. IEEE/RSJ International Conference on Intelligent Robots
and Systems, 2008. IROS 2008., pages 1210–1215, 2008.
[31] Robert Horlings, Dragos Datcu, and Leon J. M. Rothkrantz. Emotion recognition using brain activity. CompSysTech’08 ACM International Conference
Proceedings Series, 2008.
[32] Human nervous system, http://en.wikipedia.org/wiki/nervous system.
[33] M. Lyons and C. Bartneck. Hci and the face. Proceedings of the Conference on
Human Factors in Computing Systems (CHI2006), Extended Abstracts, 2006.
[34] John J. B. Allen. Frontal eeg asymmetry, emotion, and psychopathology: the
first, and the next 25 years. Biological Psychology, 67:1–5, 2004.
[35] John J. Nazarian. Issues and assumptions on the road from raw signals to
metrics of frontal eeg asymmetry in emotion. Biological Psychology, 67:183–
218, 2004.
[36] James A. Allen. Frontal eeg asymmetry as a moderator and mediator of
emotion. Biological Psychology, 67:7–49, 2004.
BIBLIOGRAPHY
63
[37] David N. Allen. A better estimate of the internal consistency reliability of
frontal eeg asymmetry scores. Psychophysiology, 46:132–142, 2009.
[38] http://en.wikipedia.org/wiki/10-20 system (eeg).
[39] John T. Cacioppo, Louis G. Tassinary, and Gary Berntson, editors. Handbook
of Psychophysiology. Cambridge University Press, March 2007.
[40] J. Allen. Issues and assumptions on the road from raw signals to metrics
of frontal eeg asymmetry in emotion. Biological Psychology, 67(1-2):183–218,
October 2004.
[41] Chih C. Chang and Chih J. Lin. LIBSVM: a library for support vector machines, 2001.
[42] Yuan-Pin Lin, Chi-Hong Wang, Tien-Lin Wu, Shyh-Kang Jeng, and JyhHorng Chen. Support vector machine for eeg signal classification during listening to emotional music. pages 127–130, October 2008.
[43] http://code.google.com/p/tempo.
[44] Leepeng P. Tan. The effelcts of background music on quality of sleep in
elementary school children. Journal of Music therapy, pages 128–150, 2004.
[45] Harmat, L´aszl´o Tak´acs, Johanna B´odizs, and R´obert. Music improves sleep
quality in students. Journal of Advanced Nursing, 62:327–335, 2008.
[46] Lai, Hui-Ling Good, and Marion. Music improves sleep quality in older adults.
Journal of Advanced Nursing, volum 53, issue 1:134, 2006.
BIBLIOGRAPHY
64
[47] Iber. The AASM Manual for the Scoring of Sleep and Associated Events:
Rules, Terminology and Technical Specification. 2007.
[48] http://en.wikipedia.org/wiki/sleep, retrieved on 29 april 2010.
[49] A. Rechtschaffen and A. Kales. A manual of standardized terminology, techniques and scoring system for sleep stages of human subjects. US Government
Printing Office, US Public Health Service, Washington DC, 1968.
[50] Michael H. Silber.
Staging sleep.
Sleep Medicine Clinics, 4(3):343–352,
September 2009.
[51] Kelly A. Carden. Recording sleep: The electrodes, 10/20 recording system,
and sleep system specifications. Sleep Medicine Clinics, 4(3):333–341, September 2009.
[52] http://www.clevemed.com/crystalmonitor/overview.shtml, retrieved on july
20, 2010.
[53] http://en.wikipedia.org/wiki/polysomnography, retrieved on july 19, 2010.
[54] M. Hirshkowitz. Commentary - standing on the shoulders of giants: the
standardized sleep manual after 30 years. Sleep Medicine Reviews, 4(2):169–
179, April 2000.
[55] S. Himanen. Response to ”standing on the shoulders of giants: The standardized sleep manual after 30 years”. Sleep Medicine Reviews, 4(2):181–182,
April 2000.
[56] S. Himanen. Limitations of rechtschaffen and kales. Sleep Medicine Reviews,
4(2):149–167, April 2000.
BIBLIOGRAPHY
65
[57] Patrick Sorenson. Generating a signal: Biopotentials, amplifiers, and filters.
Sleep Medicine Clinics, 4(3):323–331, September 2009.
[58] Elise Maher and Lawrence J. Epstein. Artifacts and troubleshooting. Sleep
Medicine Clinics, 4(3):421–434, September 2009.
[59] T. Penzel. Computer based sleep recording and analysis. Sleep Medicine
Reviews, 4(2):131–148, April 2000.
[60] T. Penzel, M. Hirshkowitz, J. Harsh, R. D. Chervin, N. Butkov, M. Kryger,
B. Malow, M. V. Vitiello, M. H. Silber, C. A. Kushida, and A. L. Chesson.
Digital analysis and technical specifications. Journal of clinical sleep medicine,
3(2):109–120, March 2007.
[61] Peter Anderer, Georg Gruber, Silvia Parapatics, Michael Woertz, Tatiana Miazhynskaia, Gerhard Klosch, Bernd Saletu, Josef Zeitlhofer, Manuel J. Barbanoj, Heidi Danker-Hopfe, Sari-Leena L. Himanen, Bob Kemp, Thomas Penzel, Michael Grozinger, Dieter Kunz, Peter Rappelsberger, Alois Schlogl, and
Georg Dorffner. An e-health solution for automatic sleep classification according to rechtschaffen and kales: validation study of the somnolyzer 24 x 7
utilizing the siesta database. Neuropsychobiology, 51(3):115–133, 2005.
[62] Peter Anderer, Georg Gruber, Silvia Parapatics, and Georg Dorffner. Automatic sleep classification according to rechtschaffen and kales. In Annual
International Conference of the IEEE Engineering in Medicine and Biology
Society, volume 2007, pages 3994–3997. IEEE, August 2007.
BIBLIOGRAPHY
66
[63] N. McGrogan, E. Braithwaite, and L. Tarassenko. Biosleep: a comprehensive
sleep analysis system. In Engineering in Medicine and Biology Society, Annual
International Conference of the IEEE, November 2002.
[64] Caffarel Jennifer, Gibson G. John, Harrison J. Phil, J. GRIFFITHS Clive,
and J. DRINNAN Michael. Comparison of manual sleep staging with automated neural network-based analysis in clinical practice. Medical & biological
engineering & computing, pages 105–110, 2006.
[65] D. Alvarez-Estevez, J. M. Fernandez-Pastoriza, and V. Moret-Bonillo. A continuous evaluation of the awake sleep state using fuzzy reasoning. In Engineering in Medicine and Biology Society, Annual International Conference of
the IEEE, 2009.
[66] U. R. Abeyratne, S. Vinayak, C. Hukins, and B. Duce. A new measure to
quantify sleepiness using higher order statistical analysis of eeg. In Engineering in Medicine and Biology Society, Annual International Conference of the
IEEE, 2009.
[67] Flexer. An automatic, continuous and probabilistic sleep stager based on a
hidden markov model. In Applied Artificial Intelligence, pages 199–207, 2002.
[68] Maryann Deak and Lawrence J. Epstein. The history of polysomnography.
Sleep Medicine Clinics, 4(3):313–321, September 2009.
[69] John C. Platt. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. In Advances in Large Margin
Classifiers, pages 61–74, 1999.
BIBLIOGRAPHY
67
[70] Ting F. Wu, Chih J. Lin, and Ruby C. Weng. Probability estimates for multiclass classification by pairwise coupling. J. Mach. Learn. Res., 5:975–1005,
2004.
[71] Marilisa G. Gibellato. Stochastic modeling of the sleep process. PhD thesis,
Ohio State University, 2005.
[72] A. L. Goldberger, L. A. N. Amaral, L. Glass, J. M. Hausdorff, Ivanov, R. G.
Mark, J. E. Mietus, G. B. Moody, C. K. Peng, and H. E. Stanley. PhysioBank,
PhysioToolkit, and PhysioNet: Components of a new research resource for
complex physiologic signals. Circulation, 101(23):e215–e220, 2000 (June 13).
[73] Beth Logan. Mel frequency cepstral coefficients for music modeling. In International Symposium/Conference on Music Information Retrieval - ISMIR,
2000.
[74] George Tzanetakis and Perry Cook. Marsyas: a framework for audio analysis.
Org. Sound, 4(3):169–175, December 1999.
[75] M. Slaney, K. Weinberger, and W. White. Learning a metric for music similarity. In ISMIR, pages 313–318, 2008.
[76] Beth Logan and Ariel Salomon. A music similarity function based on signal
analysis. In International Conference on Multimedia and Expo, volume 0,
pages 190+, Los Alamitos, CA, USA, 2001. IEEE Computer Society.
[77] Yossi Rubner, Carlo Tomasi, and Leonidas J. Guibas. The earth mover’s
distance as a metric for image retrieval. International Journal of Computer
Vision, 40, 2000.
BIBLIOGRAPHY
68
[78] Jean-Julien Aucouturier and Francois Pachet. Music similarity measures:
What’s the use? In Ircam, editor, Proceedings of the 3rd International Symposium on Music Information Retrieval, pages 157–163, Paris, France, October
2002.
[79] Jean-Julien Aucouturier and Francois Pachet. Improving timbre similarity:
How high’s the sky?
Journal of Negative Research Results in Speech and
Audio Science, 1, 2004.
[80] E. Pampalk. A matlab toolbox to compute music similarity from audio. In
Proceedings of 5th International Conference on Music Information Retrieval,
Barcelona, Spain, 2004.
[81] Elias Pampalk, Arthur Flexer, and Gerhard Widmer. Improvements of audiobased music similarity and genre classification. In Proc. of Int. Symposium
on Music Information Retrieval, 2005.
[82] Kris West and Paul Lamere. A model-based approach to constructing music
similarity functions. EURASIP J. Appl. Signal Process., 2007(1):149, January
2007.
[83] Bingjun Zhang, Jialie Shen, Qiaoliang Xiang, and Ye Wang. Compositemap:
a novel framework for music similarity measure. In Proceedings of the 32nd
international ACM SIGIR conference on Research and development in information retrieval, pages 403–410, New York, NY, USA, 2009. ACM.
[...]... recognize musical emotion from Audience’s EEG signal instead of music item An online system was built to demonstrate this concept Audience’s EEG signal is captured when s/he listens to the music items Then Alpha frontal power feature is extracted from EEG signal A SVM classifier is used to classify each music items in three musical emotions: happy, sad, and peaceful In the second project, an EEG- assisted music. .. Brain Computer Interface (BCI) [3] Two years ago, I was surprised by the amazing applications of BCI technology such as P300 speller [4], and motor-imaginary-controlled robot [5], etc At that time I came up the idea that utilizing EEG signal in traditional MIR systems I have been trying to find a scenario where EEG signal can be integrated in a MIR system So far two projects have been conducted: EEG- based... pumps blood in the blood vessel, resulting in a peak in BVP signal Heart rate (HR) signal can be extracted from BVP easily BVP is also in uenced by emotions and stress Active feeling such as anger, fear or happiness always increases the value of BVP signal Electroencephalography (EEG) , the electric signal generated by neuron cells can be captured by placing electrodes on the scalp, as described in Figure... emotion from human’s physiological signal instead of low- 4 Chapter 2 EEG- based Music Emotion Annotation System 5 Figure 2.1: Recognize Musical Emotion from Acoustic Features of Music Figure 2.2: Recognize Musical Emotion from Audience’s EEG Signal level feature of music item, as described in Figure 2.2 A physiology-based music emotion annotation approach is investigated in this part The research problem... conducted: EEG- based musical emotion recognition and EEG- assisted music recommendation system The first project is musical emotion recognition from audience’s EEG feedback Music emotion recognition is an important but challenging task in music information retrieval Due to the well-known semantic gap problem, musical emotion cannot be accurately recognized from the low level features extracted from music items... follows EEG- based Music Emotion Annotation System is presented in detail in Chapter 2 Chapter 3 discuss the EEG- assisted music recommendation system Future work and Perspective are summarized in Chapter 4 Chapter 2 EEG- based Music Emotion Annotation System 2.1 Introduction Like genre and culture, emotion is one important factor of music which has attracted much attention in MIR community Musical emotion... This work addresses a healthcare scenario, music therapy, that utilizing Chapter 1 Introduction 3 music to heal people who suffer from sleep disorders Music therapy research has indicated that music does have beneficial effects on sleep During the process of music therapy, people are asked to listen to a list of music, which is pre-selected by music therapist In spite of its clear benefits towards sleep... consuming task for music therapist to produce a personalized music list Based on this observation, an EEG- assisted music recommendation system was proposed, which automatically recommends music for user according to his sleep quality estimated from EEG signal As the first attempt, how to measure sleep quality from EEG signal is investigated This work was recently selected for poster presentation in ACM... detect emotion states in earlier works, are summarized in Table 2.2 2.3.4 Feature Extraction and Classification To decode the emotion from physiological signals, many features have been presented Two popular features are spectral density in frequency domain and statistical information in time domain Chapter 2 EEG- based Music Emotion Annotation System (a) EEG Electrode Cap and EEG Amplifier (b) Experiment... electrode channels Each channel captures EEG signals continuously To perform real-time analysis, the EEG signals are collected by the signal acquisition module The module buffers the continuous signals and feeds them as smaller EEG segments of 1s in duration, into the analysis module The analysis module calculates the spectral power density and the frontal Chapter 2 EEG- based Music Emotion Annotation System ... progresses in the field of music information retrieval (MIR), grand challenges such as the intention gap and the semantic gap still exist Inspired by the current successes in the Brain Computer Interface... features are spectral density in frequency domain and statistical information in time domain Chapter EEG- based Music Emotion Annotation System (a) EEG Electrode Cap and EEG Amplifier (b) Experiment... Chapter Automatic Sleep Scoring using EEG Signal- First Step Towards a Domain Specific Music Recommendation System 42 ities by combining the output of multiple one-against-one binary classifiers [70]
Ngày đăng: 16/10/2015, 12:00
Xem thêm: Utilizing EEG signal in music information retrieval
