Customer emotion recognition through facial expression

Customer Emotion Recognition Through Facial Expression by Hoa T Le Bachelor of Information Technology Thai Nguyen University of Information and Communication Technology – Vietnam, 2012 A Thesis Proposal Submitted to the School of Graduate Studies in Partial Fulfillment of the Requirements for the Degree of Master of Science in Computer Science Mapúa Institute of Technology June 2016 ii ACKNOWLEDGEMENTS The Author would like to express her sincere gratitude to God and to other significant persons for giving the opportunity to complete this study; To the greatest Adviser, Sir Larry A Vea, for the continuous support of this Master thesis study and related research, for his patience, motivation, and immense knowledge His guidance made this research in completion; To the Thesis Committee, Dean Kelly Balan, Sir Joel De Goma, and Sir Aresh Saharkhiz, for their time, insightful comments and encouragement, and for the hard questions which incented the author to widen and improve her research from various perspectives; To the School of Graduate Studies, Dr Jonathan Salvacion, and Sir Omar Ombergado, for their instruction to complete the format of this paper and other requirements needed; To Ms Grace Panahon – Star Circle manager and Ms Rizza Faustino, for the help to have the permission to gather data in the stores; To the Editor, for the time spent in patiently checking the errors and reviewing this manuscript; To the Family, Parents, Brother and Sister-in-law, for the support that they provided through the entire life of the author; To the Friends and Housemates, especially Jocel Marie T Gebora, for the support and provision of food and prayers to have this thesis achieved in full completion Hoa T Le iii TABLE OF CONTENTS TITLE PAGE i APPROVAL PAGE ii ACKNOWLEDGEMENTS iii TABLE OF CONTENTS iv LIST OF TABLE vii LIST OF FIGURES ix ABSTRACT xi Chapter 1: INTRODUCTION Chapter 2: REVIEW OF RELATED LITREATURE Emotion Typologies Customer Emotion Expression of Interest Expression of Happiness 10 Expression of Sadness 11 Expression of Boredom 12 Expression of Surprise 12 Facial Affect Analysis 13 Microsoft Kinect SDK and Face Tracking Outputs 13 Kinect for Xbox 360 Face Tracking Outputs 16 Kinect v2 – High Definition Face Tracking 20 Comparison of Face Tracking Results Between Kinect v1 and Kinect v2 24 Piecewise Bezier Volume Deformation 24 iv Candide-3 25 Classifiers for Emotion Detection 25 Related Works 26 Chapter 3: CUSTOMER EMOTION RECOGNITION THROUGH FACIAL EXPRESSION 29 Abstract 29 Introduction 29 Methodology 33 Research Paradigm 34 Methodology Parameters 35 Data Collection 36 Gathering Setup 36 Feature Extraction and Annotation 38 Feature Selection 41 Annotation 42 Training Classifiers 43 Model Testing 44 Prototype Development 45 Prototype Testing 45 Real World Testing 45 Analysis of the Results 45 Machine Learning and Classification 47 Results and Discussion 48 v Dataset Description 48 Animation Unit Interpretation 49 Annotation Results 52 Correlation Between the AUs 54 Test Machine 55 Definition of Terms 55 Model Development 56 Model Testing 58 Model Performance of thirty-three (33) customers of Kinect 59 Feature Selection 60 Classifier Analysis 62 Prototype Testing Result 66 Real World Testing 67 Conclusion 67 References 68 Chapter 4: CONCLUSIONS 76 Chapter 5: RECOMMENDATIONS 77 REFERENCES 78 vi LIST OF TABLE Table 1: Basic Emotions Table 2: The Angles are expressed in Degrees 18 Table 3: Action Units [AUs] which represent “deltas “from the neutral shape of the face 19 Table 4: Shape Units [SUs] which determine head shape and neutral face 20 Table 5: Face Shape Animations Enumeration 22 Table 6: Kinect v1 and Kinect v2 Face Tracking Outputs 24 Table 7: Emotion Behaviors 42 Table 8: Instances in Kinect v1 and v2 dataset 49 Table 9: Animation Unit Interpretation (Microsoft) for Kinect 49 Table 10: Animation Unit Interpretation (Microsoft) for Kinect 50 Table 11: AUs detected from Sample Face by Kinect 51 Table 12: AUs detected from Sample Face by Kinect 52 Table 13: Features Observed by the Dataset 53 Table 14: Comparison of Magnitudes of “Happy", “Interest”, "Bored”, ”Surprise”" and “Sad” in the Dataset 53 Table 15: AUs Correlation 54 Table 16: Selected features using CfsSubsetEval and BestFrist 61 Table 17: Accuracy result by using CfsSubsetEval and BestFrist Kinect 61 Table 18: Accuracy result by using CfsSubsetEval and BestFrist Kinect 62 Table 19: Base Classifiers of the Random Committee 63 Table 20: Movements considered by the Classifier 63 Table 21: Movements considered by the Classifier 63 vii Table 22: New Patterns Discovered of Customer’s Affect via the Notable Features 65 Table 23: Prototype Testing Results 66 viii LIST OF FIGURES Figure 1: Camera Space 14 Figure 2: Kinect-1-vs-Kinect-2-Tech-Comparison 16 Figure 3: Tracked Points 17 Figure 4: Head Pose Angles 18 Figure 5: Candide -3 face model 25 Figure 6: The Conceptual Framework 34 Figure 7: Research Paradigm 35 Figure 8: Star Circle, Starmall, Alabang 36 Figure 9: Camera Set-Up 37 Figure 10: Setup for Kinect Sensor Captures Full Body 38 Figure 11: Setup for Two (2) Kinect Sensors 38 Figure 12: 3D Face Mask 40 Figure 13: Tracked Face 41 Figure 14: Annotation of Videos 43 Figure 15: Sample Face captured by Kinect 51 Figure 16: Sample Face Captured by Kinect 52 Figure 17: Accuracy of Model Development Results 57 Figure 18: Kappa of Model Development Results 57 Figure 19: Accuracy of Model Testing Results 58 Figure 20: Kappa Statistic of Model Testing Results 59 Figure 21: Accuracy of Model Testing 60 Figure 22: Kappa of Model Testing 60 ix Figure 23: A section of one of the Random Committee base classifiers x 64 70 [14] Zhang S., Zhao X., Lei B (2012) Facial Expression Recognition Based on Local Binary Patterns and Local Fisher Discriminant Analysis.Wseas Transactions On Signal Processing [15] Fernandes S L., Josemin Bala Dr G (2013) A Comparative Study On ICA And LPP Based Face Recognition Under Varying Illuminations And Facial Expressions International Conference on Signal Processing, Image Processing and Pattern Recognition [16] Chen W., Er M J and Wu S (2004) Illumination Compensation and Normalization for Robust Face Recognition Using Discrete Cosine Transform in Logarithm Domain IEEE Transactions On Systems, Man, And Cybernetics— Part b: Cybernetics, Vol 36, No [17] Ma L., Xiao Y, Khorasani K., Ward R K (2004) New Facial Expression Recognition Technique using D DCT and K-Means Algorithm [18] Miyakoshi Y.and Kato S (2011).A system was developed using Bayesian network with partial occluding problem handling [19] Nidhi N Khatri#1, Zankhana H Shah#2,Samip A Patel#3#1 Facial Expression Recognition: A Survey, Computer Engineering Department, Gujarat Technological University Birla Vishvakarma Mahavidyalaya,, India#2,3 Information Technology Department, Gujarat Technological University Birla Vishvakarma Mahavidyalaya,, India [20] Shouse, E., M/C Journal 8.6 (2005) Feeling, Emotion, Affect 25 Feb 2015 71 [21] Khatri N N et al.,(IJCSIT).International Journal of Computer Science and Information Technologies, Vol [1], 2014, 149-152 [22] http://smallbusiness.chron.com/say-customers-trying-sell-product-31955.html [23] Ekman P., Friesen, W V., & Hager J C., (2002), “Facial Action Coding System, The Manual”, Published by Research Nexus division of Network Information Research Corporation, Copyright © 2002 Paul Ekman, Wallace V Friesen, & Joseph C Hager All Rights Reserved [24] Kwang E K & Sim K B.[2010].The AAM [Active appearance model] and DBN [Dynamic Bayesian network] was combined for emotion recognition [25] Munezero M., Montero C S., Sutinen E., and Pajunen J (IEEE 2014) Are They Different? Affect, Feeling, Emotion, Sentiment, and Opinion Detection in Text [26] Sumathi C P., Santhanam T and Mahadevi M (2012) Automatic Facial Expression Analysis A survey, International Journal of Computer Science & Engineering Survey [IJCSES] Vol.3, No.6 [27] Face Tracking http://msdn.microsoft.com/en-us/library/jj130970.aspx Accessed September 2014 [28] Moe, W.W (2003) “Buying, Searching, or Browsing: Differentiating between Online Shoppers using In-Store Navigational Clickstream”, Journal of Consumer Psychology, Vol 13, No 1&2: 29-39 [29] Wyrembelski A (2013) Detection of the selected, basic emotions based on face expression using Kinect [30] Kaliouby, R E (2004) Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures University of Cambridge, UK, Robinson, P 72 [31] Vineetha G R Sreeji C and Joseph L (2012) Face Expression Detection Using Microsoft Kinect with the Help of Artificial Neural Network [32] Stocchi L (2013) 3D Facial Expression Recognition Using the Microsoft Kinect School of Computing Dublin City University, Dublin, Ireland [33] Kasiran Z., Yahya S [Dr] and Ibrahim Z (2008) Facial Expression Recognition as an Implicit Customers’ Feedback [34] Carlos C M., Delos Santos J E., Fournier G., Vea L (2012), PCSC 2013 Towards the Development of an Intelligent Agent for Novice Programmers through Face Expression Recognition [35] Shergill, G.S., Diegel, O., Sarrafzadeh, A., and Shekar, A., New Zealand, (2008) Computerized Sales Assistants: The Application of Computer Technology to Measure Consumer Interest – A Conceptual Framework [36] Candide – A parameterized face [online], Available from: http://www.icg.isy.liu.se/candide/ [last accessed November 2014] [37] http://changingminds.org/explanations/explanations.htm [last accessed November 2014] [38] Frijda, N H (1986) The emotions New York: Cambridge University Press [39] Izard, C E (1977) Human emotions New York: Plenum Press [40] Tomkins, S S (1984) Affect theory In K R Scherer & P Ekman [Eds.], Approaches to emotion (pp 163-195) [41] http://www.emotionwisegroup.org/emotipedia/interest [42] Consoli, D (2009) Emotions That Influence Purchase Decisions and Their Electronic Processing 73 [43] Tao H and Huang T.S (2002) A Piecewise Bezier Volume Deformation Model and Its Applications in Facial Motion Capture in Advances in Image Processing and Understanding [44] Rydfalk, M (1987) “CANDIDE, a parameterized face” Report No LiTH-ISY-I-866, Dept of Electrical Engineering, Linköping University, Sweden [45] http://www.icg.isy.liu.se/candide/main.html [46] Dornaika F and Ahlberg J.(2004) Fast and reliable active appearance model search for 3-D face tracking IEEE Trans Systems, Man, and Cybernetics, Part B, vol 34, no 4, pp 1838-1853 [47] Desmet P M A (2012) Faces of Product Pleasure: 25 Positive Emotions in HumanProduct Interactions [48] Cohen I., Sebe N., Chen L., Garg A., Huang T S Facial Expression Recognition from Video Sequences: Temporal and Static Modeling [49] Cohn J F., Zlochower A Z., Lien J J., and Kanade T (1998) Feature-Point Tracking by Optical Flow Discriminates Subtle Differences in Facial Expression Proc Int'l Conf Automatic Face and Gesture Recognition, p 396-401 [50] Pantic M and Rothkrantz L (2000) Expert System for Automatic Analysis of Facial Expression Image and Vision Computing J., Vol 18, No 11, p 881-905 [51] Gallup, (November 1, 2012) http://www.gallup.com/businessjournal/158450/businessimpact-human-emotions.aspx [52] Jellesma F.C., & Vingerhoets A.J.J.M (2012) Sex Roles (Vol 67, Iss 7, pp 412-421) Heidelberg, Germany: Springer 74 [53] https://msdn.microsoft.com/en-us/library/dn785525.aspx [54] Picard R., 1997 Affective Computing Cambridge, MA: MIT Press [55] Nedkov S and Dimov D., 2013 Emotion recognition by face dynamics In Proceedings of the 14th International Conference on Computer Systems and Technologies (CompSysTech '13), Rachev B and Smrikarov A (Eds.) ACM, New York, NY, USA, 128-136 [56] Michel, P & el Kaliouby, R., 2003 Real time facial expression recognition in video using support vector machines Proceedings of the 5th international conference on Multimodal interfaces [57] Averill, J R (1975) A semantic atlas of emotion concepts JSAS catalogue of selected documents in psychology, 5(330), 1-64 [58] Okazaki, S., 2005 New perspectives on m-commerce research, Journal of Electronic Commerce Research, Vol 6, No 3: 160-16 [59] Srikumar, K and Bhasker, B., 2004 Personalized product selection in Internet business, Journal of Electronic Commerce Research, Vol 5, No 4: 216-27 [60] Kim, C & Galliers, R D., 2004 Deriving a diffusion framework and research agenda for web-based shopping systems, Journal of Electronic Commerce Research, Vol 5, No 3: 199-215, 2004 [61] Salovey, P, and Mayer, J., 1990 Emotional Intelligence, Imagination, Cognitive and Personality, Vol 9, No 3: 185-211 [62] Howard, D.J and Gengler, C., 2001 Emotional Contagion effects on Product Attitudes, Journal of Consumer Research, Vol 28 (Sept): 189-201 75 [63] Sirakaya, E and Sonmez, S., 2000 Gender Images in State Tourism Brouchures: An Overlooked Area in Socially Responsible Tourism Marketing Journal of Travel Research, Vol 38 (May): 353-362 [64] Yuasa, M., Yasumura, Y and Nitta, K., 2001 A Negotiation Support Tool Using Emotional Factors IEEE, Vol 5: 2906-2911 [65] Lee, R.S.T., 2002 iJADE Authenticator- An Intelligent Multiagent based Facial Authentication System, International Journal of Pattern Recognition and Artificial Intelligence, Vol.16, No 4: 481-500 [66] Ghijsen, M., Heylen, D., Nijholt, A., and op den Akker, R., 2005 Facial affect displays during tutoring sessions Affective Interactions: The Computer in the Affective Loop [67] Sarrafzadeh, A., Messom, C., Overmyer, S., Mills, H., Fan, H., Bigdeli, A and BiglariAbhari, M., 2002 The Future of Computer-assisted Management Education and Development International Journal of Management Literature, Vol 2, No 4: 214-29 Chapter CONCLUSIONS This study has presented the steps of developing a model for customer emotion recognition using the Kinect sensors The Researcher constructed two (2) training sets and two test sets were gathered using Kinect sensor version and version which are containing 3D data for sixteen (16) different customers performing the five (5) emotions She compared results from the classifiers in WEKA and between Kinect v1 and Kinect v2 From the experiment, the data analysis, and performance testing of the models has been showed that facial expressions can be used to determine the affect state of a customer but sometimes the researcher needs to consider other movements such as body languages The results shown that the most effective means to classify thee data would be using an ensemble of classifiers The experiment results show that the algorithm has the highest accuracy is Random Committee, with 92.9301% for Kinect v1 and 94.4448% for Kinect v2 After train the classifiers, tested the models and got higher results: 93.0890% (Kinect v1) and 98.8839% (Kinect v2) It can be concluded that the datasets are consistency Further inspection of the classifier showed that it worked very similarly to the Random Forest, hence explaining the small difference between the two For the comparison between Kinect v1 and Kinect v2, the results show that Kinect v2 has better kappa statistic and prediction rates higher than Kinect v1 Lastly, based on the results the Researcher noticed that there is difficulty in classifying emotions in the prototype testing and the real-world setting The kappa statistics results with the average 0.67371 of the Kinect and the Kinect got kappa average of 0.5428, it means it’s in the “moderate” to good agreement range 76 Chapter RECOMMENDATIONS The researcher recommends that that future studies the following:  Number of participants – The dataset of this study was a collection of facial features of sixteen (16) customers for both Kinect v1 and Kinect was not enough to cover some cases where customers express their emotions differently In order to cover a broader scope of face expressions, increase the size of the dataset by considering additional customers is highly recommended  Hardware - The researcher used the Kinect sensor v2 with Microsoft SDK 2.0 – HD Face Tracking C++, which is unmanaged memory leak It is recommended to add hardware to avoid memory leak in data gathering stage  Additional annotators – Only three (3) annotators annotated the video logs with the incomplete dataset, which may severely reduce the quality of the data due to human error  Prototype Testing – The prototype testing was a simple program which is the save model was embedded It is recommended for IT programmer to develop a complete software to detect customer emotions 77 REFERENCES [1] Http://smallbusiness.chron.com/say-customers-trying-sell-product-31955.html [2] http://smeenk.com/kinect-field-of-view-comparison/ [3] Keltner, D., and Ekman, P (2000) ‘Facial expression of emotion’, in Lewis, M., and Haviland, J.M [Eds.]: ‘Handbook of emotions’ [Guilford, 2000, 2nd edition], pp 236249 [4] Kanade, T., Cohn, J.F., and Tian, Y.: ‘Comprehensive database for facial expression analysis’, Fourth IEEE International Conference on Automatic Face and Gesture Recognition, 2000, FG '00, pp 46-53 [5] Witten I H and Frank E., 2005 Data Mining: Practical machine learning tools and techniques Morgan Kaufmann [6] Ekman P (1982) Emotion in the Human Face Cambridge University Press, New York, editions [7] Tian Y., Kanade T., and Cohn J., (2001) Recognizing action units for facial expression analysis IEEE Transactions on Pattern Analysis and Machine Intelligence, 23[2] [8] Bartlett M.S., Littlewort G., Frank M., Lainscsek C., Fasel I., and Movellan J (2006) Fully automatic facial action recognition in spontaneous behavior In Proceedings of the IEEE Conference on Automatic Facial and Gesture Recognition [9] Pantic M and Rothkrantz M (IEEE 2004) Facial action recognition for facial expression analysis from static face images IEEE Transactions on Systems, Man and Cybernetics [10] Littlewort G., Bartlett M., Fasel I., Susskind J., and Movellan J (2006) Dynamics of facial expression extracted automatically from video Image and Vision Computing [11] Deleuze and Guattari (1987) A Thousand Plateaus: Capitalism and Schizophrenia 78 79 [12] Ekman P.andFriesen W (1978) The Facial Action Coding System: A Technique for the Measurement of Facial Movement Consulting Psychologists Press, Inc., San Francisco, CA [13] Kittusamy R V and Chakrapani V (2012) Facial Expressions Recognition Using Eigenspaces Journal of Computer Science [14] Zhang S., Zhao X., Lei B (2012) Facial Expression Recognition Based on Local Binary Patterns and Local Fisher Discriminant Analysis.Wseas Transactions On Signal Processing [15] Fernandes S L., Josemin Bala Dr G (2013) A Comparative Study On ICA And LPP Based Face Recognition Under Varying Illuminations And Facial Expressions International Conference on Signal Processing, Image Processing and Pattern Recognition [16] Chen W., Er M J and Wu S (2004) Illumination Compensation and Normalization for Robust Face Recognition Using Discrete Cosine Transform in Logarithm Domain IEEE Transactions On Systems, Man, And Cybernetics— Part b: Cybernetics, Vol 36, No [17] Ma L., Xiao Y, Khorasani K., Ward R K (2004) New Facial Expression Recognition Technique using D DCT and K-Means Algorithm [18] Miyakoshi Y.and Kato S (2011).A system was developed using Bayesian network with partial occluding problem handling [19] Nidhi N Khatri#1, Zankhana H Shah#2,Samip A Patel#3#1 Facial Expression Recognition: A Survey, Computer Engineering Department, Gujarat Technological University Birla Vishvakarma Mahavidyalaya,, India#2,3Information Technology 80 Department, Gujarat Technological University Birla Vishvakarma Mahavidyalaya,, India [20] Shouse, E., M/C Journal 8.6 (2005) Feeling, Emotion, Affect 25 Feb 2015 [21] Khatri N N et al.,(IJCSIT).International Journal of Computer Science and Information Technologies, Vol [1], 2014, 149-152 [22] http://smallbusiness.chron.com/say-customers-trying-sell-product-31955.html [23] Ekman P., Friesen, W V., & Hager J C., (2002), “Facial Action Coding System, The Manual”, Published by Research Nexus division of Network Information Research Corporation, Copyright © 2002 Paul Ekman, Wallace V Friesen, & Joseph C Hager All Rights Reserved [24] Kwang E K & Sim K B.[2010].The AAM [Active appearance model] and DBN [Dynamic Bayesian network] was combined for emotion recognition [25] Munezero M., Montero C S., Sutinen E., and Pajunen J (IEEE 2014) Are They Different? Affect, Feeling, Emotion, Sentiment, and Opinion Detection in Text [26] Sumathi C P., Santhanam T and Mahadevi M (2012) Automatic Facial Expression Analysis A survey, International Journal of Computer Science & Engineering Survey [IJCSES] Vol.3, No.6 [27] Face Tracking http://msdn.microsoft.com/en-us/library/jj130970.aspx Accessed September 2014 [28] Moe, W.W (2003) “Buying, Searching, or Browsing: Differentiating between Online Shoppers using In-Store Navigational Clickstream”, Journal of Consumer Psychology, Vol 13, No 1&2: 29-39 81 [29] Wyrembelski A (2013) Detection of the selected, basic emotions based on face expression using Kinect [30] Kaliouby, R E (2004) Real-Time Inference of Complex Mental States from Facial Expressions and Head Gestures University of Cambridge, UK, Robinson, P [31] Vineetha G R Sreeji C and Joseph L (2012) Face Expression Detection Using Microsoft Kinect with the Help of Artificial Neural Network [32] Stocchi L (2013) 3D Facial Expression Recognition Using the Microsoft Kinect School of Computing Dublin City University, Dublin, Ireland [33] Kasiran Z., Yahya S [Dr] and Ibrahim Z (2008) Facial Expression Recognition as an Implicit Customers’ Feedback [34] Carlos C M., Delos Santos J E., Fournier G., Vea L (2012), PCSC 2013 Towards the Development of an Intelligent Agent for Novice Programmers through Face Expression Recognition [35] Shergill, G.S., Diegel, O., Sarrafzadeh, A., and Shekar, A., New Zealand, (2008) Computerized Sales Assistants: The Application of Computer Technology to Measure Consumer Interest – A Conceptual Framework [36] Candide – A parameterized face [online], Available from: http://www.icg.isy.liu.se/candide/ [last accessed November 2014] [37] http://changingminds.org/explanations/explanations.htm [last accessed November 2014] [38] Frijda, N H (1986) The emotions New York: Cambridge University Press [39] Izard, C E (1977) Human emotions New York: Plenum Press [40] Tomkins, S S (1984) Affect theory In K R Scherer & P Ekman [Eds.], Approaches to emotion (pp 163-195) 82 [41] http://www.emotionwisegroup.org/emotipedia/interest [42] Consoli, D (2009) Emotions That Influence Purchase Decisions and Their Electronic Processing [43] Tao H and Huang T.S (2002) A Piecewise Bezier Volume Deformation Model and Its Applications in Facial Motion Capture in Advances in Image Processing and Understanding [44] Rydfalk, M (1987) “CANDIDE, a parameterized face” Report No LiTH-ISY-I-866, Dept of Electrical Engineering, Linköping University, Sweden [45] http://www.icg.isy.liu.se/candide/main.html [46] Dornaika F and Ahlberg J.(2004) Fast and reliable active appearance model search for 3-D face tracking IEEE Trans Systems, Man, and Cybernetics, Part B, vol 34, no 4, pp 1838-1853 [47] Desmet P M A (2012) Faces of Product Pleasure: 25 Positive Emotions in HumanProduct Interactions [48] Cohen I., Sebe N., Chen L., Garg A., Huang T S Facial Expression Recognition from Video Sequences: Temporal and Static Modeling [49] Cohn J F., Zlochower A Z., Lien J J., and Kanade T (1998) Feature-Point Tracking by Optical Flow Discriminates Subtle Differences in Facial Expression Proc Int'l Conf Automatic Face and Gesture Recognition, p 396-401 [50] Pantic M and Rothkrantz L (2000) Expert System for Automatic Analysis of Facial Expression Image and Vision Computing J., Vol 18, No 11, p 881-905 83 [51] Gallup, (November 1, 2012) http://www.gallup.com/businessjournal/158450/businessimpact-human-emotions.aspx [52] Jellesma F.C., & Vingerhoets A.J.J.M (2012) Sex Roles (Vol 67, Iss 7, pp 412-421) Heidelberg, Germany: Springer [53] https://msdn.microsoft.com/en-us/library/dn785525.aspx [54] Picard R., 1997 Affective Computing Cambridge, MA: MIT Press [55] Nedkov S and Dimov D., 2013 Emotion recognition by face dynamics In Proceedings of the 14th International Conference on Computer Systems and Technologies (CompSysTech '13), Rachev B and Smrikarov A (Eds.) ACM, New York, NY, USA, 128-136 [56] Michel, P & el Kaliouby, R., 2003 Real time facial expression recognition in video using support vector machines Proceedings of the 5th international conference on Multimodal interfaces [57] Averill, J R (1975) A semantic atlas of emotion concepts JSAS catalogue of selected documents in psychology, 5(330), 1-64 [58] Okazaki, S., 2005 New perspectives on m-commerce research, Journal of Electronic Commerce Research, Vol 6, No 3: 160-16 [59] Srikumar, K and Bhasker, B., 2004 Personalized product selection in Internet business, Journal of Electronic Commerce Research, Vol 5, No 4: 216-27 [60] Kim, C & Galliers, R D., 2004 Deriving a diffusion framework and research agenda for web-based shopping systems, Journal of Electronic Commerce Research, Vol 5, No 3: 199-215, 2004 84 [61] Salovey, P, and Mayer, J., 1990 Emotional Intelligence, Imagination, Cognitive and Personality, Vol 9, No 3: 185-211 [62] Howard, D.J and Gengler, C., 2001 Emotional Contagion effects on Product Attitudes, Journal of Consumer Research, Vol 28 (Sept): 189-201 [63] Sirakaya, E and Sonmez, S., 2000 Gender Images in State Tourism Brouchures: An Overlooked Area in Socially Responsible Tourism Marketing Journal of Travel Research, Vol 38 (May): 353-362 [64] Yuasa, M., Yasumura, Y and Nitta, K., 2001 A Negotiation Support Tool Using Emotional Factors IEEE, Vol 5: 2906-2911 [65] Lee, R.S.T., 2002 iJADE Authenticator- An Intelligent Multiagent based Facial Authentication System, International Journal of Pattern Recognition and Artificial Intelligence, Vol.16, No 4: 481-500 [66] Ghijsen, M., Heylen, D., Nijholt, A., and op den Akker, R., 2005 Facial affect displays during tutoring sessions Affective Interactions: The Computer in the Affective Loop [67] Sarrafzadeh, A., Messom, C., Overmyer, S., Mills, H., Fan, H., Bigdeli, A and BiglariAbhari, M., 2002 The Future of Computer-assisted Management Education and Development International Journal of Management Literature, Vol 2, No 4: 214-29 ... RELATED LITREATURE Emotion Typologies Customer Emotion Expression of Interest Expression of Happiness 10 Expression of Sadness 11 Expression of Boredom 12 Expression of Surprise 12 Facial Affect Analysis... already some studies on customer emotion through facial expression using ordinary cameras In this study, the Researcher aimed to develop models that recognize customer s emotion through Kinect sensor... recognizes customer s emotion through facial expressions using Kinect sensor v1 and v2 The Researcher aims to extract and determine notable facial features that can be used to recognize customer s emotions