Nhận dạng cử chỉ động của bàn tay người sử dụng kết hợp thông tin hình ảnh và độ sâu ứng dụng trong tương tác người thiết bị

MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECNOLOGY THI HUONG GIANG DOAN DYNAMIC HAND GESTURE RECOGNITION USING RGB-D IMAGES FOR HUMAN-MACHINE INTERACTION DOCTORAL THESIS OF CONTROL ENGINEERING AND AUTOMATION Hanoi − 2017 MINISTRY OF EDUCATION AND TRAINING HANOI UNIVERSITY OF SCIENCE AND TECNOLOGY THI HUONG GIANG DOAN DYNAMIC HAND GESTURE RECOGNITION USING RGB-D IMAGES FOR HUMAN-MACHINE INTERACTION Specialty: Control Engineering and Automation Specialty Code: 62520216 DOCTORAL THESIS OF CONTROL ENGINEERING AND AUTOMATION SUPERVISORS: Dr Hai Vu Dr Thanh Hai Tran Hanoi − 2017 DECLARATION OF AUTHORSHIP I, Thi Huong Giang Doan, declare that the thesis titled, “Dynamic Hand Gesture Recognition Using RGB-D Images for Human-Machine Interaction”, and the works presented in it are my own I confirm that: This work was done wholly or mainly while in candidature for a Ph.D research degree at Hanoi University of Science and Technology Where any part of this thesis has previously been submitted for a degree or any other qualification at Hanoi University of Science and Technology or any other institution, this has been clearly stated Where I have consulted the published work of others, this is always clearly attributed Where I have quoted from the work of others, the source is always given With the exception of such quotations, this thesis is entirely my own work I have acknowledged all main sources of help Where the thesis is based on work done by myself jointly with others, I have made exactly what was done by others and what I have contributed myself Hanoi, December 2017 PhD STUDENT Thi Huong Giang DOAN SUPERVISORS Dr Hai VU Dr Thi Thanh Hai TRAN i ACKNOWLEDGEMENT This thesis was written during my doctoral study at International Research Institute Multimedia, Information, Communication and Applications (MICA), Hanoi University of Science and Technology (HUST) It is my great pleasure to thank all the people who supported me for completing this work First, I would like to express my sincere gratitude to my advisors Dr Hai Vu and Dr Thi Thanh Hai Tran for the continuous support of my Ph.D study and related research, for their patience, motivation, and immense knowledge Their guidance helped me in all the time of research and writing of this thesis I could not have imagined having a better advisor and mentor for my Ph.D study Besides my advisors, I would like to thank the scientists and the authors of the published works which are cited in this thesis, and I am provided with valuable information resources from their works for my thesis The attention at scientific conferences have always been a great experience for me to receive many the useful comments In the process of implementation and completion of my research, I have received many supports from the board of MICA directors My sincere thanks go to Prof Yen Ngoc Pham, Prof Eric Castelli and Dr Son Viet Nguyen, who provided me with an opportunity to join researching works in MICA institute, and who gave access to the laboratory and research facilities Without their precious support would it have been being impossible to conduct this research As a Ph.D student of 911 programme, I would like to thanks 911 programme for their financial support during my Ph.D course I also gratefully acknowledge the financial support for publishing papers and conference fees from research projects T2014-100, T2016-PC-189, and T2016-LN-27 I would like to thank my colleagues at Computer Vision Department and Multi-Lab of MICA institute over the years both at work and outside of work Special thanks to my family Words can not express how grateful I am to my mother and father for all of the sacrifices that they have made on my behalf I would also like to thank my beloved husband Thank you for supporting me for everything Hanoi, December 2017 Ph.D Student Thi Huong Giang DOAN ii CONTENTS DECLARATION OF AUTHORSHIP i ACKNOWLEDGEMENT ii CONTENTS vi SYMBOLS vii LIST OF TABLES xi LIST OF FIGURES xvi LITERATURE REVIEW 1.1 Completed hand gesture recognition systems for controlling home appliances 1.1.1 GUI device dependent systems 1.1.2 GUI device independent systems 1.2 Hand detection and segmentation 1.2.1 Color 1.2.2 Shape 1.2.3 Motion 1.2.4 Depth 1.2.5 Discussions 1.3 Hand gesture spotting system 1.3.1 Model-based approaches 1.3.2 Feature-based approaches 1.3.3 Discussions 1.4 Dynamic hand gesture recognition 1.4.1 HMM-based approach 1.4.2 DTW-based approach 1.4.3 SVM-based approach 1.4.4 Deep learning-based approach 1.4.5 Conclusion 1.5 Discussion and Conclusion 8 14 18 19 20 21 21 23 24 25 27 29 29 30 31 33 34 35 35 A NEW DYNAMIC HAND GESTURE SET OF CYCLIC MOVEMENT 37 iii 2.1 2.2 2.3 2.4 2.5 Defining dynamic hand gestures The existing dynamic hand gesture datasets 2.2.1 The published dynamic hand gesture datasets 2.2.1.1 The RGB hand gesture datasets 2.2.1.2 The Depth hand gesture datasets 2.2.1.3 The RGB and Depth hand gesture datasets 2.2.2 The non-published hand gesture datasets 2.2.3 Conclusion Definition of the closed-form pattern of gestures and phasing issues 2.3.1 A conducting commands of a dynamic hand gestures set 2.3.2 Definition of the closed-form pattern of gestures and phasing issues 2.3.3 Characteristics of dynamic hand gesture set Data collection 2.4.1 MICA1 dataset 2.4.2 MICA2 dataset 2.4.3 MICA3 dataset 2.4.4 MICA4 dataset Discussion and Conclusion HAND DETECTION AND GESTURE SPOTTING WITH USERGUIDE SCHEME 3.1 Introduction 3.2 Heuristic user-guide scheme 3.2.1 Assumptions 3.2.2 Proposed framework 3.2.3 Estimating heuristic parameters 3.2.3.1 Estimating parameters of background model for body detection 3.2.3.2 Estimating the distance from hand to the Kinect sensor for extracting hand candidates 3.2.3.3 Estimating skin color parameters for pruning hand regions 3.2.4 Hand detection phase using heuristic parameters 3.2.4.1 Hand detection 3.2.4.2 Hand posture recognition 3.3 Dynamic hand gesture spotting 3.3.1 Catching buffer 3.3.2 Spotting dynamic hand gesture 3.4 Experimental results 3.4.1 The required learning time for end-users iv 37 38 38 38 40 41 44 46 47 47 48 50 51 51 52 53 54 55 56 56 58 58 58 60 60 62 63 65 65 66 66 66 67 71 71 3.4.2 3.4.3 3.5 The computational time for hand segmentation and recognition Performance of the hand region segmentations 3.4.3.1 Evaluate the hand segmentation 3.4.3.2 Compare the hand posture recognition results 3.4.4 Performance of the gesture spotting algorithm Discussion and Conclusion 3.5.1 Discussions 3.5.2 Conclusions 73 75 75 75 76 78 78 78 DYNAMIC HAND GESTURE REPRESENTATION AND RECOGNITION USING SPATIAL-TEMPORAL FEATURES 79 4.1 Introduction 79 4.2 Proposed framework 80 4.2.1 Hand representation from spatial and temporal features 81 4.2.1.1 Temporal features extraction 81 4.2.1.2 Spatial features extraction using linear reduction space 83 4.2.1.3 Spatial features extraction using non-linear reduction space 84 4.2.2 DTW-based phase synchronization and KNN-based classification 86 4.2.2.1 Dynamic Time Warping for phase synchronization 86 4.2.2.2 Dynamic hand gesture recognition using K-NN method 88 4.2.3 Interpolation-based synchronization and SVM Classification 89 4.2.3.1 Dynamic hand gesture representation 89 4.2.3.2 Quasi-periodic dynamic hand gesture pattern 91 4.2.3.3 Phase synchronization using hand posture interpolation 94 4.2.3.4 Dynamic hand gesture recognition using difference classifications 96 4.3 Experimental results 97 4.3.1 Influence of temporal resolution on recognition accuracy 97 4.3.2 Tunning kernel scale parameters RBF-SVM classifier 98 4.3.3 Performance evaluation of the proposed method 99 4.3.4 Impacts of the phase normalization 100 4.3.5 Further evaluations on public datasets 101 4.4 Discussion and Conclusion 103 4.4.1 Discussion 103 4.4.2 Conclusion 103 CONTROLLING HOME APPLIANCES USING DYNAMIC HAND GESTURES 105 5.1 Introduction 105 v 5.2 5.3 5.4 Deployment of control systems using hand gestures 5.2.1 Assignment of hand gestures to commands 5.2.2 Different modes of operations carried out by hand gestures 5.2.2.1 Different states of lamp and their transitions 5.2.2.2 Different states of fan and their transition 5.2.3 Implementation of the control system 5.2.3.1 Main components of the control system using hand gestures 5.2.3.2 Integration of hand gesture recognition modules Experiments of control systems using hand gestures 5.3.1 Environment and material setup 5.3.2 Pre-built script 5.3.3 Experimental results 5.3.3.1 Evaluation of hand gesture recognition 5.3.3.2 Evaluation of time costs 5.3.4 Evaluation of usability Discussion and Conclusion 5.4.1 Discussions 5.4.2 Conclusion Bibliography 105 105 107 107 108 108 108 109 115 115 116 117 118 119 120 121 121 122 126 vi ABBREVIATIONS TT Abbreviation Meaning ANN Artifical Neural Network ASL American Sign Language BB Bounding Box BGS Background Subtraction BW Baum Welch BOW Bag Of Words C3D Convolutional 3D CD Compact Disc CIF Common Intermediate Format 10 CNN Convolution Neural Network 11 CPU Central Processing Unit 12 CRFs Conditional Random Fields 13 CSI Channel State Information 14 DBN Deep Belief Network 15 DDNN Deep Dynamic Neural Networks 16 DoF Degree of Freedom 17 DT Decision Tree 18 DTM Dense Trajectories Motion 19 DTW Dynamic Time Warping 20 FAR False Acceptance Rate 21 FD Fourier Descriptor 22 FP False Positive 23 FN False Negative 24 FSM Finite State Machine 25 fps f rame per second 26 GA Genetic Algorithm 27 GMM Gaussian Mixture Model 28 GT Ground True 29 GUI Graphic User Interface 30 HCI Human Computer Interaction vii 31 HCRFs Hidden Conditional Random Fields 32 HNN Hopfield Neural Network 33 HMM Hidden Markov Model 34 HOG Histogram of Oriented Gradient 35 HSV Hue Saturation Value 36 ID IDentification 37 IP Internet Protocol 38 IR InfRared 39 ISOMAP ISOmetric MAPing 40 JI Jaccard Index 41 KLT Kanade Lucas Tomasi 42 KNN K Nearest Neighbors 43 LAN Local Area Network 44 LE Laplacian Eigenmaps 45 LLE Locally Linear Embedding 46 LRB Left Right Banded 47 MOG Mixture of Gaussian 48 MFC Microsoft Founding Classes 49 MSC Mean Shift Clustering 50 MR Magic Ring 51 NB Naive Bayesian 52 PC Persional Computer 53 PCA Principal Component Analysis 54 PDF Probability Distribution Function 55 PNG Portable Network Graphics 56 QCIF Quarter Common Intermediate Format 57 RAM Random Acess Memory 58 RANSAC RANdom SAmple Consensus 59 RBF Radial Basic Function 60 RF Random Forest 61 RGB Red Green Blue 62 RGB-D Red Green Blue Depth 63 RMSE Root Mean Square Error 64 ROI Region of Interest 65 RNN Recurrent Neural Network viii further improvements could be: – For the hand detection and segmentation: three heuristic parameters are proposed In order to improve the performance of this step, the adaptive distance can be applied by using tracking algorithms (e.g., Kalman or particle filters) These schemes also could be a research direction for the hand detection task – For the spotting dynamic hand gesture: The improvement could be a combination of appearance features of the proposed cyclical gesture patterns such as motion direction, or constraints of the closed-form on manifold space (from starting to ending points in a gesture) It is a promising research to improve the performances of the dynamic hand gesture recognition Evaluate the robustness of proposed spotting method on a extension dataset – For the hand gesture recognition: extracting common latent manifold from multiple manifolds which are built from multiple modalities of the data by suppressing specific variables, and thereby extracting the essence of data and separating irrelevant data The proposed method is further compared with the state-of-the-art methods such as C3D, HMM on both a lager size and number of gestures ❼ For real application: the proposed methods are intended to deploy with the newest version of Kinect sensors (Kinect Version 2) The depth features provided by the newest Kinect are much better than the current one Besides, multiple Kinect sensors are forwarded to combine together so that the system can identify the 3-D shape and human position in this space By this way, building a fully smart-room application for natural, friendly, and real-time interaction is a feasible solution 125 Bibliography [1] (2010) http://research.microsoft.com/en-us/um/people/zliu/actionrecorsrc/ [2] (2012) http://www.microsoft.com/en-us/kinectforwindows [3] (2012) http://www.omron.com/media/press/2012/05/e0528.html [4] (2017) http://www.samsung.com/ph/smarttv/common/guide book 3p si/waving.html [5] Abid M.R., Meszaros P.E., d Silva R.F., and Petriu E.M (2014) Dynamic hand gesture recognition for human-robot and inter-robot communication In Proceedings of The IEEE International Conference on Computational Intelligence and Virtual Environments for Measurement Systems and Applications (CIVEMSA), pp 12–17 [6] Abid M.R., Petriu E.M., and Amjadian E (2015) Dynamic sign language recognition for smart home interactive application using stochastic linear formal grammar IEEE Transactions on Instrumentation and Measurement, 64(3):pp 596– 605 [7] Ahammed M.T and Banik P.P (2015) Home appliances control using mobile phone In Proceedings of The International Conference on Advances in Electrical Engineering (ICAEE), pp 251–254 [8] Ahmed W., Chanda K., and Mitra S (2016) Vision based hand gesture recognition using dynamic time warping for indian sign language In Proceedings of The International Conference on Information Science (ICIS), pp 120–125 [9] Al-qaness M.A.A and Li F (2016) WiGeR: WiFi-Based Gesture Recognition System ISPRS International Journal of Geo-Information, 5:pp 1–17 [10] Arpit Mittal A.Z and Torr P (2011) Hand detection using multiple proposals In Proceedings of The British Machine Vision Conference (BMVC), pp 75.1–75.11 [11] Baig F., Beg S., and Fahad Khan M (2012) Controlling Home Appliances Remotely through Voice Command International Journal of Computer Applications, 48(17):pp 1–4 [12] Barczewska K and Drozd A (2013) Comparison of methods for hand gesture recognition based on dynamic time warping algorithm In Proceedings of The Federated Conference on Computer Science and Information Systems (FedCSIS), pp 207–210 [13] Baudel T and Beaudouin-Lafon M (1993) Charade: Remote control of objects using free-hand gestures Communications of the ACM Journal , 36(7):pp 28–35 126 [14] Bayer I and Silbermann T (2013) A multi modal approach to gesture recognition from audio and video data Proceedings of The 15th ACM on International Conference on Multimodal Interaction (ICMI), pp 461–466 [15] Belkin M and Niyogi P (2003) Laplacian eigenmaps for dimensionality reduction and data representation Journal of Neural Computation, 15(6):pp 1373– 1396 [16] Belongie S., Malik J., and Puzicha J (2002) Shape matching and object recognition using shape contexts IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 24(4):pp 509–522 [17] Bergh M and Van-Gool L (2011) Combining RGB and ToF Cameras for Realtime 3D Hand Gesture Interaction In Proceedings of The IEEE Workshop on Applications of Computer Vision (WACV), pp 66–72 [18] Biju V.G., Mohan A., Akhil K.A., Kumar A.K., and Varghese M.C (2016) Hand Gesture Based Low-Cost Appliance Control System IPASJ International Journal of Electronics and Communication (IIJEC), 4(4):pp 19–22 [19] Binh N.D and Ejima T (2006) A new approach dedicated to hand gesture recognition In Proceedings of The 5th IEEE International Conference on Cognitive Informatics (ICCI), volume 1, pp 62–67 [20] Bo L and Sminchisescu C (2009) Efficient match kernel between sets of features for visual recognition In Neural Information Processing Systems (NIPS), pp 135–143 [21] Bodiroˇza S., Doisy G., and Hafner V.V (2013) Position-invariant, real-time gesture recognition based on dynamic time warping In Proceedings of The 8th ACM/IEEE International Conference on Human-robot Interaction (HRI), pp 87–88 [22] Breiman L (2001) Random forests Journal of Machine Learning, 45(1):pp 5–32 [23] Bretzner L., Laptev I., and Lindeberg T (2002) Hand gesture recognition using multi-scale colour features, hierarchical models and particle filtering In Proceedings of The 6th IEEE International Conference on Automatic Face Gesture Recognition (FG), pp 423–428 [24] Burges C.J (1998) A tutorial on support vector machines for pattern recognition Data Mining and Knowledge Discovery, 2(2):pp 121–167 127 [25] Celebi S., Aydin A., Temiz T., and Arici T (2013) Gesture Recognition Using Skeleton Data with Weighted Dynamic Time Warping In Proceedings of The International Conference on Computer Vision Theory and Applications (VISAPP), volume 1, pp 620–625 [26] Cerlinca T.I and Pentiuc S.G (2011) Robust 3d hand detection for gestures recognition In Intelligent Distributed Computing V Studies in Computational Intelligence 2011 , volume 382, pp 259–264 [27] Chang C.C., Chen J.J., Tai W.K., and Han C.C (2006) New approach for static gesture recognition Journal of Information Science and Engineering (JISE), 22(5):pp 1047–1057 [28] Chen Q., El-Sawah A., Joslin C., and Georganas N.D (2005) A dynamic gesture interface for virtual environments based on hidden markov models In Proceedings of The IEEE International Workshop on Haptic Audio Visual Environments and their Applications (HAVE), pp 109–114 [29] Chen X and Koskela M (2013) Online rgb-d gesture recognition with extreme learning machines In Proceedings of The 15th ACM on International Conference on Multimodal Interaction (ICMI), pp 467–474 [30] Cheng J., Xie C., Bian W., and Tao D (2012) Feature fusion for 3d hand gesture recognition by learning a shared hidden space Pattern Recognition Letters Journal , 33(4):pp 476–484 [31] Choi J., Seo B.K., and Park J.I (2009) Robust hand detection for augmented reality interface In Proceedings of The 8th International Conference on Virtual Reality Continuum and Its Applications in Industry (VRCAI), pp 319–321 [32] Chung W.K., Wu X., and Xu Y (2009) A realtime hand gesture recognition based on haar wavelet representation In Proceedings of The IEEE International Conference on Robotics and Biomimetics (ROBIO), pp 336–341 [33] Company P (2013) PointGrab brings gesture control to home appliances [34] Cover T and Hart P (1967) Nearest neighbor pattern classification IEEE Transactions on Information Theory, 13(1):pp 21–27 [35] Dardas N.H and Georganas N.D (2011) Real-Time Hand Gesture Detection and Recognition Using Bag-of-Features and Support Vector Machine Techniques IEEE Transactions on Instrumentation and Measurement, 60(11):pp 3592–3607 128 [36] De Silvaa L., Morikawab C., and Petraa I (2012) State of the art of smart homes Engineering Applications of Artificial Intelligence Journal (EAAI), 25:pp 1313– 1321 [37] Deng L.Y., Lee D.L., Keh H.C., and Liu Y.J (2010) Shape context based matching for hand gesture recognition In Proceedings of The IET International Conference on Frontier Computing(FC), pp 436–444 [38] Dipietro L., Sabatini M., and Dario P (2008) A survey of glove-based systems and their applications IEEE Transactions on Systems, Man, and CyberneticsPart C , 38:pp 461–482 [39] Do J.h., Jang H., Jung S.H., Jung J., and Bien Z (2002) Soft Remote Control System in the Intelligent Sweet Home Humanfriendly Welfare Robotic Systems Journal , pp 70–30 [40] Doliotis P., Athitsos V., Kosmopoulos D., and Perantonis S (2012) Hand Shape and 3D Pose Estimation using Depth Data from a Single Cluttered Frame In Proceedings of The International Symposium on Visual Computing (ISVC), pp 148–158 [41] Elmezain M., Al-Hamadi A., and Michaelis B (2009) A novel system for automatic hand gesture spotting and recognition in stereo color image sequences Journal of WSCG, 17(1-3):pp 89–96 [42] Elmezain M., Al-Hamadi A., and Michaelis C (2008) Real-Time Capable System for Hand Gesture Recognition Using Hidden Markov Models in Stereo Color Image Sequences Journal of WSCG, 16:pp 65–72 [43] Elmezain M., Al-Hamadi A., Sadek S., and Michaelis B (2010) Robust methods for hand gesture spotting and recognition using hidden markov models and conditional random fields In Proceedings of The 10th IEEE International Symposium on Signal Processing and Information Technology, pp 131–136 [44] Escalera S., Gonzalez J., Baro X., Reyes M., Lopes O., Guyon I., Athitsos V., and Escalante H (2013) Multi-modal gesture recognition challenge 2013: Dataset and results In Proceedings of The 15th ACM on International Conference on Multimodal Interaction (ICMI), pp 445 – 452 [45] Escobedo-Cardenas E and Camara-Chavez G (2015) A robust gesture recognition using hand local data and skeleton trajectory In Proceedings of The IEEE International Conference on Image Processing (ICIP), pp 1240–1244 [46] Freeman W.T and Weissman C.D (1995) Television control by hand gestures 129 [47] Friedman N., Geiger D., and Goldszmidt M (1997) Bayesian network classifiers Journal of Machine Learning, 29(2-3):pp 131–163 [48] Gastaldi G., Pareschi A., Sabatini S.P., Solari F., and Bisio G.M (2005) A manmachine communication system based on the visual analysis of dynamic gestures In Proceedings of The IEEE International Conference on Image Processing, volume 3, pp 397–400 [49] Gharasuie M.M and Seyedarabi H (2013) Real-time dynamic hand gesture recognition using hidden markov models In Proceedings of The 8th Iranian Conference on Machine Vision and Image Processing (MVIP), pp 194–199 [50] Ghosh D.K and Ari S (2011) A static hand gesture recognition algorithm using k-mean based radial basis function neural network In Proceedings of The 8th International Conference on Information, Communications and Signal Processing (ICICS), pp 1–5 [51] Grif H.S and Farcas C.C (2016) Mouse Cursor Control System Based on Hand Gesture Procedia Technology Journal , 22:pp 657–661 [52] Haque S.M.A., Kamruzzaman S.M., and Islam M.A (2010) A System for Smart Home Control of Appliances based on Timer and Speech Interaction The Computing Research Repository (CoRR), abs/1009.4992:p [53] Hasan H and Kareem S.A (2014) Humancomputer interaction using visionbased hand gesture recognition systems: a survey Neural Computing and Applications, 25:pp 251–261 [54] Haykin S (2007) Neural Networks: A Comprehensive Foundation Prentice Hall PTR, 3rd edition edition [55] Henia O.B and Bouakaz S (2011) 3d hand model animation with a new datadriven method In Proceedings of the 2011 Workshop on Digital Media and Digital Content Management (DMDCM), pp 72–76 [56] Herrera D., Kannala J., and Heikkila J (2012) Joint depth and color camera calibration with distortion correction IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 34(10):pp 2058–2064 [57] Huang C.L and Huang W.Y (1998) Sign language recognition using modelbased tracking and a 3d hopfield neural network Journal of Machine Vision and Applications (MVA), 10(5-6):pp 292–307 130 [58] Huang D.Y., Hu W.C., and Chang S.H (2011) Gabor filter-based hand-pose angle estimation for hand gesture recognition under varying illumination Expert Systems with Applications Journal , 38(5):pp 6031–6042 [59] huei Liang R and Ouhyoung M (1996) A sign language recognition system using hidden markov model and context sensitive search In Proceedings of The ACM Symposium on Virtual Reality Software and Technology (VRST), pp 59–66 [60] Jaemin L., Takimoto H., Yamauchi H., Kanazawa A., and Mitsukura Y (2013) A robust gesture recognition based on depth data In Proceedings of The 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision, pp 127–132 [61] Jangyodsuk P., Conly C., and Athitsos V (2014) Sign language recognition using dynamic time warping and hand shape distance based on histogram of oriented gradient features In Proceedings of The 7th International Conference on PErvasive Technologies Related to Assistive Environments (PETRA), pp 50:1–50:6 [62] Jing L., Zhou Y., Cheng Z., and Huang T (2012) Magic Ring: A Finger-Worn Device for Multiple Appliances Control Using Static Finger Gestures Sensors Journal , 12(5):pp 5775–5790 [63] Jones M.J and Rehg J.M (1999) Statistical color models with application to skin detection In Proceedings of The IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), volume 1, pp 274 – 280 [64] J.Shi and C.Tomasi (1994) Good features to track In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 593–600 [65] Kahol K., Tripathi P., and Panchanathan S (2004) Automated gesture segmentation from dance sequences In Proceedings of The 6th IEEE International Conference on Automatic Face and Gesture Recognition (FG), pp 883–888 [66] Kang H., Lee C.W., and Jung K (2004) Recognition-based gesture spotting in video games Journal of Pattern Recognition Letters, 25(15):pp 1701–1714 [67] Kim D., Song J., and Kim D (2007) Simultaneous Gesture Segmentation and Recognition Based on Forward Spotting Accumlative HMMs Journal of Pattern Recognition Society, 40:pp 1–4 [68] Kim K., Kim J., Choi J., Kim J., and Lee S (2015) Depth Camera-Based 3D Hand Gesture Controls with Immersive Tactile Feedback for Natural Mid-Air Gesture Interactions 15(1):pp 1022–1046 131 [69] Kim M.s., Lee C.H., and Eng C (2016) Hand Gesture Recognition for Kinect v2 Sensor in the Near Distance Where Depth Data Are Not Provided 10(12):pp 407–418 [70] Lamberti L and Camastra F (2011) Real-tim hand gesture recognition using a color glove In Proceedings of The Springer 16th International conference on Image Analysis (ICIAP), pp 365–373 [71] Lee H.K and Kim J.H (1999) An hmm-based threshold model approach for gesture recognition IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 21(10):pp 961–973 [72] Lee-Ferng J., del Solar J.R., Verschae R., and Correa M (2009) Dynamic gesture recognition for human robot interaction In Proceedings of The 6th Latin American Robotics Symposium (LARS), pp 1–8 [73] Lian S., Hu W., and Wang K (2014) Automatic user state recognition for hand gesture based low-cost television control system IEEE Transactions on Consumer Electronics, 60(1):pp 107–115 [74] Lin T and Zha H (2008) Riemannian manifold learning IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 30(5):pp 796–809 [75] Liu K.C and Chan C.T (2017) Significant change spotting for periodic human motion segmentation of cleaning tasks using wearable sensors Sensors Journal , 17(1):pp 187.1–187.19 [76] Liu L and Shao L (2013) Learning discriminative representations from rgbd video data In Proceedings of The 23rd International Joint Conference on Artificial Intelligence (IJCAI), pp 1493–1500 [77] Lu W.L and Little J.J (2006) Simultaneous tracking and action recognition using the pca-hog descriptor In Proceedings of The 3rd Canadian Conference on Computer and Robot Vision (CRV), pp – [78] Lucas B.D and Kanade T (1981) An iterative image registration technique with an application to stereo vision In Proceedings of The International Joint Conference on Artificial Intelligence (IJCAI), pp 674–679 [79] Mace D., Gao W., and Coskun A (2013) Accelerometer-based hand gesture recognition using feature weighted naăve bayesian classifiers and dynamic time warping In Proceedings of The Companion Publication of the 2013 International Conference on Intelligent User Interfaces Companion (IUI), pp 83–84 132 [80] Makihara Y., Mori A., and Yagi Y (2011) Periodic Temporal Super Resolution Based on Phase Registration and Manifold Reconstruction Journal of IPSJ Transactions on Computer Vision and Applications (CVA), 3(1):pp 134–147 [81] Malima A., Ozgur E., and Cetin M A Fast Algorithm for Vision-based Hand Gesture Recognition For Robot Control, booktitle =Proceedings of The IEEE Signal Processing and Communications Applications (SIU), pages = 1–4, year =2006 [82] Maqueda A.I., del Blanco c., Jaureguizar F.G., and N (2015) Human-computer interaction based on visual recognition using volumegrams of local binary patterns In Proceedings of The IEEE International Conference on Consumer Electronics (ICCE), pp 583–584 [83] Marcel S., Bernier O., Viallet J.E., and Collobert D (2000) Hand gesture recognition using input-output hidden Markov models In Proceedings of The 4th IEEE International Conference on Automatic Face and Gesture Recognition (FG), pp 456–461 [84] Martin J., Devin V., and Crowley J.L (1998) Active hand tracking In Proceedings of The 3rd IEEE International Conference on Automatic Face and Gesture Recognition (FG), pp 573–578 [85] Matnani P (2015) Glove Based and Accelerometer based Gesture Control: A Literature International Journal of Research and Engineering (IJRE), 3(6):pp 216–221 [86] McGuinness K and O Connor N.E (2010) A comparative evaluation of interactive segmentation algorithms Pattern Recognition Journal , 43(2):pp 434–444 [87] Molchanov P., Yang X., Gupta S., Kim K., Tyree S., and Kautz J (2016) Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural networks In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp 4207–4215 [88] Molina J., Alessandro M.E.v., Pardas M., Ferran C., Bescos J., Marques F., and Martinez J.M (2013) Real-time user independent hand gesture recognition from time-of-flight camera video using static and dynamic models Machine Vision and Applications, 24:pp 187–204 [89] Morguet P and Lang M (1998) Spotting dynamic hand gestures in video image sequences using hidden markov models In Proceedings of The International Conference on Image Processing (ICIP), volume 3, pp 193–197 133 [90] Mu˜ noz Salinas R., Medina-Carnicer R., Madrid-Cuevas F.J., and CarmonaPoyato A (2008) Depth silhouettes for gesture recognition Pattern Recognition Letters Journal , 29(3):pp 319–329 [91] Nguyen V.T., Le T.L., Tran T.T.H., Remy M., and Vincent C (2015) A New Hand Representation Based on Kernels for Hand Posture Recognition In Proceedings of The 11th IEEE International Conference on Automatic Face and Gesture Recognition (FG), volume 1, pp 1–6 [92] Nichols J., Myers B.A., Higgins M., Hughes J., Harris T.K., Rosenfeld R., and Pignol M (2002) Generating remote control interfaces for complex appliances In Proceedings of The 15th annual ACM symposium on User interface software and technology (UIST), pp 161–170 [93] Ohn-bar E., Member S., and Trivedi M.M (2014) Hand Gesture Recognition in Real Time for Automotive Interfaces: A Multimodal Vision-Based Approach and Evaluations IEEE Transactions on Intelligent Transportation System, 15(6):pp 2368–2377 [94] Ong S.C.W., Ranganath S., and Venkatesh Y.V (2006) Understanding gestures with systematic variations in movement dynamics Pattern Recognition Journal , 39(9):pp 1633–1648 [95] Oreifej O and Liu Z (2013) HON4D: Histogram of oriented 4D normals for activity recognition from depth sequences In Proceedings of The IEEE Computer Society Conference on Computer Vision and Pattern Recognition(CVPR), pp 716–723 [96] Oz C and Leu C (2011) American sign language word recognition with a sensory glove using artificial neural networks Engineering Applications of Artificial Intelligence Journal (EAAI), 24(7):pp 1204–1213 [97] Park M., Hasan M., Kim J., and Chae O (2012) Hand Detection and Tracking Using Depth and Color Information [98] Pedregosa F., Varoquaux G., Gramfort A., Michel V., Thirion B., Grisel O., Blondel M., Prettenhofer P., Weiss R., Dubourg V., Vanderplas J., Passos A., Cournapeau D., Brucher M., Perrot M., and Duchesnay E (2011) Scikit-learn: Machine learning in python Journal of Machine Learning Research, 12:pp 2825– 2830 [99] Pisharady P.K., Vadakkepat P., and Loh A.P (2012) Attention Based Detection and Recognition of Hand Postures Against Complex Backgrounds International Journal of Computer Vision (IJCV), 101:pp 403–419 134 [100] Plouffe G and Cretu A.M (2016) Static and dynamic hand gesture recognition in depth data using dynamic time warping IEEE Transactions on Instrumentation and Measurement, 65(2):pp 305–316 [101] Poularakis S and Katsavounidis I (2014) Finger detection and hand posture recognition based on depth information In Proceedings of The IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp 4329– 4333 [102] Prajapat S and Upadhyay H (2017) http://www.azilen.com/blog/automationof-home-appliances-using-raspberry-pi-part-2/ [103] Premaratne P and Nguyen Q (2007) Consumer electronics control system based on hand gesture moment invariants IET Computer Vision, 1(1):pp 35–41 [104] Quinlan J.R (1986) Induction of decision trees Machine Learning Journal , 1(1):pp 81–106 [105] Ramamoorthy A., Vaswani N., Chaudhury S., and Banerjee S (2003) Recognition of dynamic hand gestures Journal of Pattern Recognition, 36(9):pp 2069 – 2081 [106] Ramamoorthy R and Keerthana K (2015) Hand Gesture Based Television Control, journal= Proceedings of The National Conference on Research Advances in Communication, Computation, Electrical Science and Structures (NCRACCESS) (8):pp 41–43 [107] Rashid O., Al-Hamadi A., and Michaelis B (2010) Integration of gesture and posture recognition systems for interpreting dynamic meanings using particle filter In Proceedings of The International Conference of Soft Computing and Pattern Recognition, pp 47–50 [108] Rautaray, S S., and Agrawal A (2015) Vision based hand gesture recognition for human computer interaction: A survey Artificial Intelligence Review Journal , 43(1):pp 1–54 [109] Rautaray S and Agrawal A (2015) Vision based hand gesture recognition for human computer interaction: a survey Artificial Intelligence Review Journal , 43(1):pp 1–54 [110] Ravikiran J., Mahesh K., Mahishi S., Dheeraj R., Sudheender S., and Pujari N.V (2009) Finger Detection for Sign Language Recognition Proceedings of The International MultiConference of Engineers and Computer Scientists (IMECS), I:pp 489–493 135 [111] Ren Z., Yuan J., and Zhang Z (2011) Robust Hand Gesture Recognition Based on Finger-Earth Movers Distance with a Commodity Depth Camera In Proceedings of The 19th ACM international conference on Multimedia (MM), pp 1093–1096 [112] R.Mccartney J Yuan H.B (2015) Gesture Recognition with the Leap Motion Controller Gesture Recognition with the Leap Motion Controller In International Conference on Image Processing, Computer Vision and Pattern Recognition (IPCV), pp 2–9 [113] Roweis S.T and Saul L.K (2000) Nonlinear Dimensionality Reduction by Locally Linear Embedding Science, 290:pp 2323 – 2326 [114] S Escalera X Baro J.G.M.A.B.M.M.M.R.V.P.L.H.J.E.J.S and Guyon I (2014) ChaLearn Looking at People Challenge 2014: Dataset and results In The European Workshops on Computer Vision (WECCV), pp 1–8 [115] S Mohamed Mansoor Roomi R.J.P and Jayalakshmi H (2010) Hand Gesture Recognition for Human-Computer Interaction Journal of Computer Science, 6(9):pp 1002–1007 [116] Sangineto E and Cupelli M (2012) Real-time viewpoint-invariant hand localization with cluttered backgrounds Image Vision Computting Journal , 30(1):pp 26–37 [117] Sharma P.K., Sheoran R., and Kathania D.K (2015) A Study of Hand Gesture Technique to Communicate with a Robot International Journal of Engineering Research and Technology (IJERT), 4:pp 12–13 [118] Shen X., Hua G., Williams L., and Wu Y (2012) Dynamic hand gesture recognition: An exemplar-based approach from motion divergence fields Image and Vision Computing Journal , 30(3):pp 227–235 [119] Shimada A., Yamashita T., and i Taniguchi R (2013) Hand gesture based tv control system In Proceedings of The 19th Korea-Japan Joint Workshop on Frontiers of Computer Vision, pp 121–126 [120] Shin S and Sung W (2016) Dynamic hand gesture recognition for wearable devices with low complexity recurrent neural networks The Computing Research Repository (CoRR), abs/1608.04080:pp 3–6 [121] Shitole S.M., Patil S.B., and Narote S.P (2013) Article: Dynamic hand gesture recognition using pca, pruning and ann International Journal of Computer Applications (IJCA), 74(2):pp 24–29 136 [122] Sigalas M., Baltzakis H., and Trahanias P (2010) Gesture recognition based on arm tracking for human-robot interaction In Proceedings of The IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp 5424– 5429 [123] Smedt Q.D., Wannous H., and Vandeborre J.P (2016) http://www-rech.telecomlille.fr/dhgdataset/ [124] Smedt Q.D., Wannous H., and Vandeborre J.P (2016) Skeleton-based dynamic hand gesture recognition In Proceedings of The IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp 1206–1214 [125] Sohn M.K., Lee S.H., Kim D.J., Kim B., and Kim H (2012) A comparison of 3d hand gesture recognition using dynamic time warping In Proceedings of The 27th Conference on Image and Vision Computing New Zealand (IVCNZ), pp 418–422 [126] Solanki U.V and Desai N.H (2011) Hand gesture based remote control for home appliances: Handmote In Proceedings of The IEEE World Congress on Information and Communication Technologies (WICT), pp 419–423 [127] Song L and Takatsuka M (2005) Real-time 3d finger pointing for an augmented desk In Proceedings of The 6th Australasian Conference on User Interface (AUIC), volume 40, pp 99–108 [128] Stauffer C and Grimson W (1999) Adaptive background mixture models for real-time tracking In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), volume 2, pp 246 – 252 [129] Suk H.I., Sin B.K., and Lee S.W (2010) Hand gesture recognition based on dynamic bayesian network framework Pattern Recognition Journal , 43(9):pp 3059–3072 [130] Tenenbaum J.B., de Silva V., and Langford J.C (2000) A global geometric framework for nonlinear dimensionality reduction Science Journal , 290(5500):pp 2319–2323 [131] Tran D., Bourdev L., Fergus R., Torresani L., and Paluri M (2015) Learning Spatiotemporal Features With 3D Convolutional Networks In Proceedings of The IEEE International Conference on Computer Vision (ICCV), pp 4489–4497 [132] Tran T.T.H and Nguyen T.T.M (2010) Invariant lighting hand posture classification In Proceedings of The 2010 IEEE International Conference on Progress in Informatics and Computing (PIC), volume 2, pp 827–831 137 [133] Verma R and Dev A (2009) Vision based hand gesture recognition using finite state machines and fuzzy logic In Proceedings of The International Conference on Ultra Modern Telecommunications Workshops, pp 1–6 [134] Viola P and Jones M.J (2004) Robust real-time face detection International Journal of Computer Vision (IJCV), 57(2):pp 137–154 [135] W Gao G Fang D.Z and Chen Y (2004) Transition movement models for large vocabulary continuous sign language recognition In Proceedings of The 6th IEEE International Conference Automatic Face and Gesture Recognition (FG), pp 553 – 558 [136] Wang T.S., Shum H.Y., Xu Y.Q., and Zheng N.N (2001) Unsupervised analysis of human gestures In Proceedings of The 2nd IEEE Pacific Rim Conference on Multimedia Beijing Advances in Multimedia Information Processing (PCM), volume 2195, pp 174–181 [137] WangGW Zhang C Z.J (2012) An application of classifier combinationmethods in hand gesture recognition Mathematical Problems in Engineering Journal , 2012:pp 1–17 [138] Wu D., Pigou L., Kindermans P.j., Le N., Shao L., Dambre J., and Odobez J.m (2016) Deep Dynamic Neural Networks for Multimodal Gesture Segmentation and Recognition IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 38(8):pp 1–16 [139] Xu D (2006) A neural network approach for hand gesture recognition in virtual reality driving training system of spg In Proceedings of The 18th International Conference on Pattern Recognition (ICPR), volume 3, pp 519–522 [140] Y S., D D., and R D (2011) Tracking Body and Hands For Gesture Recognition: NATOPS Aircraft Handling Signals Database In Proceedings of The 9th IEEE International Conference on Automatic Face and Gesture Recognition (FG), pp 500–506 [141] Yang H.D (2015) Sign Language Recognition with the Kinect Sensor Based on Conditional Random Fields Sensors Journal , 15(1):pp 135–147 [142] Yang H.D., Sclaroff S., and Lee S.W (2009) Sign Language Spotting with a Threshold Model Based on Conditional Random Fields IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 31(7):pp 1264–1277 [143] Yang S.E., Do J.H., Jang H., Jung J.W., and Bien Z (2006) Advanced Soft Remote Control System in Human-friendliness International Journal of Soft Com- 138 puting and Intelligent Systems, and Symposium on Advanced Intelligent Systems (SCIS and ISIS), 2006:pp 218–222 [144] Yang X and Tian Y (2014) Super Normal Vector for Action Recognition Using Depth Sequences In Proceedings of The IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), pp 804–811 [145] Yang Z., Li Y., Chen W., and Zheng Y (2012) Dynamic hand gesture recognition using hidden markov models In Proceedings of The 7th International Conference on Computer Science Education (ICCSE), pp 360–365 [146] Yuan R., Cheng J., Li P., Chen G., Xie C., and Xie Q (2010) View invariant hand gesture recognition using 3d trajectory In Proceedings of The 8th World Congress on Intelligent Control and Automation, pp 6315–6320 [147] Zabulis X., Baltzakis H., and Argyros A (2009) Vision-based Hand Gesture Recognition for Human Computer Interaction Lawrence Erlbaum Associates, Inc (LEA) [148] Zaman M.F., Mossarrat S.T., Islam F., and Karmaker D (2015) Real-time hand detection and tracking with depth values In Proceedings of The International Conference on Advances in Electrical Engineering (ICAEE), pp 129–132 [149] Zhang X., Chen X., Wang W.h., Yang J.h., Lantz V., and Wang K.q (2009) Hand gesture recognition and virtual game control based on 3d accelerometer and emg sensors In Proceedings of The 14th International Conference on Intelligent User Interfaces (IUI), pp 401–406 [150] Zhang Y., Yao Y., and Luo Y (2015) An improved hmm/svm dynamic hand gesture recognition algorithm In Proceedings of The SPIE , volume 9672, pp 96720D–96720D–7 [151] Zou Z., Premaratne P., Monaragala R., Bandara N., and Premaratne M (2010) Dynamic hand gesture recognition system using moment invariants In Proceedings of The 5th International Conference on Information and Automation for Sustainability, pp 108–113 139 ... 3.2.3 Estimating heuristic parameters 3.2.3.1 Estimating parameters of background model for body detection 3.2.3.2 Estimating the distance from... technique 17 Table 1.5 The existing in-air gesture-based systems 18 Table 1.6 The existing vision-based dynamic hand gesture methods 36 Table 2.1 The existing Hand gesture datasets... intelligent computing, smart devices, and new communication protocols In term of automating ability, most of advanced technologies are focusing on either saving energy or facilitating the control