Active recognition in pervasive intelligent envionments

Free ebooks ==> www.ebook777.com www.ebook777.com Free ebooks ==> www.ebook777.com ATLANTIS A MBIENT AND P ERVASIVE I NTELLIGENCE VOLUME S ERIES E DITOR : I SMAIL K HALIL Free ebooks ==> www.ebook777.com Atlantis Ambient and Pervasive Intelligence Series Editor: Ismail Khalil, Linz, Austria (ISSN: 1875-7642) Aims and scope of the series The book series ‘Atlantis Ambient and Pervasive Intelligence’ publishes high quality titles in the fields of Pervasive Computing, Mixed Reality, Wearable Computing, LocationAware Computing, Ambient Interfaces, Tangible Interfaces, Smart Environments, Intelligent Interfaces, Software Agents and other related fields We welcome submission of book proposals from researchers worldwide who aim at sharing their results in this important research area For more information on this series and our other book series, please visit our website at: www.atlantis-press.com/publications/books A MSTERDAM – PARIS c ATLANTIS PRESS www.ebook777.com Free ebooks ==> www.ebook777.com Activity Recognition in Pervasive Intelligent Environments Liming Chen Chris D Nugent School of Computing and Mathematics University of Ulster Shore Road, Newtownabbey County Antrim BT37 0QB United Kingdom School of Computing and Mathematics University of Ulster Shore Road, Newtownabbey County Antrim BT37 0QB United Kingdom Jit Biswas Jesse Hoey Networking Protocols Department Institute of Infocomm Research (I2R) Singapore School of Computer Science University of Waterloo 200 University Avenue West Waterloo, Ontario N2L 3G1 Canada A MSTERDAM – PARIS Free ebooks ==> www.ebook777.com Atlantis Press 8, square des Bouleaux 75019 Paris, France For information on all Atlantis Press publications, visit our website at: www.atlantis-press.com Copyright This book, or any parts thereof, may not be reproduced for commercial purposes in any form or by any means, electronic or mechanical, including photocopying, recording or any information storage and retrieval system known or to be invented, without prior permission from the Publisher Atlantis Ambient and Pervasive Intelligence Volume 1: Agent-Based Ubiquitous Computing – Eleni Mangina, Javier Carbo, José M Molina Volume 2: Web-Based Information Technologies and Distributed Systems – Alban Gabillon, Quan Z Sheng, Wathiq Mansoor Volume 3: Multicore Systems On-Chip: Practical Software/Hardware Design – Abderazek Ben Abdallah ISBNs Print: E-Book: ISSN: 978-90-78677-42-0 978-94-91216-05-3 1875-7642 c 2011 ATLANTIS PRESS www.ebook777.com Free ebooks ==> www.ebook777.com Preface We have now entered an era where technology is being embedded transparently and seamlessly within our surrounding environments This is being driven by decreasing hardware costs, increased functionality and battery life along with improved levels of pervasiveness With such a technology rich paradigm we are now witnessing for the first time intelligent environments with the ability to provide support within our homes, the community and in the workplace The knock-on effect has an impact on both improved living experiences within the environment along with increased levels of independence At a general level we can decompose the construct of an intelligent environment into three main components In the first instance we have the core sensing technology which has the ability to record the interactions with the environment These may be in the form of for example video, contact sensors or motion sensors A data processing module has the task to infer decisions based on the information gleaned from the sensing technology and with the third and final component providing the feedback to those within the environment via a suite of multi-modal interfaces It has been the aim of this text to focus specifically on the data processing module, specifically focusing on the notion of activity recognition Within the domain of intelligent environments some may have the view that the process of activity recognition forms the critical path in providing a truly automated environment It is tasked with extracting and establishing meaningful activities from a myriad of sensor activations Although work in this area is still deemed to be emerging, the initial results achieved have been more than impressive This text represents a consolidation of 14 Chapters presenting leading research results in the area of activity recognition The material addressed ranges from collective state-of-the-art reviews, to probabilistic and ontological based reasoning approaches to specific examples in the areas of assistance with activities of daily living v Free ebooks ==> www.ebook777.com vi Activity Recognition in Pervasive Intelligent Environments The text is intended for those working within the area of intelligent environments who require a detailed understanding of the processes of activity recognition along with their technical variances The inclusion of specific case studies assists with the further contextulisation of the theoretical concepts which have been introduced We would like to take this opportunity to thank all of the Authors for their timely contributions in provision of their Chapters in both initial and revised formats along with all of the dedicated efforts provided by those who undertook to review the material We would also like to express our gratitude to Ismail Khalil who providing the inspiration to undertake this Project and provided continual motivation and advice throughput We would also like to thank Atlantis Press for supporting the text and also for their help in producing the final version of the text We hope that the text becomes a valuable reference source within the field of activity recognition and assists in further progress the translation of research efforts into tangible large scale intelligent environments that we can all benefit from Liming Chen Chris Nugent Jit Biswas Jesse Hoey www.ebook777.com Free ebooks ==> www.ebook777.com Contents Preface v 1 Activity Recognition: Approaches, Practices and Trends Liming Chen, Ismail Khalil 1.1 Introduction 1.2 Activity recognition approaches and algorithms 1.2.1 Activity recognition approaches 1.2.2 Activity recognition algorithms 1.2.3 Ontology-based activity recognition 1.3 The practice and lifecycle of ontology-based activity recognition 10 1.3.1 Domain knowledge acquisition 11 1.3.2 Formal ontology modelling 12 1.3.3 Semantic sensor metadata creation 13 1.3.4 Semantic sensor metadata storage and retrieval 14 1.3.5 Activity recognition 15 1.3.6 Activity model learning 16 1.3.7 Activity assistance 17 1.4 An exemplar case study 18 1.5 Emerging research on activity recognition 22 1.6 1.5.1 Complex activity recognition 22 1.5.2 Domain knowledge exploitation 24 1.5.3 Infrastructure mediated activity monitoring 25 1.5.4 Abnormal activity recognition 26 Conclusions 26 References 27 vii Free ebooks ==> www.ebook777.com viii Activity Recognition in Pervasive Intelligent Environments Possibilistic Activity Recognition 33 P.C Roy, S Giroux, B Bouchard, A Bouzouane, C Phua, A Tolstikov, and J Biswas 2.1 Introduction 33 2.2 Overall Picture of Alzheimer’s disease 36 2.3 Related Work 37 2.4 Possibilistic Activity Recognition Model 41 2.5 2.6 2.4.1 Environment Representation and Context 42 2.4.2 Action Recognition 42 2.4.3 Behavior Recognition 45 2.4.4 Overview of the activity recognition process 48 Smart Home Validation 48 2.5.1 Results 50 2.5.2 Discussion 52 2.5.3 Summary of Our Contribution 55 Conclusion 56 References 56 Multi-user Activity Recognition in a Smart Home 59 Liang Wang, Tao Gu, Xianping Tao, Hanhua Chen, and Jian Lu 3.1 Introduction 59 3.2 Related Work 61 3.3 Multi-modal Wearable Sensor Platform 63 3.4 Multi-chained Temporal Probabilistic Models 64 3.5 3.4.1 Problem Statement 64 3.4.2 Feature Extraction 65 3.4.3 Coupled Hidden Markov Model 66 3.4.4 Factorial Conditional Random Field 68 3.4.5 Activity Models in CHMM and FCRF 70 Experimental Studies 71 3.5.1 Trace Collection 71 3.5.2 Evaluation Methodology 73 3.5.3 Accuracy Performance 73 www.ebook777.com Free ebooks ==> www.ebook777.com Contents ix 3.6 Conclusions and Future Work 77 References 79 Smart Environments and Activity Recognition: a Logic-based Approach 83 F Mastrogiovanni, A Scalmato, A Sgorbissa, and R Zaccaria 4.1 Introduction 83 4.2 Related Work 86 4.3 Representation of Temporal Contexts 90 4.4 Assessment of Temporal Contexts 97 4.5 Experimental Results and Discussion 98 4.5.1 An Example of System Usage 100 4.5.2 A Discussion about context assessment complexity and system performance 105 4.6 Conclusion 106 References 108 ElderCare: An Interactive TV-based Ambient Assisted Living Platform 111 D Lopez-de-Ipina, S Blanco, X Laiseca, I Diaz-de-Sarralde 5.1 Introduction 111 5.2 Related Work 112 5.3 The ElderCare Platform 114 5.3.1 5.4 5.5 Eldercare Platform Components 116 Implementation Overview 117 5.4.1 Eldercare’s Local System 118 5.4.2 ElderCare’s Central Server 119 5.4.3 ElderCare’s Mobile Client 120 Conclusion and Further Work 124 References 124 An Ontology-based Context-aware Approach for Behaviour Analysis 127 Shumei Zhang, Paul McCullagh, Chris Nugent, Huiru Zheng 6.1 Introduction 127 6.2 Related Work 129 6.3 Data Collection and Ontological Context Extraction 131 Free ebooks ==> www.ebook777.com Activity Recognition and Healthier Food Preparation 14.2 315 The Role of Technology for Healthier Eating There is convincing evidence that a healthier diet generally protects against diet related conditions such as obesity, diabetes, heart disease and even cognitive decline Conventional intervention and education programs usually represent the methodology of choice to direct people’s attention towards healthier nutrition – though often only with limited success For many people difficulties in preparing healthier food has been identified as one important cause for poor diet [7] Technology based approaches that support people in their kitchen activities can be the key to overcome this fundamental barrier to healthier eating 14.2.1 Current dietary guidelines Current global dietary guidelines promote a diet that is low in total fat, saturated fat and free sugars (added sugars plus those found in natural juices, syrups and honey) and high in fruits, vegetables dietary fibre This type of diet will help protect against many diet-related chronic conditions including cardiovascular disease (heart attack and stroke), some cancers (e.g bowel cancer, oral cancer), diabetes, obesity and tooth decay [8] The World Health Organization recommends that 15 to 30% of total dietary energy (kilocalories) should come from dietary fat and that intake of free sugars should be restricted to less than 10% of total energy intake [8] An intake of at least 400g of fruits and vegetables per day is recommended as they are good sources of vitamins, minerals and fibre Wholegrain varieties of cereals are recommended as these too are rich in micronutrients and dietary fibre Diets high in fat and or free sugars i.e ‘energy dense diets’ are a risk factor for the development of obesity Whereas a high intake of dietary fibre, regular physical activity and home environments that promote healthier food and activity choices may protect against development of overweight and obesity Consumption of saturated fats increases the risk of developing type diabetes, whereas a diet rich in wholegrain foods, fruits and vegetables is associated with a reduced risk of diabetes Saturated fats, especially those found in dairy products and meat, and trans fats increase the risk of heart disease and stroke whereas unsaturated fats (e.g sunflower and olive oils) and fish oils (found in oily fish) are associated with a lower risk of these diseases A high intake of salt can increase blood pressure and the risk of stroke and coronary heart disease and therefore restricting total salt intake to less than 5g per day (about one teaspoon) is recommended A high fibre diet, rich in wholegrain foods and fruits and vegetables can reduce the risk of cardiovascular disease Free ebooks ==> www.ebook777.com 316 14.2.2 Activity Recognition in Pervasive Intelligent Environments Barriers to healthier eating with focus on preparation Many barriers to consuming a healthier diet have been identified and include poor nutritional knowledge, taste preferences for less healthy foods, budgetary restrictions, restricted access to a variety of foods and chewing and eating problems In addition to these barriers research has shown that limited food preparation and cooking skills is also a barrier to eating more healthily [9] Possession of food preparation skills has the potential to empower individuals to make beneficial food choices To be able to make a range of healthy meals, a number of basic food preparation and cooking skills are required There is some evidence to show that level of food preparation within the home is related to a healthier diet [10] and that avoidance of cooking due to a busy lifestyle may be a factor contributing to higher energy (kilocalorie) intakes in women [11] 14.2.3 Why technology-based approach to healthier cooking? Cooking method can have a major impact on the nutrient profile of a meal For example, grilled meat will have significantly less fat compared with fried meat Cooking potatoes in unsaturated vegetable oil in preference to roasting in animal fat will significantly impact on the saturated fat content of the meal and adding more vegetables to dishes such as casseroles, pizza or pasta sauces and choosing and preparing wholegrain varieties of foods e.g wholemeal breads will impact positively of the nutrient profile of the prepared food Finding means of encouraging and promoting healthier food preparation practices on a daily basis within the home is therefore a challenge which could be addressed using pervasive technology Despite possession of cooking skills being one of the factors indicative of a better diet, cooking is being taught less in schools today than it was several decades ago Community based cooking initiatives, that aim to address the lack of cooking skills in today’s society, have been met with some success However, partaking in such initiatives requires motivation, commitment, traveling and time which may not suit contemporary busy lifestyles Alternative means of encouraging and developing the population’s confidence and skill to prepare food are therefore necessary Pervasive computing and use of situated digital interventions provides an alternative modern day platform to use to teach cooking within the home Such technology may be used to teach cooking skills, provide prompts to cook using healthier ingredients and methods and to increase overall cooking confidence The ability to provide individualized feedback on the nutritional profile of the food prepared and the impact of changing cooking methods and ingredients on the nutritional profile of www.ebook777.com Free ebooks ==> www.ebook777.com Activity Recognition and Healthier Food Preparation 317 the food is also possible, and warrants exploration as providing individualized feedback is one effective means of encouraging positive dietary behavior 14.2.4 Evaluation and assessment of cooking skills Nutrition education programs often promote increased cooking as a means to improve diet, yet there are few data on the relationship between the actual level of cooking activity and skill and food and nutrient intake One explanation for this could be the lack of valid and sensitive measures of cooking activity and skill Most research to date has relied on self reported perception of cooking confidence or used questionnaires to assess level of cooking knowledge and or activity Employing embedded sensing technology to monitoring food preparation behavior and activity in situ may provide a more scientifically robust means of assessing cooking skills, activity and the impact on dietary behavior 14.3 Activity Recognition in the Kitchen – The State-of-the-Art Technology supported food preparation substantially relies on the automatic analysis of activities taking place in people’s kitchens In the literature a number of approaches have been described where, for this purpose, sensors capable of sensing the environment at sufficient granularity have been added to either traditional or enhanced lab-based kitchens Given this technological basis, a number of applications using activity recognition and contextaware computing are imaginable In the following we will provide a summary of related work, which includes a brief overview of sensor-based activity recognition in general and a summary of the most relevant instrumented kitchen applications 14.3.1 Sensor-based Activity Recognition A considerable body of research addresses activity recognition using wearable sensors (see the recent survey of Atallah and Young [12]) In many of these studies accelerometers are worn by people engaged in activities of daily living (ADL) These ADL normally involve gross movements of the body such as sitting down, walking, cycling, etc In a small number of cases sensors have been embedded into tools and utensils to allow the analysis of more fine grained ADL [6, 13] Most AR techniques utilize a sliding window approach, extracting fixed length portions of sensor data for which features are calculated – Huyn & Schiele provide a comparison of feature extraction methods for AR [14] The most widely adopted approach is to use Free ebooks ==> www.ebook777.com 318 Activity Recognition in Pervasive Intelligent Environments either frame-based simple statistical features (such as the mean and variance) or Fourier descriptors Differences exist in the details of different parameterizations used (window lengths, selection of coefficients etc.) However, there is no clear consensus as to the most suitable statistical classifier 14.3.2 Instrumented Kitchens AR in the home has been explored in various guises, but only a small number of projects have considered the issues of technology-augmented kitchens One of the first projects was MIT’s CounterIntelligence – an augmented kitchen that provides instructive information to users while they are cooking [15] CounterIntelligence highlighted issues of situated interaction (rather than activity recognition) and included several different examples of how information could be displayed on kitchen surfaces to direct a user’s attention appropriately Chi and colleagues [16] developed an application of RFID technology embedded in a kitchen counter in which food ingredients on the kitchen counter was intended to raise a user’s awareness of the healthy quality of food ingredients through the presentation of nutritional information on a display and speaker (an extension of the “dietary aware table” [17]) A number of other design proposals also relate to the provision of situated advice on food and cooking [18] In the following, more general projects and initiatives related to instrumented homes (including kitchens) are described in more detail PlaceLab@House_n As part of MIT’s House_n research initiative on “how new tech- nologies, materials, and strategies for design can make possible dynamic, evolving places that respond to the complexities of life” [19], the PlaceLab represents a “living lab”, i.e., a laboratory environment to study how people interact with new technologies integrated into real-world living environments [20] Among others the PlaceLab also contains an instrumented kitchen environment PlaceLab utilizes a multitude of (ubiquitous) sensors that are embedded into the environment (furniture, walls, floors, etc.) For example, by means of simple state-change sensors, occupants’ activities of daily living (ADL) have been monitored in a simple and non-intrusive way [21] Additionally, activity recognition is also performed based on the analysis of body-worn sensors, which enables the analysis of ADL such as walking, running, cycling and so forth [22] Quality of Life Technology Centre QoLT is a joint research endeavor of the University of Pittsburgh and the Carnegie Mellon University with the goal to develop “technologies that will improve and sustain the quality of life for all people” [23] In addition to a multi- www.ebook777.com Free ebooks ==> www.ebook777.com Activity Recognition and Healthier Food Preparation 319 tude of general cross-disciplinary research activities that are related to the development of methods that enable older adults and people with disabilities to live independently, QoLT also addresses the analysis of real-world kitchen scenarios For example, Spriggs and colleagues developed an activity segmentation and classification system that analyzes food preparation activities (baking and cooking) in the kitchen [24] Computer vision techniques, as well as the analysis of inertial measurement units, are used as the basis for the recognition system The analysis of multi-modal sequential data is performed by means of statistical modeling techniques such as hidden Markov models Although the reliance on body-worn sensors raises some practical questions impressive recognition results have been reported Aware Kitchen The Aware Kitchen (also known as the Assistive Kitchen [13]) is an in- strumented kitchen environment that has been developed at TU Munich It comprises of a mobile robot and networked sensing and actuation devices that are physically embedded into the environment The main purpose of the Aware Kitchen is to provide “a comprehensive demonstration and challenge scenario for technical cognitive systems” [13], i.e., a testbed for computer vision, robotics, and sensor analysis techniques For the analysis of food preparation activities a number of sensors are integrated into kitchen utensils This includes, force-sensors in the handle of a knife, load cells and accelerometers in a chopping board, and body-worn sensors such as RFID-enabled gloves Sensor data have been analyzed in certain kitchen related exemplary applications (activity and context recognition) Ambient Kitchen The Ambient Kitchen is our own high fidelity prototype for exploring the design of pervasive computing algorithms and applications for everyday environments, which has been created at Newcastle University’s Culture Lab [25] The environment integrates data projectors, cameras, RFID tags and readers, object mounted accelerometers, and under-floor pressure sensing using a combination of wired and wireless networks The Ambient Kitchen is a lab-based replication of a real kitchen where careful design has hidden the additional technology, and allows both the evaluation of pervasive computing prototypes and the simultaneous capture of the multiple synchronized streams of sensor data Previous work exploring the requirements for situated support for people with cognitive impairments motivated the design of the physical and technical infrastructure The lab-based prototype has been put to use as: a design tool for designers; a design tool for users; an observatory to collect sensor data for activity recognition algorithm development, and an evaluation test bed The recognition system for the analysis of kitchen activities has been integrated into Free ebooks ==> www.ebook777.com 320 Activity Recognition in Pervasive Intelligent Environments the Ambient Kitchen project For real-time analysis it continuously runs as a background service simultaneously analyzing data from multiple, synchronized acceleration sensors Philips ExperienceLabs The ExperienceLabs is part of the research infrastructure at Philips Research (Eindhoven) and recreates a natural setting for the innovation of technologies and applications in real living environments [26] This also includes kitchens and all sorts of utensils and appliances used therein Crucially, the ExperienceLabs have been designed to be comfortable enough for people to stay in for some time (longer than in most classical usability studies) and are equipped with a sensor and computing infrastructure to observe and monitor the behavior of the participants in studies 14.4 Automatic Analysis of Food Preparation Processes Technology based support for kitchen activities that eventually shall lead to the consumption of healthier food aims to encourage manual food preparation and to aid in related kitchen tasks Such an assistive system would automatically keep track of the progress the cook makes while following a recipe It would explicitly guide the cook to the particular steps, which are necessary to actually prepare a dish Furthermore, it would give hints regarding potential pitfalls and how to circumvent them If, for example, some dish requires especially careful chopping of the ingredients it would monitor the performance of the cook while he is chopping and would give hints if, e.g., finer chopping is necessary During the cooking process an assistive system could also provide valuable additional information, such as background knowledge on the recipe, on the ingredients, and even related to the way the food is treated It could give recommendations for side dishes, (healthier) alternatives to certain ingredients or treatments of the food (e.g steaming vegetables instead of frying) Such situated support would substantially improve the cooking experience, and hence implicitly promote healthier eating The foundation for a hands-free assistive system for situated support for kitchen tasks is a means for automatically and unobtrusively monitoring the activities the cook is pursuing while he is working in the kitchen Hence, activity recognition represents the technological link between encouraging people to cook more and healthier eating 14.4.1 Activity Recognition in the Ambient Kitchen Typical activity recognition approaches in pervasive computing applications are based on the analysis of body-worn sensors such as bracelets with embedded accelerometers Al- www.ebook777.com Free ebooks ==> www.ebook777.com Activity Recognition and Healthier Food Preparation 321 ternatively, video based activity recognition using computer vision techniques can also be utilized Unfortunately, both approaches exhibit substantial drawbacks for real-world applications in the kitchen (either physically or in terms of privacy) Generally video cameras undermine the privacy of a home environment, and body-worn sensors are generally unwelcome encumbrances The system presented in this chapter provides a means for activity recognition without violating people’s privacy thereby performing highly reliably in real-world food preparation tasks (see also our previous work [6]) It is part of the Ambient Kitchen (see Section 14.3 and left hand image of Figure 14.1) We enhanced ordinary kitchen utensils including knives and spoons by embedding accelerometers within their handles Figure 14.1 (right side) illustrates the current state of the sensor-equipped utensils where modified Wii-Remote accelerometer boards are integrated into custom-made handles of the kitchen utensils We are also currently working on the integration of much smaller custom-made wireless accelerometers, which will be integrated into standard kitchen utensils (without changing the design of their handles) Fig 14.1 Ambient Kitchen (left) – the framework for activity recognition supported food preparation – and the sensor equipped utensils (right) See text for explanation The recognition system allows for continuous monitoring of sensory data in an integrative manner Technically this corresponds to mining low-cost sensor networks for activity recognition People preparing their food not have to worry about wearing the sensors, they just use the utensils as in their regular daily food preparation and cooking activities Through the use of different utensils significant insights about the activities can be gained and larger scale context analysis becomes possible In the following we provide a system overview together with the description of an experimental evaluation of activity recognition for food preparation tasks Free ebooks ==> www.ebook777.com 322 14.4.2 Activity Recognition in Pervasive Intelligent Environments System Description Analyzing food preparation for situated support requires fast and robust recognition of the actions as they are performed Consequently the described AR system processes sensor data in a strictly time-synchronous manner Raw acceleration data in form of x/y/z coordinates are captured continuously using the sensors integrated into the utensils Our configuration utilizes Wii-Remotes sampled at 40Hz with each sample having a precision of bits By using a sliding window procedure, frames of 64 samples length (50% overlap) are extracted and statistical features are computed for every frame Capturing approximately two seconds of sensor data per frame represents a reasonable compromise between sufficient amounts of context to analyze and low latency for real-time processing The features extracted comprise mean, standard deviation, energy, entropy and correlation They are computed for each frame for x-, y-, z-, pitch- and roll-acceleration resulting in 23-element vectors f ∈ R23 The back-end of the recognition framework is based on a descriptive modeling approach, where statistical models GA – namely Gaussian mixture density models with parameters ΘA = (μi , Ci , wi | i = 1, , KA ) – are extracted from training data to model the likelihood of feature vectors for every activity of interest A: KA p( f | ΘA ) = GA = ∑ wi N ( f | μi , Ci ) (14.1) i=1 where N denotes a Normal probability distribution with mean vector μ and covariance matrix C: N ( f | μ , C) = T −1 e− ( f −μ ) C ( f −μ ) | 2π C| (14.2) wi represent the prior probabilities for the i-th mixture components (out of KA Gaussians in total) In our previous work we extensively investigated different modeling techniques for robust and reliable activity recognition based on the analysis of accelerometer data [6] For a fully functional activity recognition system that can be successfully applied to the analysis of real-world kitchen tasks, it is essential to process all recorded data automatically This especially requires proper treatment of unknown activities, i.e., sensor data that have been recorded while none of the particular activities of interest were conducted Consequently, the AR system of the Ambient Kitchen also contains a rejection model that robustly covers unknown activities Training mixture models for known activities can be done in a straightforward manner By means of standard k-means clustering and maximum likelihood (ML) optimization, www.ebook777.com Free ebooks ==> www.ebook777.com Activity Recognition and Healthier Food Preparation 323 Gaussians, i.e., the parameters wi , μi , Ci , are estimated on class-specific training data Note that the number KA of Gaussians to be used for representing an activity A needs to be determined via cross-validation As a rule of thumb the classification performance is proportional to the number of Gaussians that can be robustly estimated The estimation of a proper rejection model (also referred to as background model) is slightly different The meaning of “unknown” is dependent on the particular task, i.e., on the set of activities that are of interest It is unrealistic to assume the availability of a specific training set that only contains “unknown” samples, i.e., that is disjoint from the known activities Thus, a straightforward modeling approach is impractical The applied pattern recognition literature extensively addresses procedures for mixture model estimation where the availability of sample data is complicated [27] The usual way of deriving the models is to start with the estimation of the background model on any – domain related – sample data that is available without considering potential annotations By means of this unsupervised estimation procedure relatively general Gaussians are derived In order to obtain mixture models for the activities of interest the background model is specifically adapted using model transformation techniques and activity specific sample data only Different adaptation techniques can be applied Examples of which are Maximum Likelihood Linear Regression (MLLR [28]), Maximum a-posteriori adaptation (MAP [29]), or standard expectation maximization (EM [30]) training In our experiments EM-training performed best During recognition all mixture models including the background model are evaluated in parallel and the one with the highest a-posteriori probability for some feature vector determines the classification result: A = arg max p(ΘA | f ) = A arg maxA p( f |ΘA )p(A) p( f ) ≈ arg max p( f |ΘA )p(A) A (14.3) Bayes’ rule is applied with prior class probabilities p(A) estimated on the training set, and neglecting sample priors p( f ) as they are irrelevant for the maximization For known activities the particular model’s score will be much higher compared to the rejection model, whereas for unknown data the more generic background model will produce higher scores The activity recognition system is trained subject independently exploiting training samples that are annotated at the level of the activities of interest Consequently, the system can be used as is without the need for user-specific training or adaptation Free ebooks ==> www.ebook777.com 324 14.4.3 Activity Recognition in Pervasive Intelligent Environments Experimental Evaluation The activity recognition system represents the basis for all higher-level applications within the Ambient Kitchen that address the analysis and support of food preparation activities In order to evaluate the capabilities of the system we conducted practical experiments where volunteers were asked to prepare meals within the instrumented kitchen environment using the sensor enhanced utensils Based on an observation of real world food preparation and language used in instructional cooking videos, ten activities were modeled by Gaussian mixture models: chopping, peeling, slicing, dicing, scraping, shaving, scooping, stirring, coring, and spreading (cf figure 14.2 for an illustration of two typical activities) An additional rejection model has been estimated as described above Fig 14.2 Examples of typical food preparation activities performed using the sensor-enhanced utensils: peeling (left) and chopping (right) For the experimental evaluation twenty participants prepared sandwiches or a mixed salad using a range of ingredients with which they were provided with No further instructions were given, so the task was conducted in a relatively unconstrained manner, resulting in substantial variance in the time taken to complete the task Sessions with lengths varying from to 16 minutes were recorded Videos of sessions showed that all subjects performed a significant number of chopping, scooping, and peeling actions Only small subsets of subjects performed eating (i.e using the knife or spoon to eat ingredients – considered as unknown), scraping (rather than peeling) or dicing (i.e fine grained rapid chopping) actions The video recordings were independently annotated by three annotators Every annotator was provided with an informal description of the ten activities of interest, and was asked to independently code the actions for each subject’s recording After annotation, the www.ebook777.com Free ebooks ==> www.ebook777.com Activity Recognition and Healthier Food Preparation 325 intersection of the three coded data sets was created; this served as the ground truth annotation Here only labeled data for which all three annotators agreed was extracted, that is, data where there is complete agreement between the annotators as to the action being performed It should be noted that the boundaries between the ten activity labels is often unclear In total almost four hours of sensor data were recorded covering approximately two hours of the ten modeled activities Classification experiments were performed as 10-fold crossvalidation and results are averaged accordingly The classification results of the experimental evaluation are summarized in Table 14.1 Processing the whole dataset, i.e., including samples that not belong to one of the known activities (open set), 90.7% accuracy is achieved with a level of statistical significance of ±0.7% Since almost 50% of the data covered unknown/idle activities, an additional evaluation of only the known activities of interest has been performed For this closed dataset the accuracy was 86% Table 14.1 Classification results for food preparation experiments dataset open set (incl unknown act.) closed set (w/o unknown act.) accuracy (stat significance) 90.7% (± 0.7%) 86% (± 2.4%) More detailed insights into the recognition performance of the system can be gained by means of a confusion analysis Tables 14.2 and 14.3 represent the confusion matrices for the experimental evaluation based on the open and the closed datasets (each for tenfold cross validation experiments) The rejection model works very reasonably since only very few known activities are erroneously classified as being unknown (422 false positives compared to 44173 unknown frames in total, which is below 1% – cf the first row and the first column of Table 14.2) The rate of false negative predictions, i.e., of erroneous rejections of known activities, was also reasonably low at approximately 8% Slicing and dicing activities were quite frequently confused with chopping activities For example, for the closed set approximately 46% of the dicing activities are classified as being chopping Analyzing the video footage of the food preparation experiments it becomes clear that the majority of these failures can be explained by erroneous annotations (across the three annotators), i.e., ground truth errors Free ebooks ==> www.ebook777.com 326 Activity Recognition in Pervasive Intelligent Environments chopping scraping peeling slicing dicing coring spreading stirring scooping shaving Confusion matrix (# frames) for open set experiments (10-fold cross eval.) unknown Table 14.2 40672 79 57 43 33 37 95 61 12 1398 2327 91 161 30 61 49 30 71 0 19 0 283 474 0 0 331 263 283 41 0 35 85 0 200 1 0 38 2 100 21 453 34 2 19 171 19 0 0 19 209 25 482 10 41 851 432 29 23 0 29 493 unknown chopping scraping peeling slicing dicing coring spreading stirring scooping shaving 14.5 chopping scraping peeling slicing dicing coring spreading stirring scooping shaving Table 14.3 Confusion matrix (# frames) for closed set experiments (10-fold cross eval.) 2390 92 183 30 77 51 71 0 19 0 498 0 0 263 283 41 0 90 0 207 1 2 100 24 34 2 19 171 0 0 20 238 29 15 58 20 12 17 104 904 29 23 34 500 chopping scraping peeling slicing dicing coring spreading stirring scooping shaving Activity recognition and the promotion of health and wellbeing Our approach to activity recognition in the kitchen appears to have the potential to promote healthier eating As we described in the introduction to this chapter, food preparation and cooking skills have been identified as a barrier to healthier food choices, and the design space for imaginative technological interventions is wide open, both to support people’s actual food preparation and to assess and enhance their skills Our own activity recogni- www.ebook777.com Free ebooks ==> www.ebook777.com Activity Recognition and Healthier Food Preparation 327 tion framework allows for real-time analysis of kitchen activities using statistical modeling techniques Although we have demonstrated that our system can make sense of the actions of naive users, there is still a set of significant challenges before such AR capabilities can be integrated into support systems in regular households In particular, the problem of recognizing the food items themselves Though RFID has long been proposed as a technology that might afford embedded sensing of ingredients (in their packaging), the benefits for the retailers of individually labeling packages (rather than cartons or pallets) are not clear RFID would also require either significant infrastructure in the kitchen (embedded antennae) or wrist worn readers as have been developed in a number of ubiquitous computing initiatives Alternatively, kitchens have always sites for specialized appliances for food preparation, such as manual and mechanized tools The practicalities of kitchenware retailing also means that if the sensing required can be achieved through an enhancement to an existing appliance, then take-up is more likely Beetz et al.’s knife illustrates the potential clearly [13] By embedding force sensors in the knife, as well as accelerometers, the knife could be used to sense the ingredients on which it was used Our own vision recognizes the potential of such embedded sensors, and we see significant opportunities for further augmentation of the tools and utensils themselves with well chosen (and unobtrusive) sensors However, activity recognition and sensor design and deployment are just a subset of the elements of the solution required A mature solution will need to aggregate the atomic actions in reasoning about the preparation of a meal, address the problem of parallel actions, reason about the skill level of people cooking, and even the impact of their actions on the nutritional value of the ingredients Furthermore, some practical problems need to be solved, which mostly relates to handling issues for the embedded sensory The next version of sensor equipped utensils, which we are currently developing, will be based on much smaller, water- and heat-proof hardware Integrating this tailored hardware into kitchen utensils, which can be used and washed as usual, will pave the way for real-life application in private households References [1] National Institute for Health and Clinical Excellence Obesity: the prevention, identification, assessment and management of overweight and obesity in adults and children (May, 2006) [2] B A Swinburn Increased energy intake alone virtually explains all the increase in body weight in the united states from 1970s to the 2000s In Proc European Congress on Obesity, (2009) [3] G Block, T Block, P Wakimoto, and C H Block, Demonstration of an e-mailed worksite Free ebooks ==> www.ebook777.com 328 Activity Recognition in Pervasive Intelligent Environments nutrition intervention program, Preventing Chronic Disease 1(4) (Jan, 2004) [4] S Jebb, T Steer, and C Holmes The ‘healthy living’ social marketing initiative: A review of the evidence (Mar, 2007) [5] J Maitland, M Chalmers, and K A Siek Persuasion not required – improving our understanding of the sociotechnical context of dietary behavioural change In Proc Int Conf Pervasive Computing Technologies for Health Care (Feb, 2009) [6] C Pham and P Olivier Slice&dice: Recognizing food preparation activities using embedded accelerometers In Proc Europ Conf Ambient Intelligence, pp 34–43, (2009) [7] C Byrd-Bredbenner, Food preparation knowledge and attitudes of young adults: Implications for nutrition practice, Topics in Clinical Nutrition 19, 154–163, (2004) [8] World Health Organization Diet, nutrition and the prevention of chronic diseases WHO Technical Report Series, number 916, (2003) [9] E Winkler and G Turrell, Confidence to cook vegetables and the buying habits of australian households, Journal of the American Dietetic Association 109(10), 1759–1768 (Oct., 2009) [10] N I Larson, M Story, M E Eisenbergy, and D Neumark-Sztainer, Food preparation and purchasing roles among adolescents: associations with sociodemographic characteristics and diet quality, Journal of the American Dietetic Association 106, 211–218, (2006) [11] N Sudo, D Degeneffe, H Vue, E Merkle, J Kinsey, K Ghosh, and M Reicks, Relationship between attitudes and indicators of obesity for midlife women, Health Educ Behav 36(6), 1082–1094, (2009) [12] L Atallah and G Yang, The use of pervasive sensing for behaviour profiling – a survey, Pervasive and Mobile Computing pp 447–464, (2009) [13] M Beetz, J Bandouch, A Kirsch, A Maldonado, and R B Rusu The assistive kitchen—a demonstration scenario for cognitive technical systems In IEEE 17th Int Symp Robot and Human Interactive Communication (RO-MAN), pp 1—8 (Jan, 2008) [14] T Huynh and B Schiele Analyzing features for activity recognition In Proc Joint Conf on Smart Objects and AmI, pp 159–163, (2005) [15] L Bonanni, C Lee, and T Selker CounterIntelligence: Augmented Reality Kitchen In Proc CHI, pp 2239–2245, (2005) [16] P Chi, J.-H Chen, H.-H Chu, and B.-Y Chen Enabling nutrition-aware cooking in a smart kitchen In Proc CHI – Extended Abstracts on Human Factors in Computing Systems, pp 2333–Ð2338, (2007) [17] K Chang, S.-Y Liu, H.-H Chu, J Hsu, C Chen, T.-Y Lin, and P Huang Dietary-aware dining table: Observing dietary behaviors over tabletop surface In Proc Int Conf Pervasive Computing, pp 366–Ð382, (2006) [18] Q T Tran, G Calcaterra, and E D Mynatt Cooks collage: Déjà vu display for a home kitchen In Proc Int Conf Home-Oriented Informatics and Telematics (HOIT), pp 15–Ð32, (2005) [19] House_n http://architecture.mit.edu/house_n/ – visited 9th April 2010 [20] S S Intille, K Larson, E Mungia-Tapia, J S Beaudin, P Kaushik, J Nawyn, and R Rockinson Using a live-in laboratory for ubiquitous computing research In Proc Int Conf Pervasive Computing, pp 349–365 (Dec, 2006) [21] E Mungia-Tapia, S S Intille, and K Larson Activity recognition in the home setting using simple and ubiquitous sensors In Proc Int Conf Pervasive Computing, (2004) [22] L Bao and S S Intille Activity recognition from user-annotated acceleration data In Proc Int Conf Pervasive Computing, (2004) [23] Quality of Life Technology Center – QoLT http://www.cmu.edu/qolt/ – visited 9th April 2010 [24] E H Spriggs, F D L Torre, and M Hebert Temporal segmentation and activity classification from first-person sensing In IEEE Workshop on Egocentric Vision, CVPR 2009 (June, 2009) [25] P Olivier, A Monk, G Xu, and J Hoey Ambient kitchen: Designing situated services using a www.ebook777.com Free ebooks ==> www.ebook777.com Activity Recognition and Healthier Food Preparation [26] [27] [28] [29] [30] 329 high fidelity prototyping environment In Workshop on Affect & Behaviour Related Assistance in the Support of the Elderly, PETRA-09, (2009) H Hoonhout ExperienceLabs: investigating peopleÕs experiences in realistic lab settings In Proc Int Conf Designing Pleasurable Products and Interfaces (DPPI), (2007) T Plötz and G A Fink, Pattern recognition methods for advanced stochastic protein sequence analysis using HMMs, Pattern Recognition, Special Issue on Bioinformatics 39, 2267–2280, (2006) C J Leggetter and P C Woodland, Maximum likelihood linear regression for speaker adaptation of continuous density Hidden Markov Models, Computer Speech & Language pp 171– 185, (1995) J.-L Gauvain and C.-H Lee Map estimation of continuous density HMM: Theory and applications In Proc DARPA Speech and Natural Language Workshop, (1992) A Dempster, L N.M., and D Rubin, Maximum likelihood from incomplete data via the em algorithm, Journal of the Royal Statistical Society 39, 1–38, (1977) Series B (methodological) ... activities of daily living v Free ebooks ==> www.ebook777.com vi Activity Recognition in Pervasive Intelligent Environments The text is intended for those working within the area of intelligent environments... quality titles in the fields of Pervasive Computing, Mixed Reality, Wearable Computing, LocationAware Computing, Ambient Interfaces, Tangible Interfaces, Smart Environments, Intelligent Interfaces,... networks and pervasive computing infrastructures become technically mature and financially affordable It has been, in particular, under vigorous investigation in the creation of intelligent pervasive