Safer Surgery 124 Thus, the playback software allows the observer to watch all four video feeds simultaneously and to jump all the videos instantly to a different point based on any marked event (that was marked by the observer watching the case using the RATE event-marking software) (see Figure 8.2). The RATE Event-Marking Software Figure 8.2 shows a screen shot of the RATE event-marking software just after a case has started. The left half of the screen is for conversation tracking and the right half of the screen is for event tracking. In particular, the upper portion of the left half of the screen allows observers to manually track conversations between members of the team by marking who is talking to whom, using what type of communication (e.g., joking, requesting, coaching) and about what topic (e.g., the patient, the room, the surgical tools, other cases). The observer clicks four times in the four columns, or can use ‘type-ahead’ and the ‘Tab’ key to mark a communication event. The summary is placed in the box to the right of these pick lists and is double-clicked by the observer when the communication event is nished. One can also single click on one of these conversations and click ‘Answer Back’ (below the ‘Talk’ button) and this will cause the ‘From’ and ‘To’ columns to automatically switch (e.g., ‘Surgery Attending (SA) Other Discussing Other Cases’ automatically gets selected in the four boxes as ‘Other SA Reply Other Cases’) and then the observer Figure 8.1 The RATE software RATE 125 Figure 8.2 The RATE event-marking software Safer Surgery 126 hits the ‘Talk’ button or edits one of these entries rst if desired. The software user interface enables keeping up with conversations fairly well so long as the observer does not spend too much time typing in free-text comments (possible using the text entry box on the bottom left of the screen). Based on these conversation traces, RATE tracks communication counts of each individual and each pair of individuals, by communication type and by categorized communication content (as judged by the observer when tracking the conversations live during the case). This enables summary analyses such as determining the (surprisingly large) percentage of time spent directing a medical student on how to operate the camera effectively during a case. In the bottom half of the left side of the screen, we put in a few entries that we specically wanted to track regarding communication during the case, e.g., we checked off these boxes if the following aspects were specically discussed by the surgeon during a pre-operative brieng prior to rst skin incision: introductions of people by name and their roles during the case, conrmation of antibiotics dosed, patient history, whether a cholangiogram procedure was planned, the likelihood the case might convert to an open procedure, any planned turnovers of personnel during the case, and conrming that all equipment and appliances were set up and ready for case start. These events could likely have been inferred from our more generic conversation tracker above, but they were important enough for our study to warrant a separate checkbox just for those events. We also placed here the ability to give an ‘overall’ communication score to the team at the end of the case, but found it was impossible to judge this and we thus quickly gave up using this feature. The right half of the screen enables marking one time events that should/could occur for every case (left column) and things that could happen more than once (right column). The former are very useful for jumping to a particular part of the surgery later, and the latter are counts of errors with the added ability to jump the case to just prior to any of these upon review. Determine Coding Scheme between Observers for Communication In general, the software is designed to be customizable. All elds and events are editable, either directly through the Access back end, or by using the ‘Edit’ buttons that are placed around the screen. The observers need to dene the codes and rules to follow when encoding data. The three communication categories that we used are team members, communication type (event), and the content as shown in Figure 8.2 (with an option to type in a free-text keyword, comment or summary of what was said). The team members and communication content are domain specic and can be changed depending on the process being observed, but we hypothesize (but have not tested) that the communication types, such as coaching, empowering and greeting, are generic. RATE 127 Conclusion This chapter described a hardware/software system called RATE that was developed to integrate data collection, processing and analysis for monitoring and training team performance. RATE allows observers to code real-time events with time codes, align and combine multiple codes, and compare for inter-rater agreement. Because RATE is a multimedia system, the time-stamps can be used to replay back the video segments of marked events immediately. A complete user’s manual with screen shots of RATE, along with the executable software, is free to download for non-prot use from <http://www.sys.virginia.edu/hci/hcilab.asp>. References Endsley, M. (1995) Measurement of situation awareness in dynamic systems. Human Factors 37(1), 65–84. Flin, R. and Maran, N. (2004) Identifying and training non-technical skills for teams in acute medicine. Quality and Safety in Healthcare 13, i80–i84. Gaba, D. (1989) Human error in anesthetic mishaps. International Anesthesiology Clinics 27(3), 137–47. Guerlain, S., Turrentine, B., Calland, J.F., and Adams R. (2004) Using video data for analysis and training of medical personnel. Cognition Technology and Work 6, 131–8. Guerlain S., Adams R.B., Turrentine F.B., Shin T., Guo H., Collins S.R. and Calland J.F. (2005) Assessing team performance in the operating room: Development and use of a ‘black-box’ recorder and other tools for the intraoperative environment. Journal of the American College of Surgeons 200(1), 29–37. Guerlain, S. Turrentine, F. Collins, S., Calland, J.F. and Adams, R. (2008) Crew resource management for surgeons: Feasibility and impact. Cognition, Technology and Work 10(4), 255–64. Helmreich, R. and Schaefer, H. (1994) Team performance in the operating room. In M. Bogner (ed.), Human Error in Medicine (pp. 225–53). Hillsdale, NJ: Lawrence Erlbaum. Law, J. and Sherman, P. (1995) Do raters agree? Assessing inter-rater agreement in the evaluation of air crew resource management skills. In R. Jensen (ed.), Proceedings of the 8th Symposium of Aviation Psychology 608–12. Moorthy, K., Munz, Y., Adams, S., Pandey, V. and Darzi, A. (2005) A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Annals of Surgery 242(5), 631–9. Rouse, W., Cannon-Bowers, J. and Salas, E. (1992) The role of mental models in team performance in complex system. Institute of Electrical and Electronics Engineers 22(6), 1296–308. Safer Surgery 128 Sexton, J. and Helmreich, R. (2000) Analyzing cockpit communication: The links between language, performance, error, and workload. Human Performance in Extreme Environments 5(1), 63–8. Sexton, J., Marsch, S., Helmreich, R., Betzendoerfer, D., Kochre, T. and Scheidegger, D. (2002) Jumpseating in the operating room. In L. Henson, A. Lee and A. Basford (eds), Simulators in Anesthesiology Education (pp. 107– 108). New York: Plenum. Shin, T. (2003) Team Performance Measurement: Measuring Inter-Rater Reliability/Agreement of Independent Team Scores. University of Virginia Masters thesis. Simon, H., and Ericsson, K. (1993) Protocol Analysis: Verbal Reports as Data. Cambridge, Mass: Massachusetts Institute of Technology Press. Sledd, R. Bass, E. Borowitz, S. and Waggoner-Fountain, L. (2006) Supporting the characterization of sign-out in acute care wards. In Proceedings of the 2006 IEEE Conference on Systems, Man, and Cybernetics, 5215–20. Xiao, Y., Mackenzie, C., Seagull, F., and Jaberi, M. (2000) Managing the monitors. An analysis of alarm silencing activities during an anesthetic procedure. Proceedings of International Ergonomics Association 2000 Human Factors and Ergonomics Society 2000 Congress 4, 250–3. Xiao, Y. and The LOTAS Group (2001) Understanding coordination in a dynamic medical environment. Methods and results. In M. McNeese, E. Salas and M. Endsley (eds), New Trends in Cooperative Activities (pp. 242–58). Santa Monica, CA: Human Factors and Ergonomics Society. Chapter 9 A-TEAM: Targets for Training, Feedback and Assessment of all OR Members’ Teamwork Carl-Johan Wallin, Leif Hedman, Lisbet Meurling and Li Felländer-Tsai Background The objective for our research is to increase our knowledge about the teamwork training process that helps account for improving teamwork outcome and patient safety. In this chapter we will focus on the development of a new instrument to assist team performance assessment in a variety of real and simulated clinical events in the operating theatre. This tool could be used in the study of the relationship between the teamwork process and teamwork outcome, as well as for feedback during training. Laborious work has been done earlier in this eld and we want to further explore whether renements of existing behaviour scales will add to our understanding of teamwork in general and to understanding of the team training process in particular (for reviews see Baker et al. 2005b, Rosen et al. 2008). The process of providing healthcare is inherently interdisciplinary, requiring physicians, nurses and allied health professionals from different specialties to work in teams. Compared to teams in other industries, medical teams work under conditions that change frequently, may be assembled ad hoc, have a dynamically changing team membership, often work together for a short period of time, consist of different specialists and have to integrate different professional cultures (Manser 2009). In order to effectively train teamwork, it is necessary to reliably assess behaviours associated with effective teamwork and their interplay in relation to clinical performance ratings and ultimately to patient outcome (Baker et al. 2005a, Rosen et al. 2008). There are three parts in this chapter: (i) rationale for developing the all team members’ behaviour scale (A-TEAM), (ii) presentation of A-TEAM, and (iii) application of A-TEAM in an operating theatre setting. Safer Surgery 130 Rationale for Developing the A-TEAM Scale A Generic Behavioural Scale for Teamwork in Healthcare In the operating theatre, failures in communication and ineffective teamwork during surgical procedures have observable consequences such as delay, tension among team members, or procedural errors (Catchpole et al. 2007, Lingard et al. 2004a, Wiegmann et al. 2007). Although it is often claimed that many adverse events could have been prevented by improved teamwork, most studies do not make explicit exactly which aspects of teamwork that have to be improved (Baker et al. 2005b, Manser 2009). To exactly identify the demands in each particular setting in the operating theatre is a prerequisite for efcient training, although research in this eld has just started (Rosen et al. 2008). However, when looking at the coordination problems that occur in healthcare, it is obvious that generic well-known coordination behaviours are not always applied and providers have not received formalized training in how to interact with one another. Therefore health care authorities recommend implementation of team training to improve teamwork (Agency for Healthcare Research and Quality 2006, Burke et al. 2006, Kohn et al. 2000, Veterans Affairs National Center for Patient Safety 2004). Murray and Foster (2000) recommended bringing together individuals from different healthcare professions and specialties to take part as ‘strangers‘ in team training using generic team behaviours as targets for training. Ostergaard et al. (2004) have also argued for multidisciplinary team training in medicine aiming at improving team skills in order to increase patient safety. Frankel and collaborators (2007) argued for teamwork skills that are widely applicable and reect good practice for a wide variety of healthcare professions. Applying generic coordination principles in team training of the operating theatre staff would be an advantage also for the safety of the surgical patient. For this purpose, a generic behaviour scale as target for team training in medicine is needed. This should also be applicable in the operating theatre. Rening the training programme in future by applying specic behaviours revealed by research would add to this advantage. A Scale Feasible to use as a Target for Training, Formative Assessment and Feedback Applying cognitive behaviour principles (for review see McGuire 2000) in industrial and organizational training settings (Cannon-Bowers and Salas 1997, Salas et al. 1999) aligns target for training with formative assessment for feedback, ‘assessment is the tail that wags the dog’ as someone said. In order not to tax trainees’ limited working memory (Baddley and Logie 1999, Cowan 2005, Miyake et al. 2001) during training, targets for training should be limited. Social psychology has made considerable progress over the last few decades regarding the development of observational methods. Despite this headway, there still remains a number of methodological issues about which little can be said. One such issue is how to A-TEAM 131 consider ndings of human working memory when developing observational methods and protocols. Tools for observation and rating of behaviour should be very easy to understand and use, since observational methods are vulnerable to limitations of human perceivers. Weick (1968, p. 433) also advocated the use of: simple, unequivocal behavioural indices [i.e., checklists] … [which] are easy to grasp [i.e., easy to train to a high degree of interjudge agreement], and do not impose excessive demands on the observer. Working memory, previously termed short-term memory, is a theoretical construct studied within cognitive psychology which has been applied to image guided surgical simulation (Hedman et al. 2007). It refers to the mental structures and processes used for temporarily storing and elaborating information. Working memory is generally considered to have limited capacity. We cannot focus on everything at once. We quickly become overburdened. In his classical article in the history of psychology, Miller (1956) was the rst cognitive psychologist who introduced the ‘magical number seven’ as a quantication of the capacity limit associated with short-term memory. The memory span of young adults was around seven elements (chunks), regardless whether the elements were letters, words, digits or other units. However, more recent research has shown that many different factors affect a person’s measured span. It is therefore generally difcult to state that the capacity of short-term or working memory is an absolute number of chunks. Nonetheless, Cowan (2005) suggested that working memory may have a capacity of about four chunks in young adults, and fewer in children and old adults. Numerous theories of working memory exist. For example, according to Baddeley and Logie (1999), working memory is the information we are actively thinking about and processing at any given moment, and working memory is consequently closely related to the allocation of our attention to the most important events in a situation. Baddeley and Hitch (1974) introduced their multi-component model of working memory, which proposes that two slave systems are responsible for short-term maintenance of information, and a central executive is responsible for coordinating the slave systems as well as for the supervision of information integration. For Cowan (2005) working memory is not as a separate theoretical system, but a part of long-term memory. The hypothesis that working memory is crucial for reducing distraction by maintaining the prioritization of relevant information was tested by de Fockert and collaborators (2001) by functional magnetic resonance imaging (fMRI) and psychological experiments in humans. The researchers demonstrated that when a person’s working memory is occupied his or her brain cannot lter out distracting sights in a separate attention task. Hence, working memory is crucial for reducing distraction by maintaining the prioritization of relevant information. It therefore became very important for us to select as few behavioural categories and descriptions of poor to procient Safer Surgery 132 behaviours as possible for a behaviour observation tool, in order not to tax the observer’s limited working memory. Taking into consideration ndings on working memory, we designed the A-TEAM scale with only a few and distinct categories of behaviour to be identied. The scale should be exceedingly simple since the task for the observers is to simultaneously observe the behaviours of both leaders and followers. Moreover, an easy way to align targets and assessment is to use the same scale for both purposes. For training teamwork in medicine, we need a scale that has a limited number of behaviour items, easily understood by trainees and feasible to use for formative feedback. In order to evaluate the training per se the scale must also be feasible to use for summative assessment, i.e., provide pseudo-quantitative data to measure progress in teamwork after training. Most of the available behaviour rating instruments for healthcare settings, such as ACRM (Anesthesia Crisis Resource Management; Howard et al. 1992), revised teamwork behaviour matrix (Small et al. 1999), Team Dimensions Rating Form (Morey et al. 2002), OTAS (Observational Teamwork Assessment for Surgery; Carthey et al. 2003, Healey et al. 2004), ANTS (Anaesthetists’ Non- Technical Skills; Fletcher et al. 2003, Flin et al. 2004), EMCRM (Emergency Medicine Crisis Resource Management Behavioural Performance Evaluation; Reznek et al. 2003, Wallin et al. 2007), BARS (Behaviourally Anchored Rating Scales, Shapiro et al. 2004), NOTSS (Surgeons’ Non-Technical skills in the operating theatre; Flin et al. 2006, Yule et al. 2008), Ottawa GRS (Ottawa Crisis Resource Management Global Rating Scale, Kim et al. 2006), CATS (Communication and Teamwork Skills; Frankel et al. 2007), are designed for summative use, and as such too overburdened with elements to be used effectively in training. A Behaviour Scale for Recognition of all Individuals in the Team The core feature of teamwork is coordination of task execution to achieve a specied and shared goal. Timely and correct execution of tasks is coordinated through collaborative information sampling to build a shared mental model of the situation, collaborative decision-making, prioritizing and delegating tasks. The process can be described with help of a straightforward decision-making strategy as presented by St. Pierre et al. for individuals (2008, p. 122) applied to a team (see Figure 9.1). They emphasize that all strategies and decisional aids in the literature of decision-making in high stake environments (e.g., Gaba 1992, Murray and Foster 2000) contain at least the following ve steps of a good strategy: preparedness; analysis of the situation (gathering of information, building mental models); planning of actions (formulating of goals, risk assessment, planning, decision-making); execution of action; review of effects (review of actions, revision of strategy, self-reection). 1. 2. 3. 4. 5. A-TEAM 133 Team members are delegated different tasks while working together; task functions are not interchangeable, although change of the distribution of tasks between members may vary over time. In order to collaborate, a team member must adjust his/her behaviour to other members’, quoting Brannick and Prince (1997, p. 10): The interpersonal part of the process can thus be thought of as providing the grease that keeps the part of the team working together smoothly. On the other hand, difculties in individual member performance can act like sand in creating friction within a team, thus interfering with interpersonal harmony. Leading and following are signicant collaborative adjustment behaviours; an evolutionary strategy for solving social coordination problems (Van Vugt et al. 2008). All individuals in healthcare are involved in teamwork, most often in the follower position; some individuals also regularly take a leader position. No single individual has the privilege to always be the leader, more likely he/she will work together with others in a team where somebody else holds the leader position. The situation during work may also demand a quick change in roles between a leader and one of the followers in the team. All healthcare personnel must thus be able to function as both a follower and a leader in a team and to quickly change position without prestige. The relationship is symbiotic; no one can lead without followers. Leader and follower are distinct entities but of equal value to team collaboration and both are partners in the team. Both leader and follower need to support the mutual role assignment to support the integrity of team structure and process. Some Figure 9.1 A schematic presentation of a structured team decision-making process . happen more than once (right column). The former are very useful for jumping to a particular part of the surgery later, and the latter are counts of errors with the added ability to jump the. the observer Figure 8.1 The RATE software RATE 125 Figure 8.2 The RATE event-marking software Safer Surgery 126 hits the ‘Talk’ button or edits one of these entries rst if desired. The software. Safer Surgery 124 Thus, the playback software allows the observer to watch all four video feeds simultaneously