Safer Surgery 94 Exemplar behaviours and demonstrative scenarios for each sub-team/stage of a procedure are fully described in the OTAS user manual (Undre and Healey 2006, freely available for research use at: <http://www.csru.org.uk>). Further Empirical Testing: Urological Cases (Undre et al. 2007a) This study aimed to further assess: feasibility of the revised OTAS © tool; usefulness of the revisions; reliability in the behavioural scoring. The study also aimed to compare general surgery with urology elective procedures. As in the previous study, care was taken to inform staff about the study and to reassure them that data would be used for research purposes only. Methods Data were collected in 50 urological surgery operations in two operating theatres, one in our own institution (central London teaching hospital) and the other at a treatment centre. Twenty operations were the rst operation of the list; the remaining 30 operations were the second or subsequent operation. The typical mix of operations contained cystoscopy, ureteroscopy, ureterorenoscopy, transurethral resection of the prostate (TURP) and short procedures such as orchidectomy, vasectomy and circumcisions. Data were collected from procedures that lasted 30–240 minutes. Tasks and behaviours were assessed from Pre-op Stage 1 to Post-op Stage 2. The last OTAS © stage was not feasible to assess. In six additional procedures, behavioural ratings only were collected by two psychologist observers to assess inter-observer reliability. Results and Comments Task completion Table 6.2 (urology columns) presents the task completion rates. Overall, task completion was higher in urology than in general surgery. The pattern of task completion rates between different types of tasks was strikingly similar, with patient tasks showing highest completion rates, followed by equipment/provisions and communication tasks. In addition, some variability was observed in urology theatres too, with signicantly lower levels of equipment tasks in the Pre-op Phase than in the other two phases and signicantly higher levels of communication tasks in the Pre- and Post-op Phases than in the Intra- operative Phase. • • • Observing and Assessing Surgical Teams 95 Behaviour ratings As in the previous study, these were relatively high (scores above four on a seven point scale). Of signicance: Anaesthetists’ and nurses’ ratings were highest on cooperation and lowest on communication, with no signicant different across operative phases. Surgeons’ ratings exhibited a similar pattern, but, in addition, their scores were signicantly lower in the post-operative phase. The Pearson r correlation coefcients between the two psychologists’ ratings were as follows: – communication: 0.35, p < 0.05; – coordination: 0.72, p < 0.001; – cooperation/back up behaviour: 0.64, p < 0.001; – leadership: 0.62, p <0.001; – monitoring/awareness: 0.53, p < 0.001. In conclusion, team assessment appeared feasible in urology theatres. Importantly, the revised OTAS © application was successful. The revised tool replicated some of the ndings of the initial version in both task completion and behaviour ratings. The acceptable reliability of the behavioural scoring suggests that the addition of exemplar behaviours and demonstrative scenarios did assist the behavioural assessment, as intended. Current Work and Future Directions Comprehensive and robust assessment of teamwork in the context of surgery is becoming increasingly important. It has been shown that poor teamworking in surgical teams is associated with the occurrence of adverse events to patients (e.g., Davenport et al. 2007, Gawande et al. 2003, Greenberg et al. 2007, JCAHO 2000). Recent increases in shift-working in the delivery of surgical services mean that operating theatre staff are now much less likely than in the past to be working in stable teams, in which individuals know each other as well as their strengths and weaknesses (Royal College of Surgeons of England 2007). Such changes have been followed by an increased emphasis on teamwork skills in the modern surgical training curriculum (ISCP 2005). Taken together, such developments are likely to increase the importance of robust teamwork assessment. The aim of this chapter was to present in detail the development and initial application of the Observational Teamwork Assessment for Surgery © (OTAS © ). The origins of OTAS © can be traced in the empirical work on teamwork in complex work environments that started more than 40 years ago with a focus on military teams. Conceptually, OTAS © is grounded on (Dickinson and McIntyre 1997) a model of teamwork. Empirically, it follows attempts to assess teamwork in expert teams that work in complex environments via observation. These attempts include • • • Safer Surgery 96 the work on development of behavioural markers systems for assessing cockpit crews, as well as early work related to healthcare and surgery with a focus on teamwork or communication, threats and errors, or error recovery strategies in operating theatres. Building on this work, our research team constructed the initial version of the OTAS © , which consisted of a task checklist (to be completed by a surgeon observer) and ve behavioural scales (to be scored by a psychologist observer). Empirical testing of this rst version suggested that team observations and assessment is indeed feasible and also led to modications, aiming to assist the behavioural scoring – thus enhancing reliability and also allowing new observers to be trained. Subsequent testing supported empirically the feasibility, applicability and reliability of OTAS © . At present, we are conducting more developmental as well as validation work using OTAS © . Regarding the validation process, three aspects are currently under investigation. First of all, we are in the process of assessing effects of observer’s expertise in the behavioural scoring of OTAS © . The hypothesis is that ratings obtained from expert OTAS © users should exhibit higher correlations than those obtained by expert and novice users. Initial empirical evidence is consistent with this hypothesis. Secondly, we are in the process of using OTAS © to analyse teamwork in simulated crises-ridden procedures (Undre et al. 2007a). Procedures have been carried out, they have been video/audio-recorded and analysed using OTAS © . Initial evidence from the ratings of these procedures suggests that OTAS © can be used to rate teams retrospectively, in addition to its original real-time usage. Thirdly, we are using OTAS © alongside other observational tools that we have developed and that assess aspects of surgical process other than teamwork – including interruptions and disruptions to surgical workow. Meaningful correlations between aspects of teamwork as captured by OTAS © and other surgical processes will contribute to the cross-validation of all tools involved. Regarding the developmental work that is being carried out, we are in the process of producing a version of OTAS © to be used in simulation-based team training. Current work carried out by members of our group suggests that simulation-based training with formative feedback/debrieng has potential application to training surgeons how to cope most effectively with stressors that occur during procedures (Arora et al. 2009). Such stressors include technical difculties (e.g., unexpected bleeding), but also lack/failure of equipment, unnecessary distractions and uncooperative team members. This work builds on previous team training modules addressed to the entire operating theatre team that we have piloted successfully (Moorthy et al. 2005, 2006, Undre et al. 2007a). Successful application of OTAS © in a training context will expand the domain of application of the tool and will contribute to its further validation. OTAS © is one of the rst assessment tools to be designed exclusively for surgical teams. In addition to OTAS © , we have also developed and tested a version of the NOTECHS (Avermate van 1998, Flin et al. 2003) to be used in surgical teams (Sevdalis et al. 2008a). In the past ve years, other research groups working in Observing and Assessing Surgical Teams 97 parallel have developed tools that also aim to capture teamwork skills – including the Anaesthetists’ Non Technical Skill ((ANTS) Fletcher et al. 2003), the Non Technical Skills for Surgeons ((NOTSS) Yule et al. 2006b, 2008) and the Mayo High Performance Teamwork Scale (MHPTS) (Malec et al. 2007 – see Chapters 2, 11 and 12 in this volume). The proliferation of tools with a focus on operating theatre teams signals an increasing recognition of the importance for teamwork in surgery. The uniqueness of OTAS © lies in that it captures the entire team, instead of individual members of it (i.e., the NOTECHS-based approach). Thus OTAS © offers a holistic, non-threatening assessment that can be used to assess teamwork via observation (as opposed to self-report) in any operating theatre. Importantly, it can also be used to provide formative feedback as part of a crisis management training module; such modules are becoming increasingly embedded in surgical training. Taken together with other tools and complementary measures, OTAS © can contribute to our understanding of surgical teamwork and is a potentially useful tool in attempts to improve teamwork, train surgical teams and, ultimately enhance surgical patient care. Acknowledgements This chapter is based on a large and long-lasting research programme on teamwork in surgical teams that is being carried out by our research group. Dr Andrew N. Healey, former member of the group, played an instrumental role in the shaping and development of the OTAS © since its inception and over a number of years. Dr Mary Koutantji and Mr Peter McCulloch also contributed signicantly to this work – especially to the renement of the tool. The authors would like to thank the Department of Health: Patient Safety Research Programme (CAV), the BUPA Foundation (CAV), the Smith and Nephew Foundation (CAV), the British Academy (NS) and the Economic and Social Research Council (ESRC) Centre for Economic Learning and Social Evolution (NS) for funding for the work reported in this chapter. References Arora, S., Sevdalis, N., Nestel, D., Tierney, T., Woloshynowych, M. and Kneebone, R. (2009) Managing intra-operative stress: What do surgeons want from a crisis training programme. American Journal of Surgery 197, 537–543. Avermate van, J.A.G. (1998) NOTECHS: The Evaluation of Non-technical Skills of Multi-pilot Air Crew in Relation to the JAR-FCL Requirements. Research Report for the European Commission. Amsterdam: National Aerospace Laboratory, DGVII NLR-CR-98443. Beard, J.D. (2007) Assessment of surgical competence. British Journal of Surgery 94, 1315–16. Safer Surgery 98 Calland, J.F., Guerlain, S., Adams, R.B., Tribble, C.G., Foley, E. and Chekan, E.G. (2002) A systems approach to surgical safety. Surgical Endoscopy 16, 1005–14. Carthey, J. (2003) The role of structured observational research in health care. Quality of Safety in Health Care 12, Suppl 2, ii13–ii16. Catchpole, K R., Giddings, A.E.B., de Leval, M.R., Peek, G.J., Godden, P.J., Utley, M., Gallivan, S., Hirst, G. and Dale, T. (2006) Identication of systems failure in successful paediatric cardiac surgery. Ergonomics 49, 567–88. Davenport, D.L., Henderson, W.G., Mosca, C.L., Khuri, S.F. and Mentzer, R.M. (2007) Risk-adjusted morbidity in teaching hospitals correlates with reported levels of communication and collaboration in surgical teams but not with scale measures of teamwork climate, safety culture or working conditions, Journal of the American College of Surgeons 205, 778–84. de Leval, M.R., Carthey, J., Wright, D.J., Farewell, V.T. and Reason, J.T. (2000) Human factors and cardiac surgery: A multicenter study, Journal of Thoracic and Cardiovascular Surgery 119, Pt 1, 661–72. Dickinson, T.L. and McIntyre, R.M. (1997) A conceptual framework for teamwork measurement. In M.T. Brannick, E. Salas and C. Prince (eds) Team Performance Assessment and Measurement, 1st edition (pp. 19–44). New Jersey: Lawrence Erlbaum Associates, Inc. Dwyer, D.J., Fowlkes, J.E., Oser, R.L., Salas, E. and Lane, N.E. (1997) Team performance measurement in distributed environments: The TARGETs methodology. In M.T. Brannick, E. Salas and C. Prince (eds) Team Performance Assessment and Measurement, 1st edition (pp. 137–54). New Jersey: Lawrence Erlbaum Associates, Inc. Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran, N. and Patey, R. (2003) Anaesthetists’ Non-Technical Skills (ANTS): Evaluation of a behavioural marker system. British Journal of Anaesthesia 90(5), 580–8. Flin, R., Martin, L., Goeters, K., Hoermann, J., Amalberti, R., Valot, C. and Nijhuis, H. (2003) Development of the NOTECHS (non-technical skills) system for assessing pilots’ CRM skills. Human Factors and Aerospace Safety 3, 95–117. Fowlkes, J.E., Lane, N.E., Salas, E., Franz, T. and Oser, R.L. (1994) Improving the measurement of team performance: The TARGETs methodology. Military Psychology 6, 47–61. Fried, G.M. and Feldman, L.S. (2008) Objective assessment of technical performance. World Journal of Surgery 32, 156–60. Gawande, A.A., Zinner, M.J., Studdert, D.M . and Brennan, T.A. (2003) Analysis of errors reported by surgeons at three teaching hospitals. Surgery 133(6), 614–21. Greenberg, C.C., Regenbogen, S.E. and Studdert, D.M. (2007) Patterns of communication breakdowns resulting in injury to patients. Journal of the American College of Surgeons 204, 533–40. Observing and Assessing Surgical Teams 99 Guerlain, S., Shin, T., Guo, H., Adams, R. and Calland, J.F. (2002) A Team Performance Data Collection and Analysis System. Proceedings of the Human Factors and Ergonomics Society, 46th Annual Meeting. Guerlain, S., Adams, R.B., Turrentine, F.B., Shin, T., Guo, H., Collins, S.R. and Calland, J.F. (2005) Assessing team performance in the operating room: Development and use of a ‘black-box’ recorder and other tools for the intraoperative environment. Journal of the American College of Surgeons 200(1), 29–37. Guerlain, S., Shin, T., Guo, H., Adams, R. and Calland, J.F. (2007) A Team Performance Data Collection and Analysis System. Proceedings of the Human Factors and Ergonomics Society 51st Annual Meeting, Baltimore, USA. Healey, A.N., Undre, S. and Vincent, C.A. (2004) Developing observational measures of performance in surgical teams. Quality of Safety in Health Care 13, Suppl 1, i33–i40. Healey, A.N., Sevdalis, N. and Vincent, C. (2006a) Measuring intra-operative interference from distraction and interruption observed in the operating theatre. Ergonomics 49, 589–604. Healey, A.N., Undre, S. and Vincent, C.A. (2006b) Dening the technical skills of teamwork in surgery. Quality of Safety in Health Care 15(4), 231–4. Healey, A.N., Undre, S., Sevdalis, N., Koutantji, M. and Vincent, C.A. (2006c) The complexity of measuring interprofessional teamwork in the operating theatre. Journal of Interprofessional Care 20(5), 485–95. Healey, A.N. and Vincent, C. (2007) The systems of surgery, Theoretical Issues in Ergonomic Science 8, 429–43. Helmreich, R.L. and Davies, J. (2007) Human factors in the operating: Interpersonal determinants of safety, efciency and morale. In A.R. Aitkenhead (ed.) Clinical Anaesthesiology: Safety and Risk Management in Anaesthesia. London: Balliere Tindall. Helmreich, R.L. and Foushee, H.C. (1993) Why crew resource management? Empirical and theoretical basis of human factors training in aviation. In E.L. Weiner, B.G. Kanki and R.L. Helmreich (eds) Cockpit Resource Management (pp. 3–45). New York: Academic. Helmreich, R.L., Schaefer, H.G. and Sexton, B. (1995) The Operating Room Checklist. Technical Report 95-10. Austin, TX: NASA, University of Texas, FAA. Helmreich, R.L, Klinect, J.R., Wilhelm, J.A., Tesmer, B., Gunther, D., Thomas, R., Romeo, C., Sumwalt, R. and Maurino, D. (2002) Line Operations Safety Audit (LOSA). DOC 9803-AN/761. Montreal: ICAO. ISCP (2008) Intercollegiate Surgical Curriculum Programme. Available at: <http://www.iscp.ac.uk> [last accessed November 2008]. JCAHO (2000) The Joint Commission: Operative and post-operative complications: Lessons for the future. Sentinel Events Alert 12. Available at: <http://www. jointcommission.org/sentinelevents/sentineleventalert/> [last accessed January 2008]. Safer Surgery 100 Kaufman, L. (2003) The wrong nephrectomy. Bulletin of the Royal College of Anaesthetists 7, 848–9. Klampfer, B., Flin, R. Helmreich, R, Häusler, R., Sexton, B., Fletcher, G., Field, P., Staender, S., Lauche, K., Dieckmann, P. and Amacher, A. (2001) Enhancing Performance in High Risk Environments: Recommendations for the Use of Behavioural Markers. Report from the behavioural markers workshop, Zürich, June. Berlin: Damler Benz Foundation. Klinect, J.R., Murray, P., Merritt, A.C. and Helmreich, R.L. (2003) Line Operations Safety Audit (LOSA): Denition and operating characteristics. Proceedings of the 12th International Symposium on Aviation Psychology (pp. 663–8). Dayton, OH: The Ohio State University. Lingard, L., Reznick, R., Espin, S., Regehr, G. and DeVito, I. (2002) Team communications in the operating room: Talk patterns, sites of tension, and implications for novices. Academic Medicine 77(3), 232–7. Lingard, L., Espin, S., Whyte, S., Regehr, G., Baker, G.R., Reznick, R., Bohnen, J., Orser, B., Doran, D. and Grober, E. (2004a) Communication failures in the operating room: An observational classication of recurrent types and effects. Quality of Safety in Health Care 13(5), 330–4. Lingard, L., Garwood, S. and Poenaru, D. (2004b) Tensions inuencing operating room team function: Does institutional context make a difference. Medical Education 38, 691–9. Mackenzie, C.F., Jefferies, N.J., Hunter, W.A., Bernhard, W.N. and Xiao, Y. (1996) Comparison of self-reporting of deciencies in airway management with video analyses of actual performance. LOTAS Group. Level One Trauma Anesthesia Simulation. Human Factors 38(4), 623–35. Mackenzie, C.F., Xiao, Y. and The IPO Group (2003) Observational Analysis of Video Records of Team Performance. Proceedings of the Human Factors and Ergonomic Society 47th Annual Meeting, Denver, Colorado, 1493–7. Malec, J.F., Torsher, L.C., Dunn, W.F., Weingmann, D.A., Arnold, J.J., Brown, D.A. and Phatak, V. (2007) The Mayo high performance teamwork scale: Reliability and validity for evaluating key crew resource management skills. Simulation in Healthcare 2, 4–10. Moorthy, K., Munz, Y., Adams, S., Pandey, V. and Darzi, A. (2005) A human factors analysis of technical and team skills among surgical trainees during procedural simulations in a simulated operating theatre. Annals of Surgery 242(5), 631–9. Moorthy, K., Munz, Y., Forrest, D., Pandey, V., Undre, S. and Vincent, C. (2006) Surgical crisis management skills training and assessment: A simulation-based approach to enhancing operating room performance. Annals of Surgery 242, 631–9. O’Connor, P., Hormann, H.J., Flin, R., Lodge, M. and Goeters, K.M. (2002) Developing a method for evaluating crew resource management skills: A European perspective. International Journal of Aviation Psychology 12(3), 263–85. Observing and Assessing Surgical Teams 101 Paris, C.R., Salas, E. and Cannon-Bowers, J.A. (2000) Teamwork in multi-person systems: A review and analysis. Ergonomics 43(8), 1052–75. Roth, E.M., Christian, C.K., Gustafson, M., Sheridan, T.B., Dwyer, K., Gandhi, T.K., Zinner, M.J. and Dierks, M.M. (2004) Using eld observations as a tool for discovery: Analysing cognitive and collaborative demands in the operating room. Cognition, Technology & Work 6(3), 148–57. Royal College of Surgeons of England (2007) Safe Shift Working for Surgeons in Training: Revised Policy Statement from the Working Time Directive Working Party. London: Royal College of Surgeons of England. Sevdalis, N., Davis, R., Koutantji, M., Undre, S., Darzi, A. and Vincent, C. (2008a) Reliability of a revised NOTECHS scale for use in surgical teams. American Journal of Surgery 196(2), 184–90. Sevdalis, N., Forrest, D., Undre, S., Darzi, A. and Vincent, C. (2008b) Annoyances, disruptions and interruptions in surgery: The Disruption in Surgery Index (Disi). World Journal of Surgery 32(8), 1643–50. Undre, S. and Healey, A.N. (2006) Observational Teamwork Assessment for Surgery © OTAS © User Manual. Available at: <http://www.csru.org.uk> [accessed June 2008]. Undre, S., Healey, A.N., Darzi, A. and Vincent, C.A. (2006a) Observational assessment of surgical teamwork: A feasibility study. World Journal of Surgery 30(10), 1774–83. Undre, S., Sevdalis, N., Healey, A.N., Darzi, S.A. and Vincent, C.A. (2006b) Teamwork in the operating theatre: Cohesion or confusion? Journal or Evaluation in Clinical Practice 12(2), 182–9. Undre, S., Koutantji, M., Sevdalis, N., Selvapatt, N., Williams, S., Gautama, S., McCulloch, P., Darzi, A. and Vincent, C. (2007a) Multi-disciplinary crisis simulations: The way forward for training surgical teams. World Journal of Surgery 31, 1843–53. Undre, S., Sevdalis, N., Healey, A.N., Darzi, A. and Vincent, C.A. (2007b) Observational teamwork assessment for surgery (OTAS): Renement and application in urological surgery. World Journal of Surgery 31(7), 1373–81. Vincent, C., Moorthy, K., Sarker, S. K., Chang, A. and Darzi, A.W. (2004) Systems approaches to surgical quality and safety: From concept to measurement. Annals of Surgery 239(4), 475–82. Yule, S., Flin, R., Paterson-Brown, S. and Maran, N. (2006a) Non-technical skills for surgeons in the operating room: A review of the literature. Surgery 139(2), 140–9. Yule, S., Flin, R., Paterson-Brown, S., Maran, N. and Rowley, D.R. (2006b) Development of a rating system for surgeons’ non-technical skills. Medical Education 40, 1098–104. Yule, S., Flin, R., Maran, N., Rowley, D. R., Youngson, G.G. and Paterson-Brown, S. (2008) Surgeons’ non-technical skills in the operating room: Reliability testing of the NOTSS behaviour rating system. World Journal of Surgery 32, 548–56. This page has been left blank intentionally Chapter 7 Rating Operating Theatre Teams – Surgical NOTECHS Ami Mishra, Ken Catchpole, Guy Hirst, Trevor Dale and Peter McCulloch Introduction In this chapter we discuss the development of a tool for rating operating teams, and sub-teams of surgeons, anaesthetists and nurses, on their non-technical performance in the operating theatre. Validated in laparoscopic cholecystectomy (LC) and carotid endarterectomy (CEA), this work built upon earlier studies of behaviour and process in orthopaedic and paediatric cardiac surgery, where the importance of non-technical skills was identied through direct observation of behaviour and process in the operating theatre (Catchpole et al. 2005, 2006, 2007). No operation observed was performed perfectly, and, in all, deviations from the optimal course of the operation were found. In some, these small deviations escalated into more serious situations that compromised the safety of the patient or the success of the operation. Often these problems derived from threats in the system of surgery that originated from outside the operating theatre – or at least could not be attributed solely to errors by the teams. As our understanding of how small events can escalate to more serious problems, and the role that non-technical skills may play in reducing or increasing the chances of harm, our methods of measuring those skills have been rened, and in turn our understanding has become more sophisticated. In this chapter, we attempt rst to describe the process of intellectual and methodological development, and to provide a substantive analysis of our current tool for assessing non-technical skills in the operating room. A Model of Error Causation in Surgery Early work by the research team in orthopaedic and paediatric cardiac surgery at Great Ormond Street Hospital, London (Catchpole et al. 2005, 2006) set out to examine why errors in operating theatres occurred using a model similar to that proposed by Helmreich and colleagues for aviation (Helmreich and Musson 2000, Helmreich et al. 1999). The model adapted for surgery (see Figure 7.1) suggests that system threats can predispose and lead to human errors, revealing further . NLR-CR-98443. Beard, J.D. (2007) Assessment of surgical competence. British Journal of Surgery 94, 131 5–16. Safer Surgery 98 Calland, J.F., Guerlain, S., Adams, R.B., Tribble, C.G., Foley, E. and. attempts include • • • Safer Surgery 96 the work on development of behavioural markers systems for assessing cockpit crews, as well as early work related to healthcare and surgery with a focus. Journal of Surgery 32, 156–60. Gawande, A.A., Zinner, M.J., Studdert, D.M . and Brennan, T.A. (2003) Analysis of errors reported by surgeons at three teaching hospitals. Surgery 133 (6), 614–21. Greenberg,