Safer Surgery 4 Millman, M. (1976) The Unkindest Cut. Life in the Backrooms of Medicine. New York: Morrow Quill. Ruhlman, M. (2003) Walk on Water. Inside an Elite Paediatric Unit. New York: Viking. Weston, G. (2009) Direct Red. A Surgeon’s Story. London: Jonathan Cape. Wilpert, B. (1996) Psychology in high hazard systems: Contribution to safety and reliability. In J. Georgas, M. Manthouli, E. Besevegis and A. Kokkevi (eds) Contemporary Psychology in Europe. Proceedings of the IVth European Congress of Psychology. Seattle, WA: Hogrefe and Huber. PART I Tools for Measuring Behaviour in the Operating Theatre This page has been left blank intentionally Chapter 2 Development and Evaluation of the NOTSS Behaviour Rating System for Intraoperative Surgery (2003–2008) Steven Yule, Rhona Flin, Nikki Maran, David Rowley, George Youngson, John Duncan and Simon Paterson-Brown Introduction In 2002, a number of surgeons in Scotland were intrigued by the development of the ANTS (Anaesthetists’ Non-Technical Skills) system and the use of behaviour rating checklists in other industries such as nuclear power and civil aviation. There was a realization in the healthcare and medical literature that adverse events occurred in the operating theatre. Surgeons and their patients have also had to come to terms with the uncovering and analysis of the true nature and extent of surgical misadventure and failure. Ten years ago, it was not generally realized that a signicant number of surgical patients were harmed not as a result of underlying illness or disease but as a result of their treatment (Vincent et al. 2001). Further analysis of this problem revealed that non-technical aspects of performance play a contributory role in the multifaceted nature of surgical adverse events; failures in decision-making, teamwork, coordination and leadership have all emerged from case reviews and studies of behaviour in the operating theatre (Gawande et al. 2003, Studdert et al. 2006, Christian et al. 2006). Non-technical skills are dened as the critical cognitive and interpersonal skills that complement surgeons’ technical ability (Yule et al. 2006a). Despite the fact that the behavioural (Baldwin et al. 1999) and cognitive demands of surgery have been recognized as a critical part of surgical performance (Hall et al. 2002, Jacklin et al. 2008), and that effective leadership has been shown to improve team performance (Edmondson 2003), non-technical skills are often referred to as ‘non-operative’ and deemed not as important as clinical science in the surgical literature. Scant attention has been paid to the cognitive and social processes that underpin intra-operative performance in training as well: training and assessment in these skills are only conducted in a rather tacit and discretionary basis, and the surgical curriculum in the UK does not yet extend to non-technical skills. Safer Surgery 8 Amid this backdrop, the surgical profession has also been rapidly changing to cope with internal and external pressures such as the European Working Time Directive, which restricts the working week to 48 hours; the challenges of new professional roles such as nurse practitioners (Kneebone and Darzi 2005), the modernization of training and education including Medical Training Application Service (MTAS) and new technology. These changes mean that trainees have fewer training opportunities than their trainers had, before reaching consultant level, so there is now a greater need to maximize the available learning opportunities. Changes to the conguration of surgical training and education are currently underway in the UK to attempt to streamline development of doctors and ensure that they are skilled at communicating and working as effective members of a team. This approach recommends that progress through and completion of surgical training be based on competence; it has moved the emphasis of assessment away from set-piece examinations of knowledge towards learning and assessment of skills in the workplace (see Pitts and Rowley, Chapter 3 of this volume). Selection of trainees into surgical specialties has also been radically altered and provides an opportunity to formalize the role of non-technical skills in surgical education and assessment. The main methods of workplace-based assessment of surgical trainees in the UK are observational tools which cover skills such as ability to work in a multi-professional team (Mini-PAT: Peer Assessment Tool) and communication (Mini-CEX: Clinical Evaluation Exercise). However, these tools are for the assessment of perioperative skills, often using interactions with patients as a basis for assessment. This is to be encouraged, but the skills assessed do not necessarily relate to those required for working with other professionals during a surgical procedure, commonly with an anaesthetized patient. The systems that are used to assess trainees’ intra-operative competence, such as surgical DOPS (Direct Observation of Procedural Skills) and Procedure Based Assessment (PBA) are focused almost entirely on technical ability. However PBAs, which are written for specic index procedures, sometimes integrate non-technical aspects (see in this volume Chapter 3 by Pitts and Rowley, Chapter 4 by Marriott et al.). The cognitive and social skills which underpin clinical and technical prociency are recognized as requirements for surgical competency and rank highly as core competencies within organizations such as CanMeds (Frank 2005), the General Medical Council (GMC 2001), and the Royal Colleges of Surgeons in the UK (Youngson 2000, Giddings and Williamson 2007) but until recently there were no tools to reliably assess these skills in the workplace. To begin to address this, a behavioural observation and rating system called NOTSS (Non-Technical Skills for Surgeons) was developed and tested under funding from the Royal College of Surgeons of Edinburgh and NHS Education for Scotland, from 2003 to 2007. This chapter outlines the development and evaluation of the NOTSS system. Like similar systems in civil aviation and anaesthesia, the behaviour rating system was based on a skills taxonomy which was developed with subject matter experts. Development and Evaluation of the NOTSS Behaviour Rating System 9 NOTSS Project Design The project was run by the University of Aberdeen, with a multidisciplinary steering group of surgeons, psychologists and an anaesthetist. The research drew on previous work in Scotland on surgical competence, professionalism and the skills surgeons required to operate safely and followed on from a similar project which developed a behaviour rating system for anaesthetists – the ANTS system (Fletcher et al. 2004, see Chapter 11 in this volume by Glavin and Patey). The aim of the NOTSS project was to develop and test an educational system for assessment and training based on observed skills in the intra-operative phase of surgery. The system was developed from the bottom up with subject matter experts (consultant surgeons), instead of adapting existing frameworks used in other industries. It was considered important to recognize and understand the unique aspects of non-technical skills in surgery, and not to assume that those non-technical skills identied for pilots, nuclear power controllers or anaesthetists would be exactly mirrored in, or be relevant to, surgery. The NOTSS system is in surgical language for suitably trained surgeons to observe, rate and provide feedback on non-technical skills in a structured manner. An adaptation of Gordon’s (1993) model of systems design was used to guide the iterative development of NOTSS. This three-phase model maps the process from task analysis through system design to evaluation. The phases relate to the three objectives set by the NOTSS steering group in 2003: to identify the relevant non-technical skills required by surgeons, to develop a system to allow surgeons to rate these skills, and to test the system for reliability and usability. A fourth phase was added to cover a trial in the operating theatre using NOTSS to debrief surgical trainees over the course of an attachment (see Figure 2.1). Phase 1: Task Analysis In Phase 1 we used three main methods to collect data on individual surgeons’ intra-operative non-technical skills, as follows: Literature review on surgeons’ non-technical skills (Yule et al. 2006a). Survey of theatre personnel attitudes to teamwork, error and safety (Flin et al. 2006a). Critical incident interviews with subject matter experts (Yule et al. 2006b). These methods were supported by eld notes taken during observation sessions in the operating theatre during operative surgery and a review of surgical adverse event and mortality reports. 1. 2. 3. Safer Surgery 10 Literature Review The aims of the literature review (Yule et al. 2006a) were to examine the surgical and psychological literature on surgeons’ intra-operative non-technical skills in order to (i) identify the non-technical skills required by surgeons in the operating theatre, and (ii) assess the behavioural marker systems that have been developed for rating surgeons’ non-technical skills. In order to achieve this, we searched the literature using dened search terms and a set of inclusion criteria. Databases searched included BioMed Central, Medline, Web-of-Knowledge, PsychLit, and ScienceDirect. Relevant studies were organized according to the source material used. This yielded published research from observational studies, questionnaires and interviews, adverse event analyses and papers on surgical education (including curricula and standards of competence). Within these, the review highlighted the main non-technical skill categories to be: anticipation, decision-making, teamworking, leadership and communication. At the time of the review (August 2005), there were three research tools in the literature which could be used to measure surgeons’ non-technical skills. On closer examination, these existing frameworks were found to be decient either in terms of their psychometric Figure 2.1 Developing the NOTSS system Phase 2: Design and development (Yule et al., 2006b) Iterative development (n=4 panels of consultant surgeons) Write and agree behaviour markers (n=16 consultant surgeons) Phase 4: Debriefing on non-technical skills (in progress) Phase 3: System evaluation (Yule et al., 2008a, 2008b) Reliability (standardized scenarios, n=44 consultant surgeons) Usability: 2 studies, n=27 surgeon-trainee dyads in total Phase 1: Task analysis (Yule et al., 2006a; Flin et al., 2006a) Literature review, cognitive interviews (n=27), attitude survey, Adverse event report analysis Developing the NOTSS system (based on Gordon, 1993) Development and Evaluation of the NOTSS Behaviour Rating System 11 properties or suitability for assessing individual surgeons rather than a surgical team in theatre. On the basis of this review, we concluded that further research was required to develop a taxonomy of individual surgeons’ non-technical skills for training and feedback. Attitude Survey (ORMAQ) The literature review highlighted the lack of basic data on cognitive and social skills in surgeons, and little was known about prevailing attitudes to teamwork and safety in the operating theatre. Attitude surveys of theatre personnel had been conducted in other countries and can provide useful diagnostic information relating to behaviour and safety in surgical units. There were no such data available in Scotland, so as part of our initial task analysis, we ran a baseline survey (Flin et al. 2006a) using a version of the Operating Room Management Attitudes Questionnaire (ORMAQ), initially developed by Helmreich et al. (1997) to assess surgical team members’ attitudes to safety and teamwork in operating theatres. The ORMAQ was adapted from an instrument measuring pilots’ safety attitudes in aviation. At the time (late 2005), it was the most extensively used attitudes questionnaire with operating theatre personnel with data collected from Israel, USA, Germany, Switzerland and Italy (Helmreich and Schaefer 1994, Helmreich and Davies 1996, Sexton et al. 2000). It was not clear to what extent these earlier ndings would generalize to a British sample but the questionnaire topics of leadership, teamwork, stress and fatigue and error were shown to be relevant from our literature review. The ORMAQ was modied for language only by a panel of consultant surgeons and was distributed to surgical teams in 17 hospitals in Scotland. A total of 352 responses were analysed, 138 from consultant surgeons (response rate: 47 per cent), 93 from trainee surgeons (27 per cent) and 121 from theatre nurses (19 per cent). Respondents generally demonstrated positive attitudes to behaviours associated with effective teamwork and safety. Attitudes indicating a belief in personal invulnerability to stress and fatigue were evident in both nurses and surgeons. Consultant surgeons had more positive views on the quality of surgical leadership and communication in theatre than trainees and theatre nurses. While the ubiquity of human error was well recognized, attitudes to error management strategies (incident reporting, procedural compliance) suggest that they may not be fully functioning across hospitals. While theatre staff placed a clear priority on patient safety, against other business objectives (e.g., waiting lists, cost cutting), not all of them felt that this was endorsed by their hospital management. Discrepancies were found between the views of consultants compared to trainees and nurses, in relation to leadership and teamwork. While attitudes to safety were generally positive, there were several areas where theatre staff did not seem to appreciate the impact of psychological factors on technical performance. These results were taken into consideration in the design of the NOTSS system. Safer Surgery 12 Observations To provide context and meaning for the literature review and interviews, a psychology researcher conducted observations of surgical cases. Observations were made at three hospitals in a variety of specialisms: general, orthopaedic and cardiac surgery. No formal method was used for structuring observations because we did not want to narrow the observer’s data collection at this stage. Field notes were taken. The observer also shadowed surgeons in the perioperative environment to understand how this stage impacted on operative performance. During this phase of the project, detailed eld notes revealed that surgeons displayed a range of non-technical skills, communication was variable and there often seemed to be conicting priorities between training and service delivery. There was no standard method of conducting a given operation, the atmosphere or climate in the operating theatre would change depending on which surgeon was operating that day, and the number of people in the operating theatre ranged from four to eighteen. In comparison with other industries, the formal work procedures, if they existed, were not explicit. Team members seemed to start critical tasks such as commencing the anaesthetic, positioning the patient and making the rst incision to start the operation without speaking to other members. Often operations would start without critical team members in the operating theatre and without all the information being present. Distractions seemed to be commonplace and normal; on several occasions the operating surgeon had to answer questions about another ongoing operation or speak to someone on the telephone while in the middle of what appeared to be a complex part of the operation for which he or she was scrubbed. Despite all this, the observers were struck by how well the surgeon and the team performed under those circumstances. As with any observation study, it was not possible to plan to see surgeons perform under stress or to analyse surgical adverse events in a systematic manner during live cases. For this, we selected other methods, as will be discussed below. Adverse Event and Mortality Reviews The systematic analysis of near misses, incidents and accidents is an essential diagnostic process for safety management in industry (Reason 1997) and we thought that these data sets in surgery could provide us with a rich source of information on error and surgical failings that would credibly t into the skills analysis for NOTSS. Surgical colleagues indicated that data were not usually collected on non-technical skills, so this would be a short task. In the end, we reviewed the Scottish Audit of Surgical Mortality (SASM) reports from 2001 (SASM 2003) and commented on them in the literature review. The nature of data fed back to individual hospitals and in case assessments highlights that SASM is strong on providing technical feedback and on reporting the proximal causes of error but provides relatively little in the way of human factors information and therefore offered limited insight into non-technical skills in surgery. There are two likely Development and Evaluation of the NOTSS Behaviour Rating System 13 causes of this: (i) the forms used to collect data do not adequately capture human factors or non-technical contributions to incidents, and (ii) the coding framework used to analyse the incident reports does not adequately deal with non-technical skills. A similar situation emerged in the analyses of anaesthetic adverse event reports for the ANTS project (Fletcher et al. 2004). These conditions explain the current technical (e.g., what happened) bias in published audit reports in favour of non-technical (e.g., why it happened) causes of adverse events. The SASM forms since 2007 include non-technical skills categories. Critical Incident Interviews The critical incident technique (CIT) is a type of cognitive interview (Crandall et al. 2006, Flanagan 1954, Hoffmann et al. 1998) used to identify tacit knowledge about the way an expert manages a stressful or non-routine situation at work. CITs were conducted with 27 consultant surgeons in order to identify non-technical skills used by surgeons in the intra-operative environment. By focusing on a specic memorable incident, the interviews provided insight into the surgeon’s use of information, strategies, meta-cognition, resources and interpersonal skills during an operative case (Yule et al. 2006b; see also Fletcher et al. (2004) who used this technique with anaesthetists). To summarize the method, surgeons were asked to recall events in theatre during a challenging, non-routine case and were probed about the course of events a further two times. After the surgeon described the case, the interviewer recounted the sequence of events back to the surgeon and asked for clarication and more explanation of the course of events. This second sweep of the case allowed for more detail to be gleaned. The case was then discussed for a third time with the addition of cognitive cues which recreate aspects of the case to elicit deeper-held tacit knowledge about the non- technical skills that were or were not being used. Examples of the cognitive cues used include: ‘what cues were you using to help understand the situation’ and ‘how did you re-establish goals?’ The interview questions were developed by a multidisciplinary group, based on work in other domains including anaesthesia and piloted with three consultant surgeons. The sample of surgeons interviewed were consultant surgeons (n=27) from 11 hospitals in Scotland in general surgery (n=13), orthopaedic surgery (n=10) and cardiac surgery (n=4). One of the participants was female. A variety of cases were discussed in the interviews which lasted around one hour each, including emergencies with duodenal ulcers, difculties in hip and knee replacements, problems in transplant operations and difculties with cardiac bypass. The interview transcripts were analysed using the line-by-line coding technique from grounded theory (Glaser and Strauss 1967) in order to explore the data and aid system development. Coders were asked to identify when non-technical skills were discussed in the interview and to interpret those specic skills. Three pairs of psychologists who were experienced at coding interview transcripts each coded six transcripts independently to an acceptable level of inter-rater reliability before the remaining transcripts were then coded. This . surgeons (n=27) from 11 hospitals in Scotland in general surgery (n=13), orthopaedic surgery (n=10) and cardiac surgery (n =4) . One of the participants was female. A variety of cases were discussed. sessions in the operating theatre during operative surgery and a review of surgical adverse event and mortality reports. 1. 2. 3. Safer Surgery 10 Literature Review The aims of the literature. Safer Surgery 4 Millman, M. (1976) The Unkindest Cut. Life in the Backrooms of Medicine. New York: Morrow