Safer Surgery 184 Figure 11.1 Mini CEX trainee assessment, Victoria Inrmary Mini CEX Trainee Assessment, Victoria In�irmary Demographics Trainee name: Assessor name: Date: Stage of training: Case Details Surgical subspecialty: Surgical complexity: minimal moderate high ASA: Age: Focus of Encounter: Grading Grade the following, taking into account the level of training (NB a descriptor for each competency is outlined overleaf) Unsatisfactory Satisfactory Superior CATEGORY 1 2 3 4 5 6 7 8 9 UC Patient assessment/preparation Equipment and theatre preparation Management plan Technical skills Communication skills – Patient Communication skills – Staff Clinical judgement Organization/efficiency Professionalism Overall clinical care Assessor’s Comments What did the trainee do well? Areas for improvement: Agreed Action: Feedback 1 2 3 4 5 6 7 8 9 10 Trainee satisfaction with Mini CEX Assessor satisfaction with mini CEX Assessor training (circle as many as apply) None written training web-based training face-to-face training Time taken (in minutes) observation of trainee: feedback: Interval between observation and feedback: Non-Technical Skills and Anaesthetists’ Workplace-based Assessment Tools 185 It is important to note that use of the approach outlined above does not suggest that the MiniCEX, DOPS or CBD should replace the ANTS system. However the training of consultant anaesthetists to become trainers in the use of the ANTS system could then build upon the experience gained with the other workplace- based assessment tools. There is likely to be a need for the use of the ANTS system Figure 11.2 Mini CEX competency descriptors Mini CEX Competency Descriptors Pa�ent assessment / prepara�on Elicits relevant informa�on from history and examina�on of pa�ent Gathers informa�on from pa�ent’s notes and inves�ga�ons including medica�on history and allergies Orders further inves�ga�ons or treatment appropriately Equipment and theatre prepara�on Carries out machine check Ensures that equipment necessary for case is working and is prepared, including fluid warming devices, invasive monitoring, airway equipment, equipment for regional or local anaesthesia Prepares emergency drugs Confirms that relevant personnel are present Management plan Formulates an appropriate plan for the periopera�ve care of the pa�ent, including back - up Technical skills Communica�on skills – pa�ent Communica�on skills – staff Clinical judgement Organiza�on / efficiency Professionalism Overall clinical care Safer Surgery 186 for trainee assessment in its entirety when further exploring the non-technical skills of underperforming anaesthetists. In addition, if specic classroom training in NTS is provided, and perhaps this should be happening at an undergraduate stage, the order and structure of the ANTS system will help trainees learn about the observable non-technical skills considered important for anaesthetic practice. Summary In this chapter we have attempted to outline the journey so far to incorporate non- technical skills into the training of UK anaesthetists and to make some suggestions for the next stages of development. The ANTS project allowed us to identify NTS, describe them using a common language and develop a taxonomy of the non- technical skills which are considered crucial for good practice by the anaesthetic community. This was an important rst step. Currently there remain a number of hurdles to be overcome before non-technical skills are fully integrated into training. It is apparent that there is a need for further support and training in workplace-based assessment for consultants for all clinical activities, not just for non-technical skills training. The changes in the UK training system which require the use of a variety of workplace-based assessment tools are helping to drive this process forward. References 1 Anaesthetists’ Non-Technical Skills (ANTS) System Handbook (2003). Industrial Psychology Research Centre, University of Aberdeen. Available at: <http:// www.abdn.ac.uk/iprc /ANTS> [last accessed March 2009]. Crandall, B., Klein, G. and Hoffman, R.R. (2006) Working Minds: A Practitioner’s Guide to Cognitive Task Analysis. Boston: The MIT Press. Dodd, F. (2005) Assessment of Anaesthetic Non-Technical Skills. Presentation delivered at the Society for Education in Anaesthesia (UK) Annual Scientic Meeting: Newcastle-upon-Tyne. Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran, N. and Patey, R. (2003) Anaesthetists Non-Technical Skills (ANTS): Evaluation of a behavioural marker system. British Journal of Anaesthesia 90(3), 580–8. Fletcher, G., Flin, R., McGeorge, P., Glavin, R., Maran, N. and Patey, R. (2004) Rating non-technical skills: Developing a behavioural marker system for use in anaesthesia. Cognition, Technology and Work 6, 165–71. 1 At the time of writing there are major changes taking place in postgraduate medical training in the UK and documents from the bodies concerned with training continue to evolve. Please note that the training documents cited may have been subsequently replaced. Non-Technical Skills and Anaesthetists’ Workplace-based Assessment Tools 187 Flin, R., O’Connor, P. and Crichton, M. (2008) Safety at the Sharp End: A Guide to Non-Technical Skills. Aldershot: Ashgate. Grant, J. and Stanton, F. (1999) The Effectiveness of Continuing Professional Development: A Report for the Chief Medical Ofcer’s Review of Continuing Professional Development in Practice. Edinburgh: Association for the Study of Medical Education. Helmreich, R.L. (2000) On error management: Lessons from aviation. British Medical Journal [online] 320(7237): 781–5. Patey, R., Flin, R., Fletcher, G., Maran, N. and Glavin, R. (2005) Anaesthetists’ non-technical skills (ANTS). In K Henriks (ed.) Advances in Patient Safety: From Research to Implementation. Washington: Agency for Quality and Safety in Healthcare. Modernising Medical Careers, NHS. The Foundation Programme Curriculum (June 2007). Available at: <http://www.foundationprogramme.nhs.uk/pages/ home/key-documents> [last accessed March 2009]. The Postgraduate Medical Education and Training Programme (July 2008), Standards for Curricula and Assessment Systems. Available at: <http://www. pmetb.org.uk/index.php?id=scas> [last accessed November 2008]. The Royal College of Anaesthetists. Workplace-Based Assessment – Blueprint, Forms, Guidance and Portfolio (Updated 17 March 2009). Available at: <http://www.rcoa.ac.uk/index.asp?PageID=982> [last accessed March 2009]. This page has been left blank intentionally Chapter 12 Using ANTS for Workplace Assessment Jodi Graham, Emma Giles and Graham Hocking Introduction Around the world, administrative bodies of medical specialities are faced with the challenge of assessing their trainees. While different forms of assessment have been used for many years, assessment in the workplace is thought to be the most meaningful. The challenge is to create and test tools that will be robust enough to perform high stakes assessments. In 2006, the Australian and New Zealand College of Anaesthetists (ANZCA) offered grants for projects investigating tools to assess trainee behaviour in the workplace. ANZCA hopes to add summative workplace-based assessment to the current primary and nal examinations and to replace the formative and subjective in-training assessment (ITA). In-training assessment is used currently to assess workplace performance but has low reliability and is prone to rater bias (Spike et al. 2000) particularly since there is no assessor training within ANZCA. Grants were offered to investigate three tools – Anaesthetists’ Non-Technical Skills (ANTS), Directly Observed Procedural Skills (DOPS) and Mini Clinical Evaluation Exercise (mini-CEX). We successfully tendered for a grant to quantitatively assess the reliability and qualitatively assess the feasibility of ANTS. There is growing appreciation amongst medical educators that medical competencies need to be broader than simply medical expertise (CanMEDS 2000). There is a reciprocal growing appreciation amongst certifying bodies in medicine that examinations, isolated from the complexities of the clinical environment, do not comprehensively examine a doctor’s true performance (Hays et al. 2002). Medicine is practised in multidisciplinary teams and has an increasing reliance on technology. The introduction of safer, thus shorter, working hours has led to discontinuity of care and interdependence with other colleagues. Medical practice can now rarely be modelled as an expert having a simple one-on-one relationship with a patient. There are a range of attributes which should be common to all doctors, and these are being addressed through projects such as CanMEDS (CanMEDS 2000) and the Australian Curriculum Framework for Junior Doctors (Gleason et al. 2007). Each speciality, however, will have additional specic attributes. Many of these attributes can be taught and ideally, all should be testable. The developers of ANTS have fully explored the domains of non-technical practice in anaesthesia, creating a tool with high content and face validity (Fletcher et al. 2004). Safer Surgery 190 The reliability of ANTS has previously been investigated using videos of simulated conduct of anaesthesia, with the assessed raters being anaesthetists involved in either training or assessment of anaesthesia trainees. If ANTS were to be adopted as an assessment tool, we felt these parameters would change. Performance in simulated settings could take away the dynamism of the environment, and could be unfeasible to implement with limited access to medium or high delity simulators. Widespread trainee assessment would require involvement of signicant numbers of assessors who are less experienced in education. These changes have previously been postulated to reduce reliability (Downing 2004). This would be a stumbling block for such a high stakes assessment where a reliability coefcient should be over 0.9 in a defensible programme (Downing 2004). We were also interested in the feasibility of introducing such a tool but had specic concerns regarding the complex issues its introduction would bring, which included; difculty in training raters, acceptance by anaesthetists of ANTS as a valid tool, and time and resource allocation redirected from patient care. Methodology Our hypothesis was that ‘Anaesthetists can be trained to use ANTS at an assessor level in one day’. We designed a pilot study because the one site restriction of our ANZCA grant limited the scope of the project. We were aiming to achieve inter-rater reliability. An assessor level of agreement was determined to be an r wg (within group inter-rater agreement) of 0.7, although for high stakes assessment, r > 0.8–0.9 is desirable (Downing 2004). Previous research with minimally trained ANTS raters has demonstrated an r wg of 0.5–0.7 (Fletcher et al. 2004). Study Design The study was divided into stages. A series of videos were made of anesthetists’ real-time performance and these were used for rater training. A rater training day was then developed and implemented. Reliability data were calculated from ratings of performance collected from ve separate videos. Pre- and post-workshop questionnaires were handed out with the intention of gathering qualitative information. Demographic data were collected; questions asked by the Scottish group (Fletcher et al. 2004) were repeated, other questions added and acceptability by the participants of ANTS as a summative assessment tool was determined. Video This aspect of our research has generated an enormous amount of interest as it seems that many people would like to video workplace performance. There are Using ANTS for Workplace Assessment 191 many administrative, ethical and some legal requirements to overcome. Described below is the process we followed, with very little trouble. Of course, different countries and even other Australian states have differing laws to consider. Consent for Video During the design phase of this project, the feasibility of using real-time video in the operating theatre was investigated. Looking back, this was probably one of the most signicant steps in the design of this project. Ensuring the video process was possible before embarking on the study ensured that we were able to complete the project on time. Investigating the use of video was achieved by contacting the chief executive ofcer (CEO) of our hospital for advice. Ensuring that our audio-visual department was willing and able to produce the videos was essential. Advice from our CEO indicated that we must gain written consent from both the patient and the principal anaesthetist taking part in the video. Patient protection was foremost in all our minds. Protecting the hospital from potential litigation was also going to be of major concern. Therefore, the consent process and any conditions were focused specically around the patient. The consent process was simpler for the anaesthetist and theatre staff. Written consent was obtained from the main anaesthetist in the video and verbal consent from other theatre staff. Any staff member who did not wish to be seen on video either left the theatre or was edited out if they appeared. The documentation was based on the standard hospital consent form, but had to state a number of items concerning use of the video. These included where the videos would be stored (e.g., locked cabinet), how long they would be stored for, and who would see them. The specicity of where the videos would be stored was of paramount importance. Consent could be withdrawn at any time. To ensure the video could not be used as evidence by the patient if there was a mishap of some kind, the research was registered as a quality assurance project. By doing this, the hospital was satised that any evidence captured on video would not be discoverable. Recruitment Video production was divided into phases. Volunteers were found amongst our anaesthesia registrars and consultants at Sir Charles Gairdner Hospital. We were able to nd enough volunteers to lm the 20 videos required for the project. Our colleagues who allowed us to video them at work have our gratitude. We now realize that what seemed simple at the time has in fact been very difcult for others to achieve. We suspect the strong focus on education at our institution led to our success. The audio-visual department made a number of posters which were displayed in the clinic where patients undergo pre-operative assessment. The posters were also Safer Surgery 192 seen in the day surgery admission centres and elsewhere patients may be entering the hospital for surgery. The posters explained briey that some patients might be asked if their anaesthetist could be lmed during the start of their anaesthetic. Brief details of the project were included. This ensured that when patients were approached, they were already aware of the project. Patients were more than willing to take part in the videos and no patient refused a request. Some even commented they were disappointed not to be asked so it seems there is a budding lm star in all of us! Our hospital predominantly performs elective surgery, making it somewhat easier to recruit patients pre-operatively. Even those who asked questions were pleased to participate in a project that was ultimately aimed at improving patient safety. Some requested a copy but were politely declined, understanding the reason for this when it was explained. Following the end of video production, patients were still enquiring about participating in the video after seeing the signs. Unfortunately, we had to inform them that the project had been completed. Video Production Video production commenced in January 2007 and took seven months. The videos were produced by the audio-visual department at Sir Charles Gairdner Hospital. They did an outstanding job, especially considering they were sometimes required to attend at relatively short notice. The quality of the video produced was very high, ensuring that the raters had the opportunity to observe even subtle behaviours. When assessing videoed performance, the sound and picture quality are crucial since the people on screen are being assessed on their behaviours. What the assessor sees and hears will determine their assessment. Care must be taken by the cameraman and editor to ensure behaviours are not missed. To achieve the quality required, two high quality tripod mounted video cameras were used and these were handled by professionals, not your average home video maker. One zoomed camera followed the subject anaesthetist/s and the other recorded the wider theatre view. This allowed us to choose the footage that best exhibited what needed to be seen. For example, if the anaesthetist’s behaviour was being inuenced by the other theatre staff then we were able to see this. Alternatively, if what they were doing was inuencing other theatre staff, we could also observe this. Each subject anaesthetist was asked to wear a lapel microphone. The rest of the conversation and noise in theatre was captured by a separate camera mounted microphone. The lapel microphone enabled the viewer to clearly hear almost all that was going on. Prior to the start of this project we had taken a test video with a small camera. Although adequate, it was difcult to hear all that was required. Sound is one of the most important aspects of video production for this purpose. The video taken was of anaesthetists’ real-time performance. This was chosen instead of simulated crises for a number of reasons. Simulation is not a standard assessment tool and while crisis management is relevant, this is something we may revisit in the future. Assessment in the workplace is something that can occur any day in each hospital. It is familiar and accessible. For this reason we chose to train Using ANTS for Workplace Assessment 193 our raters viewing real anaesthesia. All videos were unscripted as it was important that we were able to observe real behaviours since subtleties in behaviour are often lost by non-professional actors acting to a script. Twenty videos were decided upon for the purposes of both this project and any future projects to stem from the original. The number was not completely arbitrary. Initially, the number required for the rater training day was calculated and thereafter, the number of videos required to catch a range of behaviours to use as examples was agreed. From prior use of ANTS we knew approximately how often the less common behaviours would occur. Twenty videos were thought to be enough to allow us not only some choice, but some good examples of particular behaviours. Editing of the footage was performed by the expert raters and staff members of the audio-visual department using Final Cut Pro (Apple Inc.). The two experts rated each video. These were used as the reference points for calibration on the training day. At the time of editing, a selection of excerpts was also chosen to illustrate particular behaviours. The videos were used for many aspects of our rater training day. These included performance dimension training, behavioural observation training and calibration compared to expert raters. These aspects of the rater training are described in detail later in this chapter. ANTS Rater Training Development of the rater training day was the second phase of our project. There has been much previous research in the area of rater training. A simple search of psychology literature will reveal a large body of work on this topic. We based our training on that used in other industries (Salas et al. 2001, Woehr and Huffcutt 1994). Some of the issues surrounding rater training include: Time taken to train raters to the standard that is required for reliable and accurate assessment. Rater calibration, frequency accuracy and inter-rater agreement required between assessors. Homogeneity of assessor training to ensure standards are maintained. Rater training for behavioural marker systems has been used in aviation and other industries for a number of years. An adequate level must be achieved in their assessment to continue working or re-training must occur. Rater training aims to improve rater accuracy and agreement between raters. Strategies to improve rater training include: Rater Error Training (RET): The goal is to reduce rating error and produce more normally distributed ratings. • • • • . audio-visual department made a number of posters which were displayed in the clinic where patients undergo pre-operative assessment. The posters were also Safer Surgery 192 seen in the day surgery. Safer Surgery 186 for trainee assessment in its entirety when further exploring the non-technical skills. in anaesthesia, creating a tool with high content and face validity (Fletcher et al. 2004). Safer Surgery 190 The reliability of ANTS has previously been investigated using videos of simulated