This page has been left blank intentionally Chapter 11 Integrating Non-Technical Skills into Anaesthetists’ Workplace-based Assessment Tools Ronnie Glavin and Rona Patey Introduction In this chapter, the authors will provide a brief account of the development of the Anaesthetists’ Non-Technical Skills (ANTS) system and then consider how this system may nd a suitable home in the UK anaesthetic curriculum (Anaesthetists’ Non-Technical Skills System Handbook). But rst, what are non-technical skills (NTS)? Within the context of anaesthesia they have been dened as behaviours in the operating theatre environment, which are not directly related to the use of medical expertise, drugs or equipment (Fletcher et al. 2003). They encompass both interpersonal skills (e.g., communication, team working and leadership) and also cognitive skills (e.g., situation awareness and decision making) (Anaesthetists’ Non-Technical Skills System Handbook). NTS are important because we need them to function effectively in our professional lives. As with many important aspects of living, we really only notice them when they are absent or poorly performed. Then, our practice is not so effective and may become dangerous. In anaesthesia, and many other professions, studies of adverse incidents consistently suggest that the underlying factors are more likely to be due to difculties with the application of NTS rather than a lack of basic knowledge or practical skill (Flin et al. 2008). Is it then reasonable to ask: ‘If NTS are so important why have they not featured prominently in anaesthetic training?’ One might reply that they might have done had we known what they were exactly in the context of anaesthesia. Until now NTS have been elusive and difcult to describe or pin down. This inability to accurately dene or describe NTS has resulted in them generally only being considered under a broad umbrella category of ‘soft skills’. However, there is nothing soft about these skills. The platform they provide, along with the knowledge and skills from more conventional areas of the curriculum, is the solid base required for the development of anaesthetic practice. So far, we have referred to NTS as if they were a generic set of skills found across many circumstances and professions. Bob Helmreich, emeritus psychologist at University of Texas, Austin, played a key role in the early NASA work in this eld of key human factors and then oversaw the development of the concept from Safer Surgery 176 infancy to maturity in other industries. He reminds us that there is no generic set of NTS and has warned that different organizations/professions and cultures have their own variations of NTS (Helmreich 2000). Therefore, whilst NTS identied in other professions can give us a avour of those used in anaesthesia, we cannot just assume that they cross directly into our practice. With this in mind, in 1999, having conrmed that such work was not already under way, a research team comprising clinical anaesthetists (including both authors) and psychologists set about the business of identifying the non-technical skills important for anaesthetic practice (funded by NHS Education for Scotland). Development of the Anaesthetists’ Non-Technical Skills System How did we do this? We believed that anaesthetists had always learned and used NTS but that this was largely an unconscious process. Therefore, we could not simply ask anaesthetists ‘What non-technical skills are important in your practice?’ Nor would the quantitative or positivistic approach, which was the research tradition that the clinical anaesthetists in the team were familiar with, be appropriate. This was not an investigation where quantitative measurement, such as where a new technique is developed to impact on blood pressure or cardiac output, would help. A different approach was required. A set of task analysis techniques was used with questionnaires, observations and interviews. For the last, a form of cognitive task analysis was employed (Crandall et al. 2006, Fletcher et al. 2004). This is a qualitative research method whereby the researcher uses a set of probes, usually in the form of questions, to help the interviewee (in this case consultant anaesthetists) describe one or more episodes of clinical practice. The interviews contained the raw data and were analysed by extracting all the behaviours and skills that were mentioned. These approaches allowed access to behaviours which may not otherwise have impinged upon the consciousness of the participants. These elicited behaviours do not come neatly ordered, and so for clarity and ease of use, a system has to be assembled from the data. For this, a series of iterative steps took place where the research team developed a prototype categorization of the principal behaviours and took rudimentary versions of the system into the workplace and studied their apparent effectiveness. After several iterations the current Anaesthetists’ Non-Technical Skills (ANTS) system was arrived at (see Table 11.1). The next step was to evaluate some key properties of this system. Was it valid? Could it be used in a reliable manner – would different assessors give similar scores to the same performance? At an even more fundamental level, could clinical anaesthetists actually use the system? To address these questions, 50 consultant anaesthetists across Scotland took part in a study. They attended a one-day session comprising a morning of teaching and familiarization with the concepts behind the system and the system itself followed by an afternoon of applying the system to eight video-recorded anaesthetic scenarios that had been conceived, performed and produced by the research team. The ndings have been published elsewhere Non-Technical Skills and Anaesthetists’ Workplace-based Assessment Tools 177 (Fletcher et al. 2003), but the main conclusions were that the clinical anaesthetists could use the system and could provide broad consensus on the scores awarded, and that the system captured the NTS observed. However, these anaesthetists were scoring performances while observing them on videotape, without any other distractions. It was important to assess if the system could be used in the workplace. Here, if trainers were to apply the system to assess the performance of a trainee in order to provide feedback, they would also be required to continue to monitor the patient and the operating theatre environment. Of the original volunteers, 17 participated in the usability study. The consultant anaesthetists and the trainees being supervised were asked to complete questionnaires at the end of the theatre session. The results of this usability study, which are detailed elsewhere, were positive (Patey et al. 2005). It seemed that the system could be used in the workplace and furthermore participating trainers and trainees responded in a very positive manner to use of the system as a way of providing formative assessment and teaching of NTS. Was the anaesthetic community in the UK now ready to adopt the ANTS system into routine training? Early Experiences of Promoting the ANTS System The body responsible for standards of anaesthetic practice and training of anaesthetists in the UK is the Royal College of Anaesthetists. Presentations on the ANTS system to the relevant committees and bodies of stakeholders produced a very positive response. Following this, the next hurdle was how to best assimilate Category Element Task management Planning and preparing Prioritizing Providing and maintaining standards Identifying and utilizing resources Team working Coordinating activities with team members Exchanging information U sing authority and assertiveness Assessing capabilities Supporting others Situation awareness G athering information Recognizing and understanding A nticipating Decision-making Identifying options Balancing risks and selecting options Re-evaluating Table 11.1 The ANTS system: categories and elements Safer Surgery 178 the ANTS system into the teaching and assessment of anaesthetists. The usability study had shown that individual consultant anaesthetists could use the system, but could the ANTS system be incorporated into the teaching practice of a whole hospital department of anaesthesia? The proposed strategy for introduction to a department was to use a cascade system for training the trainers. One of the authors of this chapter would visit several UK anaesthetic departments that had volunteered to trial the use of the ANTS system and deliver a one-day teaching session for local enthusiasts. The cascade approach was based on the notion that those who had attended the one- day course would use the system, become more familiar with its application and then reach a stage where they would be comfortable to instruct their consultant colleagues. Similar to the initial system evaluation study, the training day would consist of a morning going over the background to NTS, followed by familiarization with the ANTS system. In the afternoon, the performance of anaesthetists in video-recorded scripted scenarios would be assessed allowing practice using the system. The afternoon differed from the research work in that the participants were encouraged to discuss their scores to allow calibration to take place. When the participants reached consensus and awarded very similar scores then that part of the session ended. Would this cascade approach work? Although NTS could be identied and discussed by consultants during ANTS training sessions, on the whole the cascade approach was not successful. It seems from discussion with consultants that the reasons for this were multifactorial (Dodd 2005). Firstly, most consultants only had limited familiarity with NTS and experience with the ANTS taxonomy (only the single day of training and some recommended reading) and wanted further opportunities and support to become more familiar with the tool before using it in the workplace. Accurately observing behaviours is a skill, like any other, which requires deliberate practice, and although consultants regularly participated in clinical training they had not received training in behavioural observation. Where the consultants had no formal training or experience in providing structured feedback (which was often the case), they were particularly uncomfortable. In all probability, too many new changes were introduced at one time but it is worth exploring the intended training approach and barriers to its success in greater detail. A conventional approach to instructing anaesthetic trainees in the use of the NTS highlighted in the ANTS system would be to incorporate the underlying theoretical concepts into classroom-style teaching and then build on that base during clinical training by demonstrating application of those principles. Knowledge and practice would then be formally assessed in some kind of examination. We were adopting an entirely different approach, where the system was being used to score the performance of a trainee anaesthetist in the workplace. The scores awarded for the relevant categories and elements were intended to form the basis for a discussion between consultant and trainee as a way of drawing attention to the trainees’ existing strengths and limitations in the use of NTS and then plan for further development. We had made the assumption that consultant anaesthetists were used to assessing trainees Non-Technical Skills and Anaesthetists’ Workplace-based Assessment Tools 179 on accompanied theatre lists and then providing structured feedback. However, although consultants had considerable experience assessing the performance of trainees this was done in a very informal manner. Many consultants in the UK had little experience in the use of formal workplace-based assessment tools. This was not surprising because no such tools were in widespread use in the UK. Workplace-Based Assessment and the Curriculum in Anaesthesia One result of this limited use of workplace-based assessment tools is that little attention has previously been paid to the requirements to support consultants in providing structured feedback in the workplace. Consultant training, to highlight the essential role of feedback in both optimal learning and in the use of effective feedback techniques, has been limited. In addition, it can be difcult during a busy working day for the consultant and trainee to identify time for feedback. If time can be identied, perhaps at the end of the day, it may not be close to the event under consideration, and at this time both parties may be concerned with other duties such as post-operative or pre-operative visiting. The conditions during a training session in operating theatres are also not ideal. The presence of the whole theatre team can be inhibiting, particularly where there is discussion to be had on an area where there are difculties. The authors have noted that this seems to cause particular concern where NTS are concerned and consultants perceive that they are commenting on personality-related matters. These various factors all conspire to make it easy to delay a feedback discussion. When time can be identied for feedback, use of the ANTS system should actually reduce the anxiety of commenting on a trainee’s personality traits. The system allows discussion and emphasis on specic NTS behaviours either observed or thought to be decient or absent, rather than comments which are difcult to interpret or may be interpreted as personality assessments. Discussion can then follow on how these specic behaviours impact on the ANTS system elements and categories. The consultant can then encourage the trainee to consider why these behaviours were observed or not. However, as we have previously noted, providing feedback is not the only skill in which consultant anaesthetists lack training. Accurately observing behaviour is a skill in itself and requires deliberate practice for learning. This situation, where there is limited use of formal workplace-based assessment, is changing in the UK in response to the processes brought about under the umbrella of Modernising Medical Careers (The Foundation Programme). The body with regulatory powers for postgraduate medical training (the Postgraduate Medical and Training Board, referred to as PMETB) with the right to approve curricula, including assessment tools, is promoting workplace-based assessment tools as one method for providing evidence that trainees are achieving the appropriate competencies for their respective curricula (The Postgraduate Medical Education and Training Board). The proposed workplace-based tools are being modied from tools introduced in the UK in 2005 for use during the rst two years of postgraduate training (the UK Safer Surgery 180 Foundation Training programme). These tools are the Mini Clinical Evaluation Exercise (MiniCEX), the Directly Observed Procedural Skills (DOPS), the Case Based Discussion (CBD) and a form of 360° feedback. The MiniCEX can deal with many aspects of interaction with a patient including history taking, conducting an examination or explaining benets and risks of procedures. The DOPS focuses on procedural skills including the decision-making, preparation and set up of the procedure. The CBD is intended to look at decision-making processes by referring to information recorded by trainees in the patient’s case notes, while the 360° allows peers and other colleagues from a range of professions to comment on professional attitudes and teamworking skills. Specially modied versions of these tools are now described for anaesthetic speciality training (The Royal College of Anaesthetists, Workplace-based Assessment). At the time of writing, there is a sense of concern in the anaesthetic community with regard to these tools. Consultant anaesthetists have embraced the principles behind workplace-based assessment. However, as we found when considering use of the ANTS system, consultants have expressed concern about the level of their training, support and preparation for using these tools. Moreover, for many who have not yet considered NTS, these reactions have been focused on areas of anaesthetic practice which have been traditionally assessed (technical skills). It is therefore hardly surprising that asking consultants to conduct workplace-based assessments of NTS, an area in which they had no previous experience, was a step too far. It is also likely that the cascade approach was overambitious in that too much was asked of too few people without ongoing support (Grant and Stanton 1999). Nevertheless, despite the cascade approach having had limited success, the Royal College of Anaesthetists has continued to support the concept of using the ANTS system and it featured as one of the assessment tools submitted to PMETB for approval. Using the ANTS System to Frame Teaching However, the difculty with the cascade approach to incorporating NTS into anaesthetic training was not just a problem of becoming familiar with observing behaviours, using the ANTS system and considering how best to give feedback. There were barriers because of the very nature of clinical training in the UK. In this training system the trainee spends regular time providing anaesthetic care with a trainer, gradually taking a more central role in the clinical work in a wider range of clinical settings. This would seem to be ideal in that there are regular opportunities for the trainer to observe the trainee’s practice and provide feedback. The method of use of the ANTS system which was favoured by the 17 anaesthetists during the usability study was to allocate the perioperative management of a particular patient to the trainee accompanying them on the theatre list. They would then assess the performance of the trainee and, using the observed behaviours, provide feedback after the case. The consultant, of course, retained overall responsibility and could step in and make changes if necessary but, in the absence of concerns about patient Non-Technical Skills and Anaesthetists’ Workplace-based Assessment Tools 181 or trainee safety, would let the trainee handle the management of the patient. This approach had some implications. The trainer does not only have the trainee to observe but also the clinical work. As a senior trainee has more experience than a junior trainee, the consultant is more likely to have condence that the trainee will identify the key issues involved with the management of a wider range of patients for either the whole anaesthetic management or specic parts of a case. There are therefore more opportunities to delegate responsibility to a senior trainee than a junior trainee. Consultant participants in the study reported that they were better able to use the system with senior rather than junior trainees. There is some apparent logic in this nding. If we want to look at ANTS categories such as decision-making then it is necessary to let trainees make decisions, therefore you have to delegate that responsibility. This message was also conveyed when evaluating the cascade training. One might therefore imagine that the system could be used satisfactorily with senior trainees at least. However, frequently senior trainees are given responsibility not only for individual patients but for a complete surgical list, either intentionally to provide them with required clinical experience, or to cover for leave or sickness. Although a consultant anaesthetist will have supervisory responsibility for that trainee, they may be occupied elsewhere and unable to give continuous personal supervision. These factors limit the operating lists where a consultant and a senior trainee work together, and participants in the cascade training initiative listed this as another factor contributing to their difculties in using the ANTS system. In addition, often, no sooner had a trained consultant explained the system than the trainee was re-allocated to another theatre list. Given the various difculties which had been encountered with the cascade approach, it was important to consider whether there were further approaches which could be used in addition. As the clinical anaesthetists in the research team continued to use the system in their own practice, some other uses began to become apparent. It was possible to use the system as a framework on which to base both the NTS elements and also the more conventional technical tasks during anaesthesia. The reluctance of consultants to use the ANTS system with junior trainees was partly based on a concern that anaesthetists in the early stages of training have large areas of knowledge and practical skills to assimilate and struggle to cope with an additional load of considering NTS. However, if the teacher or instructor was using the ANTS system as a framework or mental model then he or she need not communicate that framework to the learner. The consultant as instructor could use the system to provide scaffolding for the trainee without this being visible to the trainee. Some examples will help to clarify this process. Let us begin with the category of ‘Situation Awareness’. One of the elements in the ANTS system category Situation Awareness is gathering information. A key technical skill to be mastered by novice anaesthetists is taking an anaesthetic history from a patient. This skill resembles the history taking skills that medical undergraduates have to master but there is sufcient difference to require specic additional instruction. When not familiar with the ANTS system, the trainee will Safer Surgery 182 view this skill as history taking, not as gathering information. Of course there is more to gathering information for situation awareness than history taking from the patient. Information about the intended surgical procedure is required to formulate a suitable anaesthetic plan. Trainees are traditionally taught to ask for the information about the proposed surgical procedure that they require, such as how long will it take; what position will the patient be in; does the surgeon need access to the abdominal, chest or pelvic cavities; what is the anticipated blood loss? This information may be gathered from a variety of sources such as the surgeons and the operating department nursing staff. The trainee is now exchanging information and coordinating activities with team members, which are elements from the category Team Working. As a further example, during the course of anaesthesia there will be episodes where the anaesthetic trainee needs to focus on surgical events, such as when the surgeon makes the skin incision. At this time the anaesthetist should be observing the response of the patient to conrm that the depth of anaesthesia is appropriate for that patient. The relevant elements are gathering information and anticipation from Situation Awareness, prioritizing from Task Management and exchanging information from Team Working. Prioritization is included because the anaesthetist should not be involved in other activities that are less essential (such as drawing up antibiotic drugs). The anaesthetist should anticipate the skin incision by observing the progress of the prepping of the surgical eld and the placing of sterile drapes around the surgical eld. They should be alert for an exchange of information (a surgeon with appropriate NTS will ask if it is all right to perform the incision and the anaesthetist should be in a position to reply). Taking this method further, the relevant NTS behaviours can be discussed during procedural skills training and the key role of NTS for safety can be highlighted. To illustrate this technique, consider the insertion of a central venous line. The trainee is taught to attach cardiac monitoring so that they can be aware of arrhythmia generation should the guide wire be passed too far. Of course this allows information gathering to provide more accurate Situation Awareness of how the technical procedure is proceeding. There can be discussion of where the monitor is best placed for information gathering. As this is sometimes found to be out of clear sight for the person inserting the cannula, the trainee may suggest that they would ask an assistant (exchanging information) to watch the monitor during the insertion (coordinating activities). It is not uncommon for trainee anaesthetists to assume that their assistants in theatre are familiar with all their activities. In this situation the trainee may assume that the assistant will understand both what to look for and when to look at the monitor. In fact this is often not so, and facilitating discussion with ANTS elements and categories claries the importance of the exchange of information and coordination of activities so that the trainee and his/her assistant have shared situation awareness and the safety of the procedure for the patient will be enhanced. At this time, in the earliest stages of postgraduate training, it is likely that most UK medical graduates will not be aware of the concept of NTS. The novice trainee may not see the behaviours being highlighted in the examples given above Non-Technical Skills and Anaesthetists’ Workplace-based Assessment Tools 183 as NTS. They may just be seen as behaviours to be copied and applied as part of the initiation into the profession of anaesthesia. However, where the instructor carries the framework and uses it to structure the components of practice, this can help direct the behaviours of the novice into good habits of practice. Then, when the trainee is introduced to the theoretical aspects of NTS, there are examples of clinical practice readily available to be used for illustrative purposes. Furthermore, when trainees have familiarity with the ANTS system, they can be asked to incorporate NTS when they articulate discussion of cases or events. To summarize, we can say that a trainer who is familiar with the ANTS system can use that system as a framework to help trainees acquire non-technical skills from the earliest stages of training. It will also have become obvious that there can be considerable overlap between anaesthetists’ technical and non-technical skills. A trainee anaesthetist can only anticipate what will happen during an operation if they have acquired some experience of that procedure. In a similar vein a trainee can only anticipate what impact a particular drug will have on a patient if that trainee has the necessary understanding of physiology and pharmacology. It may now become a little clearer why in the early days of applying the ANTS system the emphasis was on use with more experienced trainees. Future Directions So, where do we go from here? Let us review the position briey. The ANTS system can be used in the clinical workplace. The cascade model introduced by the Royal College of Anaesthetists to introduce the ANTS system as a tool to be used at departmental level has not been successful. Workplace-based assessment tools are now being introduced into postgraduate medical training in the UK. The proposed assessment tools have been adapted from existing tools and are being modied for purpose. There appears to be an overlap between technical and non-technical skills in anaesthesia. One possible strategy is to incorporate appropriate aspects of the ANTS system into the miniCEX, DOPS and CBD workplace-based assessment tools. As anaesthetists learn to use these tools in the workplace they can begin to include NTS in the manner outlined previously. In this way, we may assist consultant anaesthetists in becoming more familiar with the ANTS system. An experimental MiniCEX form (Figure 11.1 and 11.2) was designed in a collaborative venture between UK and New Zealand departments. Results from the UK limb are in the process of being collected and analysis is awaited. 1. 2. 3. 4. 5. . that the participants were encouraged to discuss their scores to allow calibration to take place. When the participants reached consensus and awarded very similar scores then that part of. risks and selecting options Re-evaluating Table 11.1 The ANTS system: categories and elements Safer Surgery 178 the ANTS system into the teaching and assessment of anaesthetists. The usability. incorporated into the teaching practice of a whole hospital department of anaesthesia? The proposed strategy for introduction to a department was to use a cascade system for training the trainers.