Safer Surgery 24 Flin, R., Yule, S., Paterson-Brown, S., Maran, N., Rowley, D.R. and Youngson, G.G. (2007) Teaching surgeons about non-technical skills. The Surgeon 5(2), 86–9. Frank, J.R. (ed.) (2005) The CanMEDS 2005 Physician Competency Framework. Better Standards. Better Physicians. Better Care. Ottawa: The Royal College of Physicians and Surgeons of Canada. Gawande, A.A., Zinner, M.J., Studdert, D.M. and Brennan, T.A. (2003) Analysis of errors reported by surgeons at three teaching hospitals. Surgery 133, 614–21. Giddings, A.E.B. and Williamson, C. (2007) The Leadership and Management of Surgical Teams. London: The Royal College of Surgeons of England. Glaser, B.G. and Strauss, A.L. (1967) The Discovery of Grounded Theory. Chicago: Aldine. GMC (2001) Good Medical Practice. London: General Medical Council. Gordon, S.E. (1993) Systematic Training Programme Design: Maximizing Effectiveness and Minimizing Liability. Englewood Cliffs, NJ: Prentice Hall. Hall, J.C., Ellis, C. and Hamdorf, J. (2002) Surgeons and cognitive processes. British Journal of Surgery 90, 10–16. Helmreich, R. and Davies, J. (1996) Human factors in the operating room: Interpersonal determinants of safety, efciency and morale. In A. Aikenhead (ed.) Balliere’s Clinical Anaesthesiology 10(2), 277–95. Helmreich, R. and Schaefer, H. (1994) Team performance in the operating room. In M. Bogner (ed.) Human Error in Medicine. Hillsdale, NJ: LEA, 225–53. Helmreich, R., Sexton, B. and Merritt, A. (1997) The Operating Room Management Attitudes Questionnaire (ORMAQ). University of Texas Aerospace Crew Research Project Technical Report 97-6. Austin, TX: The University of Texas. Hoffmann, R., Crandall, B., and Shadbolt, N. (1998) A case study in cognitive task analysis methodology: The Critical Decision Method for the elicitation of expert knowledge. Human Factors 40, 254–76. Jacklin, R., Sevdalis, N., Darzi, A. and Vincent, C. (2008) Mapping surgical practice decision making: An interview study to evaluate decisions in surgical care. The American Journal of Surgery, 195, 689–96. Kneebone, R. and Darzi, A. (2005) New professional roles in surgery. British Medical Journal 330, 803–804. Klampfer, B., Flin, R., Helmreich, R.L., Hausler, R., Sexton, B., Fletcher, G., et al. (2001) Group Interaction in High Risk Environments: Enhancing Performance in High Risk Environments, Recommendations for the use of Behavioural Markers. Berlin: GIHRE. Available on <www.abdn.ac.uk/iprc> [last accessed November 2008]. Mishra, K., Catchpole, K., Dale, T. and McCulloch, P. (2008) The inuence of non- technical performance on technical outcome in laparoscopic cholecystectomy. Surgical Endoscopy 22, 68–73. Development and Evaluation of the NOTSS Behaviour Rating System 25 Perrow, C. (1999) Normal Accidents: Living with High-risk Technologies. Princeton, NJ: Princeton University Press. Reason, J.T. (1997) Managing the Risks of Organizational Accidents. Aldershot: Ashgate. SASM (2003) Scottish Audit of Surgical Mortality Annual Report – 2001 data. Glasgow: SASM. Seamster, T., Redding, R. and Kaempf, G. (1997) Applied Cognitive Task Analysis in Aviation. Aldershot: Avebury. Sevdalis, N., Healey, A.N. and Vincent, C.A. (2007) Distracting communications in the operating theatre. Journal of Evaluation in Clinical Practice 13, 390–4 Sevdalis, N., Davis, R., Koutantji, M., Undre, S., Darzi, A. and Vincent, C.A. (2008) Reliability of a revised NOTECHS scale for use in surgical teams. The American Journal of Surgery 196, 184–90. Sexton, B., Thomas, E. and Helmreich, R. (2000) Error, stress, and teamwork in medicine and aviation: Cross sectional surveys. British Medical Journal 320, 745–9. Studdert, D.M., Mello, M.M., Gawande, A.A., et al. (2006) Claims, errors, and compensation payments in medical malpractice litigation. The New England Journal of Medicine 354, 2024–33. Taylor-Adams, S. and Vincent, C. (2004) Systems analysis of clinical incidents: The London protocol. Clinical Risk 10, 211–20. Undre, S., Sevdalis, N., Healey, A.N., Darzi, A. and Vincent, C. (2007) Observational Teamwork Assessment for Surgery (OTAS): Renement and application in urological surgery. World Journal of Surgery 31, 1373–81. Vincent, C.A., Neale, G. and Woloshynowych, M. (2001) Adverse events in British hospitals: Preliminary restrospective record review. British Journal of Medicine 322, 517–19. Youngson, G.G. (2000) Surgical Competence: Acquisition, Measurement, and Retention. Edinburgh: The Royal College of Surgeons of Edinburgh. Yule, S., Flin, R., Paterson-Brown, S. and Maran, N. (2006a) Non-technical skills for surgeons: A review of the literature. Surgery 139, 140–9. Yule, S., Flin, R., Paterson-Brown, S., Maran, N. and Rowley, D. (2006b) Development of a rating system for surgeons’ non-technical skills. Medical Education 40, 1098–104. Yule, S., Flin, R., Maran, N., Rowley, D. R., Youngson, G.G. and Paterson-Brown, S. (2008a) Surgeons’ non-technical skills in the operating room: Reliability testing of the NOTSS behaviour rating system. World Journal of Surgery 32, 548–56. Yule, S., Flin, R., Rowley, D., Mitchell, A., Youngson, G.G., Maran, N. and Paterson-Brown, S. (2008b) Debrieng surgical trainees on non-technical skills (NOTSS). Cognition, Technology & Work 10, 265–74. Yule, S., Rowley, D., Flin, R., Maran, N., Youngson, G.G., Duncan, J. and Paterson- Brown, S. (2009) Experience matters: Comparing novice and expert ratings of non-technical skills using the NOTSS system. Australian Journal of Surgery 79, 154–160. This page has been left blank intentionally Chapter 3 Competence Evaluation in Orthopaedics – A ‘Bottom-up’ Approach David Pitts and David Rowley Introduction The design and implementation of what we now know as Procedure Based Assessments (PBAs) began in the UK in the early 1990s. In 2008, PBAs are in use in all UK surgical specialties, embedded in all surgical curricula as the primary tool for evaluating perioperative competence in the middle and later years of surgical training. The motivation driving their development has been practical problem solving. In this respect their development has much in common with other ‘need pull’ innovations (Langrish et al. 1972) in that their wider foundations can only be seen retrospectively and although they have much in common with other surgical assessments, their early development occurred completely independently. PBAs have been developed and introduced against a backdrop of transition in surgical training. Their development has involved not only the design of an assessment tool but also the battle to gain acceptance of the concept and practice of overt competence evaluation in the surgical workplace. This chapter describes the evolution of PBAs from instigation to practical usage and describes ongoing evaluation of the outcome in terms of the instrument and its use. Surgical Training in Transition Since the early 1990s UK surgical training has been in a state of constant transition. Not only have the regulations governing training changed radically but the political, social and healthcare environments in which training occurs have swung between extremes. A review of some of these changes will show why gaining acceptance by the surgical community for the use of a competence assessment tool such as the PBA has been so vital. Changes in Structure and Regulation Until the publication of the Calman Report (Department of Health 1993), surgical training in the UK involved a lengthy apprenticeship punctuated by knowledge tests but without any assessment of practical skills and no formal requirement to Safer Surgery 28 address non-technical areas such as communication or teamwork. Although the Calman reforms introduced some degree of structure, it was not until the Richards Report of 1997 (Richards 1997) and the subsequent report of the Competence Working Party of the Joint Committee for Higher Surgical Training (JCHST) in 2001 (Rowley et al. 2002) that assessments of practical ability or competence were openly recommended. Royal Colleges should give serious consideration to establishing innovative procedures, other than written exit examinations, to assess clinical competence of candidates for the award of a certicate of Completion of Specialist Training. (Richards 1997) I t is essential that trainers and trainees extend their assessment of operative and clinical performance. Speciality Advisory Committees (SACs) in surgery should determine which operations should occur and to what extent, and what level of operative ability is required for a given stage of training. Simply recording a minimum number of operations is insufcient – the quality of the training experience is more important than the number of experiences. (Rowley et al. 2002: 21) Following the publication of Unnished Business (Donaldson 2002), a report on the current state of training, further reforms were introduced and the ‘Modernizing Medical Careers’ project coincided with the inception of the Postgraduate Medical Education and Training Board (PMETB) in 2003 which insisted on the introduction of comprehensive curricula for each specialty and principles established whereby regular assessment of practical skills was encouraged. PMETB’s key task has been to establish standards dening medical education, training and assessment and to assure these standards (including competence based curricula) through external management of quality. The Trauma and Orthopaedics surgical curriculum (the rst time such a document has been produced in the specialty in the UK), in which competence- based training and assessment were enshrined, was approved by PMETB in September 2006 (Pitts et al. 2007). PBAs have been introduced against this changing structural backdrop. Changes in Public Attitude There is no doubt that the public attitude towards medicine in general, and to surgery in particular, has changed. This change was most notably precipitated by the Bristol (Kennedy 2001) and Shipman (Smith 2005) inquiries into high death rates in paediatric surgery and general practice respectively. High mortality rates in the Bristol Paediatric Cardiac Unit resulted in action from the Department of Health in 1994 and the suspension of operating in that unit in 1995. The subsequent inquiry’s report into that unit (Kennedy 2001) coincided with the very public Competence Evaluation in Orthopaedics 29 trial and eventual incarceration of Harold Shipman, a general practitioner, for actions resulting in the deaths of a number of his patients. The Shipman Inquiry, reporting from 2002–2005 (Smith 2005), revealed serious shortfalls in processes and procedures surrounding the use of controlled drugs, certication of death and the monitoring of clinical performance stretching back, in Shipman’s case, to his time as a medical trainee. The Donaldson White Paper in 2007, for new revalidation processes in the UK for clinicians and other medical professionals (Department of Health 2007), has been one of the longer-term outcomes of the Shipman Inquiry which will undoubtedly culminate in the use of PBAs or similar tools in the revalidation process. Changes in Time Available for Training The European Working Time Directive introduced in 1998 reduced the number of hours a trainee might stay in the workplace to 58 in 2004 and are likely to reduce those hours further, to 48 in 2009. There have undoubtedly been benets from this directive but it has signicantly reduced the access to surgical experience for trainees, particularly with respect to unusual trauma cases arriving out of normal working hours. Changes in Service Delivery Recent years have seen the growth of Independent Sector Treatment Centres (ISTC). Such centres, normally operating outwith the control of NHS local management, have been used to reduce waiting lists, particularly for common surgical procedures conducted on anaesthetically less challenging patients. This has further reduced the access to routine surgical experience, particularly for more junior trainees. PBAs have been developed against this background of sudden and discontinuous change with reduced access to surgical experience necessitating the introduction of training tools that help to derive maximum benet from the time available. Facilitating positive change in such circumstances is (and always has been) difcult. The innovator has for enemies those who did well under the old system and only faint friends in those who might do well under the new (Machiavelli 1515, Chapter VI) What is a PBA? A PBA is a collection of behavioural markers (elements) for observing activities around a surgical operation set in seven domains covering the whole of a surgical procedure from consent to post operative management. Safer Surgery 30 A PBA is a formal, structured assessment of a trainee’s competence in performing surgery. An individual PBA provides a formative assessment to the trainee and evidence for the trainer on which to base their future input and level of supervision. A collection of PBAs (assembled over several years, conducted by a variety of trainers) provides summative evidence of the trainee’s progress and competence in learning surgical procedures and techniques, performing them to the required protocol and quality. A PBA happens in real time, in a real operating theatre with a live patient. It is normally undertaken, without pressure, between a trainee and their trainer (with whom a relationship is already established) surrounded by an operating team who will not take any unusual measures to support the trainee. A PBA will not normally be conducted on the rst occasion a trainer and trainee operate together. It is normally conducted on a procedure with which the trainee is already familiar. There is no limit to the number of times a trainee may attempt a particular PBA so there is no pressure to succeed on a particular occasion. All of these conditions help the trainee to give a ‘normal’ performance and, more importantly, protect the patient. PBAs in Practice – Applying the Seven Domains Within each domain there are a number of related yet unique elements which identify activities which must be performed successfully in order to achieve a ‘satisfactory’ score. Most elements are identical across all procedures but in some domains there is opportunity for procedure-specic items which identify the trainee’s grasp of the unique aspects of particular surgical procedures. Table 3.2 illustrates both generic and specic items. Although supercially the structure of a PBA resembles a two-page checklist (see Figure 3.1) a PBA is not a schedule of how to perform the procedure, rather it identies places in the procedure where competence is observable. In the same 1. Consent 2. Pre-operative planning 3. Pre-operative preparation 4. E xposure and closure 5. Intra-operative technique 6. Post-operative management 7. Global summary Table 3.1 PBA domains Competence Evaluation in Orthopaedics 31 Competencies and definitions Score N/U /S Comments Intra-operative technique IT1 Follows an agreed, logical sequence or protocol for the procedure IT2 Consistently handles tissue well with minimal damage I T3 Controls bleeding promptly by an appropriate method IT4 Demonstrates a sound technique of knots and sutures/ staples IT5 Uses instruments appropriately and safely I T6 Proceeds at appropriate pace with economy of movement IT7 Anticipates and responds appropriately to variation e.g. anatomy IT8 Deals calmly and effectively with untoward events/ complications IT9 Uses assistant(s) to the best advantage at all times IT10 Communicates clearly and consistently with the scrub team IT11 C ommunicates clearly and consistently with the anaesthetist IT12 Dislocates hip safely IT13 Cuts femoral neck appropriately to match design of implant IT14 Demonstrates familiarity and understanding of acetabular preparation including osteophyte trimming medially and at rim IT15 Broaches the femur properly and prepares the bony surface IT16 Uses trials and checks component orientation properly IT17 Fix acetabular component appropriately I T18 Implants femoral component appropriately IT19 Performs nal reduction and checks for stability Table 3.2 Example elements for total hip replacement PBA, taken from T&O curriculum (Pitts et al. 2007) Safer Surgery 32 Figure 3.1 Total hip replacement PBA T&O curriculum (Pitts et al. 2007) Competence Evaluation in Orthopaedics 33 way that a driving examiner looks for key behaviours (mirrors, signal, manoeuvre) the assessor is guided by the PBA to key performance points in the procedure. Both the trainer and trainee may trigger a PBA. It is normally conducted with the trainer scrubbed (able to observe trainee’s actions closely). The trainee conducts the agreed sections of the procedure taking care to verbalize their intentions (in order to not only enable more effective assessment but also to avoid any compromise in the quality of patient care). At any point, the trainer may step in and perform all or some remaining sections of the procedure, if there is the slightest risk that the trainee will provide less than optimal care. After the surgery is complete the trainee and trainer review the PBA form and complete it. Each element of relevant domains assessed is scored as satisfactory or unsatisfactory according to whether there is sufcient evidence from the trainer’s observation that the required standard was met. The nal domain of the PBA is the global assessment (see Table 3.3). The global assessment gives the trainer the opportunity to comment on the trainee’s overall performance. Even though the individual elements may have been performed to a satisfactory nished quality, the trainer is still able to apply an overall expert judgement. For example, the trainee may have been slow or hesitant or struggled to deal with an unexpected complication. The results of the PBA are transferred to a PBA summary sheet where they are seen alongside results from other PBA assessments. This document’s key function is to demonstrate clearly, to the annual review panel, whether the trainee is making progress, to indicate if certain areas of competence require further attention or highlight whether there are serious causes for concern. Level at which completed elements of the PBA were performed Tick as appropriate Comments L evel 0 Insufcient evidence observed to support a judgement Level 1 U nable to perform the procedure under supervision Level 2 Able to perform the procedure under supervision Level 3 Able to perform the procedure with minimum supervision (would need occasional help) Level 4 Competent to perform the procedure unsupervised (could deal with complications) Table 3.3 Global assessment taken from T&O curriculum (Pitts et al. 2007) . evaluate decisions in surgical care. The American Journal of Surgery, 195, 68 9– 96. Kneebone, R. and Darzi, A. (2005) New professional roles in surgery. British Medical Journal 330, 803–804. Klampfer,. Hamdorf, J. (2002) Surgeons and cognitive processes. British Journal of Surgery 90, 10– 16. Helmreich, R. and Davies, J. (19 96) Human factors in the operating room: Interpersonal determinants of. Safer Surgery 24 Flin, R., Yule, S., Paterson-Brown, S., Maran, N., Rowley, D.R. and Youngson, G.G. (2007) Teaching surgeons about non-technical skills. The Surgeon 5(2), 86 9. Frank,