Safer Surgery 34 Which Procedure? In order to guarantee a sufciently wide range of assessment, each surgical specialty has selected a number of index procedures. These procedures are selected on the basis of their broad accessibility to trainees, observability and in most cases an aspect of the procedure which contributes something unique to the assessment range. In orthopaedics there is presently a collection of 14 index procedures (e.g., carpal tunnel decompression, total knee replacement, compression hip screw for intertrochanteric fracture neck of femur). A trainee may submit PBA assessments on any number of procedures but a successful example of all 14 must be included before the completion of training. By the end of training all the index procedures must be scored at the dened competence level of four. Naturally in the early years an intermediate score is inevitable for all or some domains. It is very important for both trainer and trainee to appreciate that it is progression towards competence which is being assessed primarily. Less than a score of four is to be expected early on in training, culminating in ‘straight fours’ towards the completion of training. PBA and the Curriculum It should be noted that PBAs are one element of a wider specialty curriculum. They are linked to the learning agreement and work in synergy with other tools that vary to some degree between specialties. Designing and Developing PBA Historically the roots of PBA go back in the authors’ experience to the early 1990s 1 when a desire to evaluate the change in performance before and after a fracture xation course lead to the development of a 20-item multisource feedback tool assessing performance in inserting a dynamic hip screw (DHS) into a fractured neck of femur. The potential of this approach went unrecognized until 2002 when the recommendations of the JCHST Competence Working Party (Rowley et al. 2002) made it possible to proceed with PBA development in orthopaedics, at which time it was referred to as a performance-based assessment. Parallel developments in other specialties led to the development of the Operative Competence (OPCOMP) tool by Jonathan Beard in vascular surgery (Thornton et al. 2003). In 2004, elements of both systems were integrated into what became the Procedure Based Assessment (PBA). In 2005 the tool was introduced to all surgical specialties through workshops conducted for the Specialist Advisory Committee Chairs (Pitts 1 The rst PBA was designed in 1994 as a follow up to a project investigating the change in competence following a training course (Oliver et al. 1997) but was not published until the report of the JCHST Competence working party (Rowley et al. 2002), when it was included as an appendix. Competence Evaluation in Orthopaedics 35 and Rowley 2005) and minor amendments made to the wording of elements and domains to make them accessible to the widest possible user group. Since this time they have been embedded in both the trauma and orthopaedic (T&O) curriculum (Pitts et al. 2007) and the Intercollegiate Surgical Curriculum Project (ISCP 2008). In the latter case some minor changes have been made but the instruments remain broadly identical. Design Considerations Features and characteristics of the surgical workplace, alongside the personality of the surgical team and requirements of assessment have inuenced the development of the PBA. Surgical environment The surgical environment is special and although many aspects of it may be simulated, there is at present no adequate simulation of the high stakes of a real operative procedure. In order to make a valid assessment of operative competence, the real world has to be used. This imposes considerable constraints on assessment, not the least of which is the central purpose of providing an overwhelmingly safe service to the patient. Each operation is unique. Not only do the physical circumstances of the operating environment vary but also the composition of the team, type of instruments in use (even for similar procedures) and most fundamentally the patients in whom there is a wide variation of largely similar anatomy and variation in the severity of disease. Nature of the surgical task The basic separation of surgical procedures into emergency and elective shows that some procedures are conducted on suitable patients who may be screened and selected for surgery by a variety of measures beforehand whereas others will arrive unscheduled with possibly life threatening conditions in a variety of states of ill health. The operating room is inevitably a stressful environment in which the formal assessment of trainees’ competence is of secondary concern. Characteristics of assessors (trainers) and trainees An ‘early years’ specialist surgical trainee is by no means a novice. She/he will have undergone at least ve years of medical school education followed by between two and ve years of postgraduate training before she/he enter specialty training. A senior surgical trainee, towards the end of training, will be a widely experienced practitioner who regularly operates on her/his own with accessible supervisors who are never the less outside the room in which the surgery is being conducted. The trainer (a consultant surgeon) is primarily responsible for the care of the patient and often for the leadership of a large team of professionals during the operative procedure. The introduction of a novel activity such as conducting a PBA is (rightly) questioned in Safer Surgery 36 order to ensure it does not compromise the primacy of provision of patient care. Scale of the community The orthopaedic community is one of the largest in surgery comprising over 40 per cent of practising surgeons. In the UK this involves approximately 3000 surgeons (including trainees) in over 450 hospital locations. This imposes considerable demands on the innovation process, not the least of which being to provide effective assessor training for the entire community Connecting to patients All patients who undergo surgery in a UK teaching hospital consent to part of their care being undertaken by trainees under supervision. PBAs form a part of that patient care process and as such should ideally be understandable by patients and their representatives. Curriculum requirements The same principles that have guided the development of the orthopaedic curriculum as a whole also guided the design of PBA. These principles were derived from a series of centre case studies. The list below is adapted from the Trauma & Orthopaedic curriculum (Pitts et al. 2007): A radical alternative – PBAs have been introduced into an environment where there were no established assessment tools and no foundations on which to build. They have been designed with the intention of gaining as much support from the orthopaedic community as possible in order to facilitate their implementation. Competence focused – There are debates about the nature or meaning of the word ‘competence’. One conceptual standpoint states that a competence is simply a demonstrable ability to do something, using directly observable performance as evidence. Another understands competence as being a holistic integration of understandings, abilities and professional judgments, where ‘competence’ is not necessarily directly observable, rather it is inferred from performance (Eraut 1994). The integration of these two aspects acknowledges a much greater level of complexity within surgical competencies and avoids the problem that individuals may well be able to demonstrate that they can ‘do’ something, but that does not necessarily mean that they understand what they are doing or why until they give evidence for it. Within our particular competence model we look not only for the three key domains i.e., knowledge, skills and attitudes, but also for the unique combination of those domains in areas such as professional judgement. The development of professional judgement is a key outcome of surgical training. Flexible and easy (intuitive) to use – PBAs have to t a variety of specialties, situations and personnel (see above). It is intended that their design will recognize this, whilst providing a consistency of standard and outcome. The hospital environment, where many trainers do not have their own ofce space and distractions abound, is hostile to nding time and space to • • • Competence Evaluation in Orthopaedics 37 meet and talk. Most surgeons join the profession to perform surgery. They acknowledge the need to train but appreciate the evaluation of training must be part and parcel of service delivery. With these factors in mind we have tried to keep PBA s straightforward and sympathetic to the paucity of time in rapidly changing settings to learn complex tools. Able to adapt to new developments (open architecture) – Many innovations, especially in social technology settings, have a lengthy gestation period. From the beginning every effort has been made to try to ensure that the PBA’s architecture is sufciently open to allow synergy with new developments and requirements. Driven by the trainee – The triggered nature of the PBA puts responsibility into the hands of those who hold largest stake in seeing training happen – the trainees. PBA require and enables the trainee to take the initiative and responsibility for her/his own training. Valid – Questions of validity (truth) may be addressed in several different ways. Does the implementation of the whole system make a valid improvement in the outcomes of training? Are the index procedures selected for assessments a valid choice? Is the internal structure of each assessment valid in terms of the measures of performance it proposes? A major problem in this area is the lack of previous measures of surgical competence. It is impossible to make comparison with anything other than examination results, which only measure a limited area of intellectual competence. Extensive efforts must be ongoing, within other constraints, to achieve detailed validation of index procedures and PBA. Reliable – PBA should be understood by all in the same way. Efforts have been made to link PBA closely to accepted practice so that a rm foundation of agreement can be laid for the future. Usable – The circumstances in which PBAs are used dictate that this area is of primary concern. ‘It might be valid and reliable but can you use it in a practical situation?’ Efforts have been made to ensure that PBAs can be used in real life contexts within the constraints of time, user skills and attitudes. Holistic in approach – It was clear from early observations that many problems encountered amongst trainees had their roots in the area of non- technical skills. Elements of the PBA address these skills (and highlight them for assessors as well as trainees). It is hoped that more elements of current non-technical advances will be incorporated into PBA in the future. Formative and summative – The notion of a summative assessment where a trainer (possibly external) observes a trainee’s performance in a pass/ fail scenario was rejected at an early stage after two pilot studies. On one hand there seemed to be insurmountable logistic and resource problems but more importantly, training in the workplace is an ongoing activity and assessment should resonate with its formative nature. It was decided that all • • • • • • • Safer Surgery 38 workplace assessments should be formative, giving feedback to the trainee to inform and guide her/his future performance. It was noted, however, that such assessments would, as a whole, be a useful summary of the trainee’s ability to learn and progress. The successful completion of a PBA is not seen as a licence to operate in that procedure but as a single component of a wider assessment of the trainee’s ability to learn operative procedures and perform them on a variety of patients with differing degrees of severity and complexity in their condition. Electronic application – If data are to be gathered from workplace-based assessments then it must have an electronic application which would facilitate this. Sadly the levels of IT ‘literacy’ encountered in pilots trials were highly variable and, more importantly, access to IT resources in NHS Trusts is extremely patchy. PBAs have been developed in a paper-based format whilst maintaining the possibility of an easy transfer to a digital system. Selection of a Rating Scale In the 1994 PBA, it was envisaged that the rater/assessor could be a scrub nurse, senior colleague or peer. The rating of any element was made on the basis of how much evidence there was for the judgment. For example, one element of the instrument asked about skin preparation, with three options: ‘Was it prepared aseptically/dry prior to draping procedure/ensure no pooling of antiseptic solutions below patient?’ (NB: the early version posed the questions in a very different way.) The available scores were: 1 = no evidence whatsoever that the stage/task/activity has been completed 2 = some evidence 3 = ample evidence This approach was taken because we were uncertain at that time whether such observations were possible and in particular, we wanted to compare the scores from professionals with differing interests (e.g., nurses and surgeons) and how much impact a training event had on the trainee’s behaviour in theatre. For the early versions of the later PBAs, we chose a similarly simple scale but from a different assessment viewpoint. By this time we were not trying to measure the impact of training, we were attempting to capture a snapshot of the trainee’s behaviour in order to assess competence. The rating scale chosen for this was: 0 = not assessed 1 = unsatisfactory 2 = satisfactory • Competence Evaluation in Orthopaedics 39 Numbers were chosen initially with a view to producing an electronic version later. We considered the use of a Likert scale and there was considerable debate as to whether this would be benecial in demonstrating degrees of progress that would have a motivational effect. We also considered the inclusion of an extra column that could be marked if a trainee showed excellence at particular points (star quality) but eventually concluded that the simplest rating options would be the most effective. A number of factors inuenced the choice of the simple scale. The rst was that we needed to cater for the possibility that not all items would be assessed. There could be no guarantee that the trainee would be able to complete the whole procedure for a variety of reasons and to complete part of the assessment would be of great benet to more junior trainees (mirroring actual training practice). Secondly, it was never considered feasible, given the numbers of assessments involved and the variety of locations, that an independent assessor would be present in theatre. Even if they were, their independence would prevent them entering the sterile area and so limit their observations. The consequence of this was that the detail of the observation would only be recorded by the assessor at the end of the procedure. The more detailed the rating scale, the more likely the assessor might be to enter an incorrect score, having remembered the performance inadequately. Thirdly, the naturally competitive personality of surgical trainees suggests that there could be lengthy debates about whether their performance should score two or three on a larger scale and this would introduce an unwanted variable (trainer personality) into the assessment process. The nal change to the rating scale came after a meeting in which the PBA was discussed by individuals (surgeons, educators and administrators) who had not been part of the original design group. One person in particular found it difcult to grasp the nominative nature of the scores and insisted on trying to calculate a minimum average score for the PBA. To avoid such problems recurring, the scale was altered to: N = not assessed U = unsatisfactory S = satisfactory Since the acceptance of PBA by all specialties, some have insisted on changing the scale from unsatisfactory to requiring further development. The authors see no advantage in this and some potential problems including the danger of increasing uncertainty through lack of denition. The inclusion of the global assessment at the end of the PBA was one of the elements acquired from the merger with the OpComp tool. The inclusion of this domain enables a qualitative triangulation of the other domains which has proved extremely benecial for the reasons of adding an element of overall professional judgement as described above. Safer Surgery 40 Validity and Reliability of PBAs The power of the PBA assessment rests in part on the fact that the PBA assesses the same competencies in a variety of procedures with a broad range of suitably qualied assessors. An orthopaedic trainee will normally have at least eight trainers, in a series of six-month attachments, during her/his training. In addition, she/he will operate in emergency situations, through rostering, with an even wider set of trainers, all of whom may act as assessors for a PBA. Internal Validity of PBA The initial selection of PBA domains and elements came from two sources. One was the original 20-element tool (Pitts and Ross 1994) the other was a series of Delphic groups involving surgeons within the orthopaedic community selected for their expertise as both trainers and surgeons. At a later stage the PBA which related to specic procedures was reviewed by a further series of individuals and groups. These revisions were to establish that in a particular procedure all elements were easily observable in a particular procedure and so that examples of positive and negative descriptors, as well as negative-passive indicators (sins of omission) could be identied. As a result, all PBAs have been validated against a standard worksheet of these descriptors for every element of every domain, an extract from which is shown as Table 3.4. The worksheet offers the opportunity to articulate specic examples (in italics) of generic competences. Validity of Index Procedures Whilst the initial selection of index procedures was made by a small group, its work was corroborated using a further set of groups consisting of 50+ surgical trainers in all. In this exercise the trainers were required to produce lists of index procedures (to the agreed criteria) on which they had achieved consensus. After the outliers were removed from the group lists, a high degree of correlation was seen with the earlier Delphic group selection. A further triangulation of the selection of index procedures was made using the orthopaedic electronic logbook to check that all selected procedures were accessible to trainees in sufcient numbers (Pitts et al. 2005). A nal review of the procedures’ list was made using a further group of surgeons, during a south east training conference, who reviewed the list from the point of view of procedures that they felt they would, in their practice, be able to use to assess trainees Reliability Establishing the inter-rater reliability of the PBA tools proved extremely difcult within the time and budgetary constraints of the PBA Orthopaedic Competence Assessment Project (OCAP) project. An early attempt at producing video material Competence Evaluation in Orthopaedics 41 Competences and definitions Positive behaviours (doing what should be done) N egative behaviours (doing what shouldn’t be done) Negative – passive behaviours (not doing what should be done) Pre-operative planning Demonstrates recognition of anatomical and pathological abnormalities and operative strategy to deal with these A rticulates the realistic clinical ndings against any investigative ndings and achieves a balance between the two Describes an operative plan without the full use of the clinical and investigative material Fails to take into account specic medical conditions that might limit the technical choices Ability to make reasoned choice of appropriate equipment, materials or devices (if any) taking into account appropriate investigations e.g., x-rays I s able to draw, write or iterate a preoperative plan Does not take into account investigative ndings when planning or selecting the equipment Fails to check the notes for relevant or unexpected ndings T akes the x-ray and any templates and plans the operation on paper checking both AP and lateral Does not consult the x-ray at all. Makes all the decisions on the AP x-ray Fails to check both AP and lateral x-rays and makes all the decisions on the AP x-ray Checks materials, equipment and device requirements with operating room staff E ither personally visits or rings up the operating theatre to check on equipment availability Delegates the task to a more junior team member with no plans to check the instruction has been carried out F ails to communicate with the theatre staff Where applicable ensures the operation site is marked Personally marks the site Delegates the task of marking the site to a junior doctor or nurse Fails to check that the site has been marked Checks patient records E nsures that the relevant information such as investigative ndings are present During the procedure asks theatre staff to look something up in the notes Fails to check notes to ensure all information is available that is needed Table 3.4 Validation worksheet example taken from T&O curriculum (Pitts et al. 2007) Safer Surgery 42 for viewing by raters was abandoned due to the difculty of obtaining sufciently high quality footage of a lengthy procedure and persuading sufcient numbers of surgical trainers to spend time scoring it. Fortunately this area has now been revisited by a team at Shefeld (Beard, Purdie et al. – see Chapter 4 in this volume). Innovation and Acceptability The positioning of the PBA tool has, from its inception, been as a device ‘designed by surgeons for surgeons’. The orthopaedic curriculum (OCAP) steering group has had some 22 members in its approximately six-year lifespan with all but one being practising surgeons. This has resulted in a high degree of face validity. We have further supplemented this with a number of audits in various aspects of the PBA (and curriculum) acceptance and adoption by the orthopaedic community and this is described below. It has as yet not been possible to replicate this work in other specialties. Baseline Survey Prior to the launch of the curriculum materials into the orthopaedic trainee population in 2005 a small survey was conducted of trainee activity using trainees attending the annual British Orthopaedic Association (BOA) congress. Amongst other results, the survey found the following: 10 per cent of respondents had no meeting with their trainer outside the operating theatre in their entire six month attachment; 40 per cent of respondents had no written aims or objectives (learning agreement) for their attachment; 55 per cent had no formal assessment of their operating skills during their attachment; although the results tted the expected picture the number of respondents was small (50) but it provided a baseline against which future progress might be measured. Acceptance Survey In the process of introducing the PBA and other curriculum tools, a number of brieng meetings were held across the UK, with varying numbers attending. At each of these meetings a survey was issued with questions relating to different tools, including the PBA. Two questions were posed: 1. Is this a good idea? and 2. Will it work? Whilst some doubts were expressed as to whether trainers would comply with the new system (or have time to do so) respondents clearly expressed the view • • • • Competence Evaluation in Orthopaedics 43 that it was a good idea and, to a lesser extent, that it would work, although all the outcomes tended to the positive. In addition to gathering broad response, the questionnaires highlighted areas of expected difculty, many of which have proven to be valid. RITA Questionnaire The Regional In service Training Assessment (RITA) has been an annual or bi- annual event for UK surgical trainees. It is in the process of being replaced by the Annual Review of Competence Progression (ARCP). In October 2005, following the launch of OCAP in August of that year, a questionnaire was issued to all trainees and programme directors to be completed before the RITA. The questionnaire asked factual questions about how many PBAs had been conducted, who triggered them and if none or few had been conducted, what the reasons were. The primary purpose of this tool was to nd out what was happening in the eld. The secondary purpose was to send a clear message that the Specialist Advisory Committee (SAC) was taking note of progress and would (and did) investigate instances of non compliance in a low key way. The results have been invaluable in identifying areas where engagement has been weak and further intervention is necessary. Subsequently, an internet survey has been conducted since 2006, annually open to all T&O trainees contacted via their electronic logbook. In January 2006 only 50 per cent of trainees had completed one or more PBA assessments but this has risen to 93 per cent by January 2008 (Boardman et al. 2008). The work will be submitted for publication shortly as a longitudinal audit study. Latest Developments PBA assessment tools are now embedded in all surgical curricula. Their development continues in a number of areas; particularly in orthopaedics but also in other specialties. Later Years of Training Orthopaedic trainees often specialise further in the later years of training preparing for a career in a sub-specialty such as spine, joint replacement, hand surgery etc. Debate is continuing as to whether there should be the same PBA assessment conducted on more difcult and specialised procedures or whether an ‘advanced’ PBA should be designed that would assess higher order surgical competencies. OCAP Online The online version of the orthopaedic curriculum (OCAP Online) was launched in August 2008. Details can be found on the website: <www.ocap.org.uk>. The . questioned in Safer Surgery 36 order to ensure it does not compromise the primacy of provision of patient care. Scale of the community The orthopaedic community is one of the largest in surgery comprising. patients All patients who undergo surgery in a UK teaching hospital consent to part of their care being undertaken by trainees under supervision. PBAs form a part of that patient care process. in Orthopaedics 37 meet and talk. Most surgeons join the profession to perform surgery. They acknowledge the need to train but appreciate the evaluation of training must be part and parcel of