Safer Surgery 54 Our study views feasibility as a key assessment characteristic and it is formally considered within the study’s design. Following each case, the supervising consultant completes a PBA or OSATS before giving feedback to the trainee. We observe this process and record the time taken to complete the assessment form and the duration of feedback. The follow-up questionnaires which we distribute to trainees and assessors approximately one month after completing assessments, in the relevant specialty, include questions to address feasibility issues, e.g., time added to operating list, available room for feedback, ease of use of tools. We have not directly addressed cost effectiveness in this study, although the data could inform future research. Are the Assessment Tools Acceptable to Stakeholders? User acceptability is the extent to which an assessment tool or method is accepted by the stakeholders involved in the assessment. It is a crucial factor in the design and successful implementation of assessment programmes. The acceptability of the assessment tools to trainees and assessors is being evaluated during the study’s implementation, as the future direction of competency- based assessment in the UK will be inuenced by the opinions of surgical trainees and trainers. After each feedback session and in the follow-up questionnaires we address the acceptability of the assessments and the assessment tools. There is some overlap of user acceptability issues with feasibility and validity. For example, an assessment tool which is very complex and time-consuming to complete is likely to have low face validity and low feasibility as well as having inferior acceptability. However, we consider acceptability from the overall perspective of the people directly involved in the assessments. We ask supervising consultants and trainees to rate their overall satisfaction with the assessment tools immediately after feedback. The subsequent questionnaires provide an opportunity for trainees and assessors to express their views on the ease or difculty of using the tools, their perspectives on the value of assessment and feedback and the impact of assessment on training and patient safety. Thematic Analysis of the Problem-solving Approach to the Study Implementation We have used a number of themes below to illustrate the problem-solving approaches we have adopted during the study implementation. We have identied lessons learnt and made some practical suggestions which may be of use to those undertaking similar work. 1. Matching the Research Team to the Study Design We have assembled a research team which has the skills required to evaluate assessment in the operating room. The team includes surgeons, a research Surgical Skills and Non-Technical Behaviours in the Operating Room 55 coordinator with surgical experience and a psychometrician whose particular area of expertise is workplace-based assessment and generalizability theory. All three independent assessors are practising surgeons with expertise in many of the surgical specialities, having received training in assessment and feedback through the ‘Training the Trainers’ course at the Royal College of Surgeons, and NOTSS training facilitated by the NOTSS research team, in Edinburgh. Lessons learnt and suggested learning points Whilst it is impossible to outline all the skills and attributes required from a research team in this eld, we consider the following to be essential: expertise in surgical knowledge, skills, attributes and competence; familiarity and condence with working in the operating room environment; rm research governance knowledge and ‘good clinical practice’ training; statistical expertise independent from the grass roots researchers; diplomacy in negotiating the socio-political surgical frameworks; tenacity towards recruitment of cases and engagement of trainees and trainers; exibility towards workload and research schedule; consistent communication and organization between team members to use the team’s resources to their full potential. Our aim was to bring together a multidisciplinary team of researchers with a mix of skills and attributes to complement the design and implementation of our study. 2. Engaging and Informing Clinicians We are carrying out research in real time in the workplace. However, surgical environments are both busy and rapidly changing. One of the challenges we faced in implementing the study was ensuring that we used a timely and appropriate method to inform the surgical teams involved. We used the following methods of communication, often in combination, to disseminate the purpose and design of the research study: email information packs; written information packs; presentations at departmental meetings; face-to-face discussions. Our main aim in the advance communication was to familiarize staff with the study before we moved the research into their specialty. This approach recognized that research and assessment in the operating room could be viewed by staff as • • • • • • • • • • • • Safer Surgery 56 threatening or unnecessary, unless we clearly explained the aims of the study within the context of work-place based assessment and the surgical curricula. The information packs include an overview of the study and examples of the assessment tools with guidance on their use for assessment and feedback. As the study has rolled out, we have recognized the need to support email communication with meetings, written information and face-to-face discussion. We have been able to use formal meetings at times but this has often been constrained by practical considerations, for example the size, organizational structure and availability of the target audience. Presentations at specialty meetings for surgeons and surgical trainees have proved useful in some specialties but impractical in others. It has not been feasible to organize formal presentations within work time for scrub team and anaesthetic staff, which we have managed by arranging smaller ad hoc meetings. Overall we have found that trainee and assessor engagement has been best achieved using face-to-face discussions in the eld with supporting written information, having provided a background to the study by email communication and presentations where possible. Lessons Learnt and Suggested Learning Points Identify the most effective way to communicate and disseminate your research to all interested parties. Consider your resources and the feasibility of your approach to communicate and disseminate the study. Be exible and revise your approach to overcome time constraints and structural barriers in the workplace. 3. Training of Assessors Further to communicating and disseminating the study methodology, there was the need for us to familiarize and train staff involved in assessing and/or giving feedback. We experienced wide variations in staff engagement and attitude towards the research, which challenged our ability to provide consistent training for all staff. The breadth of the personnel involved in the study and the feasibility of providing training within work time were also signicant barriers to the study implementation. Staff for whom training was required included: consultant surgeons: PBA and OSATS assessment tool and feedback training; consultant/senior anaesthetists: NOTSS training; scrub nurses: NOTSS training. We have made great efforts to train all assessors to ensure condence in and credibility of the assessor ratings. We have provided all assessors with information • • • • • • Surgical Skills and Non-Technical Behaviours in the Operating Room 57 packs, including the relevant assessment tools, guidance on their use and access to web-based training which we have usually supported with one to one familiarization before they have undertaken any assessments. We acknowledge that we have been unable to achieve an entirely level playing eld for training assessors in the use of workplace assessment tools, which reects the reality of the surgical workplace. Consultant assessors differ in their educational interest and awareness, as well as their uptake of assessment training, including web-based training and the Training the Trainers course. However, the study design includes several ‘quality measures’ of training. Supervising consultants are asked at the time of giving feedback to trainees and in a subsequent questionnaire what type of training they have received in assessment. This will be incorporated in the data analysis. Furthermore, our data analysis using generalizability theory will enable an examination of the variance in ratings for different assessor designations, for example a comparison of NOTSS ratings between independent assessors, anaesthetists and scrub nurses. Despite training all assessors in the appropriate use of the tools, we have observed some inconsistencies in the way the tools are used: prompting trainees too readily; being unable to allow trainees to lead the case within their level of competence, taking over decision-making, or the surgical instruments; directing trainees to operate using their preferred surgical sequence and/or technique; being reluctant to score competencies negatively (and/or give difcult feedback), particularly for senior trainees. We recognize that these training styles may inuence ratings of trainee skills and behaviours. For example, if a trainee is directed to operate using a different technique, they are unlikely to be as smooth in its delivery, affecting PBA/OSATS scores. We found it very difcult to rate skills and behaviours where a trainer repeatedly intervened and in some cases we used ‘not applicable’ to rate these skills and behaviours. The most successful assessments and training opportunities are during cases where the consultant trainer permits the trainee to operate within their limits of competence, and grants them the leadership to carry this out, prompting or intervening only when required or requested. Supervisor training to this level is beyond the scope of our study, but is what will be required of trainers in the future if training is to become more effective. Lessons Learnt and Suggested Learning Points Providing training in the workplace requires exibility and tenacity to ensure full coverage. Identify the most effective and suitable method for training different assessors. • • • • • • Safer Surgery 58 Consider consistency in the training approach you adopt, taking into account differences in staff engagement and attitude. Be prepared to abandon assessments or preclude assessors if an authentic assessment is compromised. 4. Consenting and Recruitment of Patient Participants To successfully recruit a surgical case requires the favourable alignment and accurate timing of many factors which are beyond our control. These factors are often not conrmed until the day of the operation and the case cannot go ahead as part of the study if any one of them is missing: an appropriate case; a consenting patient; a surgical/HDU bed; a suitable and consenting surgical trainee; an available consultant for providing supervision, assessment and feedback; an independent assessor; theatre staff for the list; sufcient operating time for the case; a suitable training list (some lists are for service provision only). Furthermore, the ethical requirement of the study is that patients receive a Patient Information Sheet 24 hours before they are approached for consent to give them time to consider. This requires us to provide patients with an Information Sheet before their admission, with the corollary that we need to identify suitable cases for recruitment at least three working days ahead to allow for postage time. Our selection of cases in advance is affected by a number of other last-minute changes such as lack of beds, insufcient operative time or there being no available trainee for assessment. This results in a number of cases which cannot be recruited as planned, with patient consent rarely being the determining factor. For the few cases in which patients declined to participate, we found that their decision was often surrounded by misconceptions regarding trainee involvement in performing supervised elective surgery. For some patients this was a reection of heightened concern for their surgery and a wish for a consultant to perform the operation. For some others, there was simply an expectation that a consultant would be performing the operation. We provided an open discussion of the training system, acknowledging the role of supervised operating in training. In some cases, this open discussion resulted in patients deciding to consent to participate. • • • • • • • • • • • Surgical Skills and Non-Technical Behaviours in the Operating Room 59 Lessons Learnt and Suggested Learning Points Consider the complexity of consenting and recruiting patient participants for observational studies in the surgical environment and the resources required. Appreciate the patient’s perspective towards surgical systems and training, which will inform your consent process. Communicate at all levels to make full potential of the team’s resources and to maximize the recruitment of suitable cases. Consider contingency lists, so that if a surgical case ‘goes down’, there may be other suitable cases for assessments. 5. Surgical Trainees as Study ‘Participants’ Ethical approval for the study dictates written consent from patients as participants but not from trainees. However, from a training perspective and during the implementation of the study, we have also considered surgical trainees to be study participants. It could be argued that competency-based assessment of surgical skills is now a requirement of surgical training. However, this does not extend to the additional research conditions, including independent assessors, video and NOTSS assessments. Our approach has been to seek verbal consent from surgical trainees before their involvement in the study. We provide all trainees with an invitation letter before introducing the study into a new surgical specialty and give them an opportunity to discuss the research with the study coordinator, thereby ensuring participation without duress. There have been a handful of trainees across the specialties, usually senior trainees approaching their CCT (certicate of completed training), who have declined to take part. We have experienced initial hesitancy from some trainees regarding their involvement in the study, often centred on misunderstanding the purpose of the study and concerns that research data could affect their training, for example in the event of a critical incident occurring. One of the clear messages we have conveyed is that the study is designed to assess the assessment tools themselves, in particular their validity and reliability across different surgeons, cases and specialties, and it is not assessing an individual’s level of surgical skill or competence. The majority of trainees, who decided to participate, have increasingly engaged with assessments. (See Theme 6, ‘Research versus Training Agenda’, for a fuller discussion.) We have continued in our attempts to collect data to suit the statistical model employed for the study, the optimal data being different combination of trainees and assessors across the cases. Some trainees operate more frequently, some senior surgical trainees perform complex procedures which are not covered by the study and some consultant lists have more index procedures. These differences generate unbalanced data. However, moving trainees across lists for assessments in the operating room proved unworkable. Some trainees were resistant to movement, preferring training and assessment by their regular consultant. We also realized • • • • Safer Surgery 60 that assigning a particular trainee to perform a surgical case assessed their behaviour as ‘technical operators’, rather than reecting the complete role of a holistic surgical practitioner. For the same reasons, if there were late changes to the trainee covering a particular list, we decided to exclude the case to focus the study on authentic workplace assessments. Lessons Learnt and Suggested Learning Points Consider the ethics of trainees as participants in observational surgical education studies. Engage trainees in the process of assessment by communicating effectively the research purpose and the role of their involvement. Respect the working surgical system in place: research needs to work with the surgical system rather than adapt the system to suit the research. 6. Research versus Training Agenda: A Dichotomy or Collaboration? The study protocol includes the validation of assessment tools which are in current use in the workplace for surgical training. (See the earlier background of this chapter for an overview of PBA and OSATS tools within surgical training programmes.) There are opportunities for educational research to form collaborations or conicts with the training agenda, and this is illustrated by discussing our role as independent assessors in the study. Examples of collaboration Providing opportunities for training in workplace- based assessment has moved beyond the research agenda to provide trainees and trainers with valuable, timely training on the tools which are integral to the new surgical curricula. The study has provided ring-fenced opportunities for training and assessment, which has resulted in the increasing engagement of trainees in the study. Trainees have made comments such as, ‘I’m happy to take part; it means I get to operate’ and ‘It’ll guarantee some assessments for my portfolio’. Participation in the study has also encouraged a number of trainers and trainees to use the PBA tool for the rst time. We have been able to show trainers and trainees that suitable cases for workplace training can be identied opportunistically, and that the process of assessment and feedback is feasible, adding little time to an operating list. Highlighting the role of formative assessment in driving learning has encouraged appropriate use of the PBA within the curriculum. For example, use of parts of the PBA for trainees not ready to complete the whole operation under supervision. Our ‘eld testing’ of the assessment tools has generated suggestions for tool modication which have been forwarded to the relevant bodies for consideration. For example, carotid endarterectomy and caesarean section are performed under local or regional anaesthetic. The trainee’s communication with the patient • • • Surgical Skills and Non-Technical Behaviours in the Operating Room 61 therefore becomes an important assessment item which was not part of the original template. Examples of dichotomy All assessors score independently without conferring or discussion until the assessment tools for the case are completed. There is no discussion of scoring between cases, either between independent assessors or with trainers, which could have a convergent effect on assessor ratings. However, it is inevitable in the clinical setting for some discussion to take place with the presence of trained independent assessors who are facilitating cases for assessment purposes. Assessment in the operating theatre is a relatively new training method, and the research team are seen to represent a body of expertise in assessment and feedback. We have found that our role as a complete observer in the independent assessment of skills and behaviours in the operating room is both unrealistic and unworkable. We suggest that our role within this study is more aligned to observer-as-participant, part way along the continuum from complete observer, to observer-as-participant, onto participant-as-observer, then to complete participant (Gold 1958). The process of assessment itself prompts discussion; for example, what constitutes a good surgical technique and why? Judgements on skills and behaviours are rated independently between assessors, although discussion surrounding the subjectivity of skills and behaviours is seen as a necessary outcome of the study implementation. Another area of conict we have recognized relates to the quantity and quality of the research data. The supervising consultant is often not present in the operating room until the case commences, which omits the pre-operative preparation section of the PBA and equivalent sections on the OSATS. Do we prompt consultants to be present at the whole case to maximize data quantity or is the research question most accurately answered by the researchers assuming the observer role? Do we encourage consultants to allow trainees to lead the case, which provides the most authentic assessment of surgical performance or observe the real-life training situation? We have found it challenging, particularly when new trainees and/or trainers become involved in the study, to provide sufciently general information to support the assessments and the use of the assessment tools, without introducing specic prompts which would affect the assessment ratings and overall data quality. We have made compromises to uphold the research agenda, as from a training standpoint, independent assessor(s) would prompt trainers on the appropriate use of the assessment tools, such as to address the inconsistencies in tool use highlighted above in Theme 3. Lessons Learnt and Suggested Learning Points Educational research can engender collaboration and conicts between the research and training agendas. • Safer Surgery 62 Consider your role as researchers in upholding the research agenda whilst drawing upon collaborations with the training agenda. Some discussions surrounding the assessment of skills and behaviours are a necessary condition for implementing workplace assessments. 7. Developments in Study Design and Methodology during Implementation Developments to the study design and methodology have arisen to meet the requirements of the study aim and to promote future research directions. It is only when the study design and methodology are subjected to eld testing during their implementation that the full requirements of the study protocol can be realized. These developments have been driven by the foresight of the research team and with guidance from the Studys Steering Committee. External review of the original study protocol advised us to consider more than two assessment tools (originally PBA and OSATS). The development of rating non-technical skills in the operating room, supported by the literature evidence for the relationship of these skills to surgical skills and safety, stimulated the addition of NOTSS to the study protocol. We originally recruited patients from one specialty at a time because of the logistical difculties in working in multiple specialties simultaneously. However, in order to improve recruitment and maximize the reliability data it has been essential for us to recruit from two, sometimes three specialties at any one time. This has been achieved by securing funding for a further independent assessor who has also taken responsibility for coordination of cases within her own specialty of obstetrics and gynaecology. An early ethics amendment approved the addition of obstetrics and gynaecology as a surgical specialty with the inclusion of four extra index procedures. This has provided the opportunity to compare PBA with OSATS in a specialty that uses OSATS as the current workplace assessment tool. We needed to collect a larger dataset of assessments in this specialty compared to the others to allow a comparison of the tools. However, recruiting cases in obstetrics and gynaecology has been very successful because it lends itself to providing large numbers of suitable cases (see Theme 8, ‘The Signicance of Context’, for a full discussion). The addition of this specialty has included a non-elective index procedure, urgent caesarean, which provides an opportunity for video assessment and use of the NOTSS tool in urgent surgical cases. The completion of more than one assessment tool has a potential to confound each assessment. The best case scenario is for each independent assessor to complete one tool per case. However, the number of cases required to obtain sufcient data to test each of the assessment tools would be very large, certainly unachievable within this study. The nal study protocol accepts that there will be completion of more than one tool by independent assessors (PBA or OSATS and NOTSS). We recognize that differences exist between the assessment tools which make their simultaneous use in assessment problematic. For the PBA tool, there is an expectation that trainees should verbalize their • • Surgical Skills and Non-Technical Behaviours in the Operating Room 63 intentions throughout, also advised in the PBA validation document for training assessors. The NOTSS tool has been designed for use in as naturalistic a setting as possible, without prompting trainee communication or decision-making. Our concern surrounding the completion of two tools was raised with the Steering Committee and the decision has been to minimize the completion of two tools. For example, if there are two independent assessors observing a case, both complete the PBA or OSATS but only one completes a NOTSS assessment. Where two tools have been completed, the impact of this can be considered using a post hoc analysis. Recruitment has been better in some specialties than in others. In specialties where we realized that recruitment would not improve, we decided to move on rather than risk a training effect by remaining in the speciality. Our intention is to return to specialties to capture another cohort of trainees next year, prioritizing the specialties in which further recruitment is needed. Lessons Learnt and Suggested Learning Points Developments to study design may be required to meet the research aim or to take advantage of new research directions. Preliminary eldwork can require changes to be made to the study design and methodology, which may involve ethics amendments. A exible team approach with good foresight encourages the negotiation of study developments. 8. The Signicance of Context: Inter-specialty Differences The individual surgical specialties have offered different advantages and disadvantages to the study implementation. The obstetrics and gynaecology specialty has leant itself well to the study methodology. Index cases occur frequently on operating lists, for example between three to four elective caesareans every weekday. All the index cases are relatively short so it is feasible to obtain several trainee assessments per list. The specialty uses a team consultant structure, with each trainee assigned to a team of three to four consultants, which enables each trainee to operate with several consultants each week. Other specialties presented signicant problems for study implementation. Within orthopaedics, operating lists were often amended at short notice, giving insufcient time to inform patients about the study. Staff structuring, with each consultant allocated a single trainee limited the combinations of trainees to trainers for assessment. We also found the culture of surgical assessment and training within each surgical specialty signicantly different. In obstetrics and gynaecology there is an established culture for objective assessments of surgical competence and a requirement of consultants to provide assessment and feedback to trainees, which facilitated our introduction of the study. The RCOG has phased in the use of • • • . to observer-as-participant, part way along the continuum from complete observer, to observer-as-participant, onto participant-as-observer, then to complete participant (Gold 195 8). The process. training and assessment by their regular consultant. We also realized • • • • Safer Surgery 60 that assigning a particular trainee to perform a surgical case assessed their behaviour as ‘technical. coverage. Identify the most effective and suitable method for training different assessors. • • • • • • Safer Surgery 58 Consider consistency in the training approach you adopt, taking into account differences