the-assessment-of-observed-practice-keele-university-july-2011

Department of Medicines Management School of Pharmacy Keele University The Assessment of Observed Practice: A Literature Review Dr Elizabeth Mills Professor Alison Blenkinsopp Professor Robert K McKinley Dr Patricia Black June 2011 The Assessment of Observed Practice: A Literature Review Dr Elizabeth Mills, Prof Alison Blenkinsopp, Prof Robert K McKinley, Dr Patricia Black Contents List of Tables Acknowledgements Glossary List of Abbreviations Executive Summary Background .29 Introduction 31 What needs to be assessed? 32 Methods of assessment .34 Principles of selection of assessment methods 35 Methods 37 Utility Index Ratings .40 Presentation of results .41 Results 42 Direct Observation of Clinical Skills (DOCS e.g mini-CEX) .44 Direct Observation of Procedural/Practical Skills (DOPS) 50 Objective Structured Clinical Examination (OSCE) .56 Incognito Standardised Patients 64 Multi Source Feedback .67 Patient Surveys 73 Case-based Discussion 76 The Long Case 78 Portfolios of Evidence 81 Programmes of assessment 85 Assessor training 92 Discussion 94 Utility of individual tools 94 Programmes of assessment .100 Assessor training 101 Strengths and limitations of this project 101 Utility of the current pre-registration assessments 102 Towards the future 103 Conclusion .105 Key issues for consideration by GPhC 105 References 106 Appendices 109 List of Tables Table Table Table Table Table Table Table Table Table Table 10 Table 11 Table 12 Table 13 Table 14 Table 15 Table 16 Table 17 Summary of practice-based assessment methods and tools (adapted from Clarke, 2007) …………………………………………………………………………………………………………………………………….34 Summary of van der Vleuten’s Utility Index (van der Vleuten, 1996) 36 Summary of case studies 39 Summary of papers included in the literature review .42 Assessment methods and tools identified in the literature review, case studies and practice examples 43 Summary of Utility of DOCS .49 Summary of Utility for DOPS 55 Pharmacy Pre-registration Stakeholders Currently Using OSCEs .62 Summary of Utility of OSCEs 63 Summary of Utility for Incognito Standardised Patients 66 Summary of Utility of MSF 72 Summary of Utility of Patient Surveys 75 Summary of Utility of the Long Case .80 Summary of Utility for Portfolios of Evidence 84 Pharmacy Pre-registration Assessment Programmes in the UK, Canada, New Zealand and Ireland 85 Assessment tools in the Foundation Programme 91 Areas of the Clinical Encounter Assessed by mini-CEX 95 Acknowledgements This project was funded by the General Pharmaceutical Council We are grateful to the following for their help in this work: • Dr Claire Stevens, Information Scientist, Department of Medicines Management, Keele University, for input and support in designing and conducting the literature search and sourcing papers • The members of RPSGB Pre-registration Liaison Group, and the NHS Pharmacy Education and Development Committee for responding to the e-mail request for information on the use of workplace based assessment in pre-registration training • All those who contributed information towards the case studies: o Professor Zubin Austin, Lesley Dan Faculty of Pharmacy, University of Toronto, Canada o Professor Graham Davies, Department of Pharmacy, King’s College London o Owain George, Competence Advisor, Pharmacy Council of New Zealand o Dr Delyth James, Senior Lecturer & Course Director (MSc in Pharmacy Clinical Practice), Welsh School of Pharmacy, Cardiff o Jignesh Patel, School of Pharmacy, University of Brighton o Professor Paul J Gallagher, Head of the School of Pharmacy, Royal College of Surgeons in Ireland • All those who contributed information towards the practice examples referred to throughout this report: o Dr Jane Portlock, School of Pharmacy, University of Portsmouth o Roisin Kavanagh, Programme Director JPB Diploma in General Pharmacy Practice, School of Pharmacy, University of London o Janet Gilbertson, All Wales Principal Pharmacist Education, Training and Personal Development Glossary Practice based assessments In order for there to be a shared understanding of the scope of this project we have defined ‘practice-based assessments’ as: “Assessments where performance is evaluated: - in the workplace or in simulated practice situations - including the use of standardised or simulated patients - including practice placements during undergraduate studies - excluding written case simulations Utility Index After many years of research and debate van der Vleuten proposed a Utility Index for assessments with ratings for the criteria reliability, validity, feasibility, acceptability and educational impact (van der Vleuten, 1996) The Utility Index has become a widely accepted tool which acknowledges that constraints of feasibility and acceptability have to be taken into account when selecting assessments The table below briefly describes each criterion of the Index Criterion Description Reliability The reproducibility of the scores obtained from an assessment Validity Validity is concerned with whether an instrument measures what it is purported to Feasibility Considers issues such as the logistics of running the assessment including trainee numbers and geographical spread; the venue for the assessment; capacity such as number of assessors required and training of assessors; the costs involved to the HEI, regulator and trainee; and the need for any equipment or IT development Acceptability Acceptability is considered from the perspectives of different stakeholders including: trainees, Teachers, Higher Education Institute / Regulator, Employer / Workplace, Assessors, Patient Educational impact Considers the impact of the assessment on learning for example: congruence between the educational objectives and the assessment; how feedback on the assessment is provided and sustained; how formative and summative assessments are combined; how much assessment is needed; how the assessment is spread over time; in what context the assessment takes place Workplace Based Assessments Assessments that take place purely in the workplace The Foundation Programme refers to the assessments used in the programme as workplace based assessments List of Abbreviations ACLF Advanced to Consultant Level Framework CbD Case Based Discussion DEPS Developing Educational & Practice Supervisors DOCS Direct Observation of Clinical Skills DOPS Direct Observation of Procedural Skills OR Direct Observation of Practical Skills GLF General Level Framework GPhC General Pharmaceutical Council JPB Joint Programmes Board MCQ Multiple Choice Questions mini-CEX Mini-Clinical Evaluation Exercise Mini-PAT Mini-Peer Assessment Tool MRCF Medicines Related Consultation Framework MSF Multi-Source Feedback OSCE Objective Structured Clinical Assessment REA Rapid Evidence Assessment RCGP Royal College or General Practitioners SP Standardised Patient SPRAT Sheffield Peer Review Assessment Tool TAB Team Assessment Behaviour WPBA Workplace Based Assessment Executive Summary Background The General Pharmaceutical Council (GPhC) is responsible for setting and checking standards for the initial education and training of pharmacists, including the pre-registration training period Like other regulators the GPhC has responsibility for ensuring, in the public interest, that the assessments used are of appropriate quality and rigour and can withstand scrutiny and possible challenge Pharmacy pre-registration trainees are currently assessed through four ‘signoffs’ by their preregistration tutor (three progress reports and a final declaration) and also through the national Registration Assessment (a multiple-choice examination) The purpose of the signoffs is for the tutor, based on observation in the practice setting, to certify that the trainee has met the GPhC’s Preregistration Performance Standards and is fit to work as a pharmacist The Registration Assessment provides a national benchmark which mitigates potential variation in individual pre-registration tutors’ practice-based assessment of trainees Current practice-based and national assessments in pharmacy pre-registration training have been the subject of some criticism and are recognised to have a number of weaknesses Assessment of observed practice is used to evaluate performance in the workplace or in simulated practice situations The validity and reliability of such assessments (which we will refer to as practice-based assessments) are dependent on both the assessment tool/s being used and the capability of the assessor/s Furthermore the resource implications of operationalising the different forms of assessment may have substantial infrastructure requirements The GPhC has indicated that it may wish in future to review the assessments used in the pre-registration training period and commissioned this study to consolidate existing evidence in order that any future discussions can draw on an informed synthesis Aim - To provide evidence on the strengths and weaknesses of practice-based assessments in the context of pharmacy pre-registration training for use in policy decision making by GPhC Objectives - From a critical review of evidence, to: Describe the range of practice-based assessments available, their designs and educational basis Set out the evidence for their effectiveness and evaluate their utility using van der Vleuten’s Utility Index (reliability, validity, feasibility, acceptability and educational impact) Illustrate and discuss challenges in operationalisation including scalability and sources of variability Consider and describe resource implications for different assessments Critically reflect on existing and possible future assessments, focusing on utility in the pharmacy pre-registration context Take account of both the current and proposed future integrated UG/Pre-registration programmes Make preliminary recommendations for consideration by GPhC Methods A Rapid Evidence Assessment was undertaken using the following methods: i) a review of peer reviewed published evidence; ii) collection and review of information to produce a small number of case studies to illustrate key issues and how they have been addressed in practice; iii) identification of tools currently in use in pharmacy training as ‘practice examples’ The analytical framework used to review and synthesise evidence was the van der Vleuten Utility Index for educational assessment tools (reliability, validity, feasibility, acceptability and educational impact) Review of Peer Reviewed Published Evidence Review question: “What is the utility of practice-based assessments in the context of pharmacy preregistration training?” The review used a pre-defined protocol, and a search strategy designed and implemented iteratively with support from an information specialist focusing on peer reviewed published systematic reviews, with structured extraction of data and synthesis Predefined inclusion criteria were used to identify relevant papers from title and abstract Full papers were obtained for those remaining and assessed for the following inclusion criteria: Published in English; A systematic review or a structured nonsystematic review; Topic of paper is an assessment tool that meets the project’s definition of 10 Appendix Search Strategies Medline Search Strategy(Yielded 944 papers) Limits of search: Humans; review (publication type); 01/01/1990 to 28/02/2011 SEARCH Educational measurement (MeSH term) AND (clinical OR skills OR competence OR practice OR work OR workplace OR work-based OR work based) AND Education, Pharmacy (MeSH term) OR (Education, Medical (MeSH term) AND (trainee OR student OR foundation)) OR Nursing Education Research (MeSH term) OR Nurse midwives/education (MeSH term subheading) OR Physical therapy (speciality) (Mesh term) OR occupational therapy (Mesh) OR dentists (MeSH) OR Educational AND (impact OR measurement OR standard OR effects) OR Utility OR validity OR reliability OR feasibility OR acceptability SEARCH Educational measurement (MeSH term) AND OSCE OR observed structured clinical examination OR simulated patient encounter OR standardised patient exam OR “direct observation” OR 111 “multi source feedback” OR MSF OR “team assessment behaviour” OR TAB OR “peer assessment tool” OR mini PAT OR mini-PAT OR “clinical evaluation exercise” OR mini-CEX OR mini CEX OR “case based discussion” or CbD SEARCH “Medication related consultation framework” International Pharmaceutical Abstracts (yielded 123 results) Limits of search: 01.01.1990 to 28.2.2011, human studies SEARCH (Educational measurement) AND (clinical OR skills OR competence OR practice OR work OR workplace OR work-based OR work based) boolian search Limited to subjects: Pharmacy – students Education, pharmaceutical pharmacy students Students – pharmacy Interventions – education, pharmaceutical Curriculum – education, pharmaceutical AND Systematic OR review SEARCH OSCE OR observed structured clinical examination OR simulated patient encounter OR standardised patient exam 112 AND Systematic OR review SEARCH direct observation AND assessment AND Systematic OR review SEARCH “multi source feedback” OR MSF OR “team assessment behaviour” OR TAB OR “peer assessment tool” OR mini PAT OR mini-PAT AND Systematic OR review SEARCH Clinical evaluation exercise OR mini CEX OR mini-CEX AND Systematic OR review SEARCH case based discussion OR CbD AND Systematic OR review SEARCH medication related consultation framework CINAHL, British Nursing Index and PsycINFO (yielded 393) Limits of search: 01.01.1990 to 28.2.2011, human studies 113 SEARCH Cinalhl major heading: (clinical assessment tools OR competency assessment) AND (educational measurement ) AND (systematic OR review (keywords) ) SEARCH (All subject Terms)educational measurement and (clinical or skills or competence or practice or work or workplace or work-based or work based) AND (MH educational measurement) AND systematic OR review AND (subject terms) Pharmacy education OR medical Education OR nurse education OR (education AND physical therapy OR occupational therapy OR dentist) SEARCH OSCE OR observed structured clinical examination OR simulated patient encounter OR standardised patient exam AND Major heading (educational measurement) AND Systematic OR review SEARCH Major heading (educational measurement) AND Direct observation SEARCH “multi source feedback” OR MSF OR “team assessment behaviour” OR TAB OR “peer assessment tool” OR mini PAT OR mini-PAT 114 OR Clinical evaluation exercise OR mini CEX OR mini-CEX OR Case based discussion OR cbd AND Education AND systematic OR review 115 Appendix Comparison of Key Activities to be undertaken in a Medication-Related Consultation against the Medical Consultation Models Key Activities of a MedicationRelated Consultation Establish FULL Medication History Explore patients’ understanding about medicines Explore patients’ illness understanding Elicit and address patients’ concerns about treatment Explore patients’ adherence with prescribed treatment Negotiate medicines management plan with patient Check patients understanding The Model of Patientcentred Care The Pendleton Consultation Model The Inner Consultation The Information Exchange Model The Three Function Model The DREAM consultation The E4 model for physicianpatient communication The CalgaryCambridge Consultation Model The SEGUE Framework No Partially met No No No No No No Partially met No No Partially met Partially met No No Partially met Partially met Partially met Yes Yes Yes Yes Yes Yes Yes Yes Yes No Partially met Partially met Partially met Partially met No Partially met Partially met Partially met No No No Partially met Yes No Yes Partially met Partially met Yes Yes Yes Yes Yes Partially met Yes Yes Yes Yes Yes Yes Partially met Yes Partially met Yes Yes Partially met No No No No No No No Partially met No No No No Partially met Partially met Check patient’s ‘practical’ No No ability to follow treatment plan Refer to other healthcare Yes Yes professionals (if appropriate) Key: Yes =activity undertaken, No =activity not undertaken Source: Abdel Tawab R PhD Thesis (2005) Development and Psychometric Validation of a Framework for Medication-related Consultations 116 Appendix Foundation Programme Competencies and Assessments Competence Professionalism Assessments 1.1 Behaviour in the workplace TAB, CBD, probity declaration and clinical supervisor's report 1.2 Health and handling stress and fatigue TAB, clinical supervisor’s report and health declaration 1.3 Time management and continuity of care TAB, clinical supervisor’s report, feedback form and survey receipts Good clinical care 2.1 Eliciting a history Mini-CEX and CBD 2.2 Examination Mini-CEX 2.3 Diagnosis and clinical decision-making Min-CEX and CBD 2.4 Safe prescribing CBD 2.5 Medical record-keeping and correspondance CBD 2.6 Safe use of medical devices Min-CEX, DOPs, log book and CBD Recognition and management of the acutely ill patient 3.1 Promptly assesses the acutely ill or collapsed patient TAB, log book, Mini-CEX & CBD 3.2 Identifies and responds to acutely abnormal physiology TAB, log book, Mini-CEX & CBD 3.3 Where appropriate, delivers a fluid challenge safely to an acutely ill patient TAB, log book, Mini-CEX & CBD 3.4 Reassesses ill patients appropriately after starting treatment TAB, log book, Mini-CEX & CBD 3.5 Undertakes a further patient review to establish a differential diagnosis TAB, log book, Mini-CEX & CBD 3.6 Obtains an arterial blood gas sample safely, interprets results correctly TAB, log book, Mini-CEX & CBD 3.7 Manages patients with impaired consciousness, including convulsions TAB, log book, Mini-CEX & CBD 3.8 Uses common analgesic drugs safely and effectively TAB, log book, Mini-CEX & CBD 3.9 Understands and applies the principles of managing a patient with acute mental disorder including self harm TAB, log book, Mini-CEX & CBD 117 3.1 Ensures safe continuing care of patients on handover between shifts, on call staff or with ‘hospital at night’ team by meticulous attention to detail and reflection on performance TAB, log book, Mini-CEX & CBD Resuscitation 4.1 Resuscitation TAB, CBD and ILS/ALS/equivalent course 4.2 Discusses Do Not Attempt Resuscitation (DNAR) orders/advance directives appropriately TAB and CBD Discharge and planning for chronic disease management 5.1 Discharge planning CBD and TAB 5.2 Chronic disease management CBD and TAB Relationship with patients and communication skills 6.1 Within a consultation Mini-CEX, DOPS and TAB 6.2 Breaking bad news CBD and TAB Patient safety within clinical governance 7.1 Treats the patients as the centre of care CBD and TAB 7.2 Makes patient safety a priority in own clinical practice CBD and TAB 7.3 Promotes patient safety through good team working CBD and TAB 7.4 Understands the principles of quality and safety improvement CBD and TAB 7.5 Complaints TAB and clinical supervisor’s report Infection Control 8.1 Infection Control Nutritional care 9.1 TAB, logbook and DOPS Nutritional care TAB, CBD and mini-CEX 118 10 10.1 Health promotion, patient education and public health Educating patients Mini-CEX, CBD and TAB 10.2 Environmental, biological and lifestyle risk factors Mini-CEX, CBD and TAB 10.3 Smoking Mini-CEX, CBD and TAB 10.4 Alcohol Mini-CEX, CBD and TAB 10.5 Epidemiology and screening Mini-CEX, CBD and TAB 11 Ethical and legal issues 11.1 Medical ethical principles and confidentiality CBD, TAB and clinical supervisor’s report 11.2 Valid consent Mini-CEX, CBD and DOPS 11.3 Legal framework of medical practice CBD and clinical supervisor’s report 11.4 Relevance of outside bodies Reflective reports, probity and health declarations, CBD 12 Maintaining good medical practice 12.1 Lifelong learning CBD and TAB 12.2 Research, evidence, guidelines and care protocols TAB, CBD and mini-CEX 12.3 Audit Audit project review and e-portfolio 13 13.1 14 Teaching and Training Teaching and Training Developing the clinical teacher Working with colleagues 14.1 Communication with colleagues and teamwork for patient safety TAB and CBD 14.2 Interface with different specialties and with other professionals TAB and CBD 119 Appendix Assessment Tools that Could be Used to Assess the Pharmacy Pre-registration Performance Standards Performance Standard Unit A - Personal Effectiveness Assessment MRCF Number Description MSF A1 Manage yourself Behave in a manner consistent with membership of the profession y A1.1 A1.2 A1.7 Manage your time effectively Recognise your personal and professional limitations and refer appropriately Respond with willingness and flexibility to new situations and change Remain composed and personally effective in all situations Make decisions which demonstrate clear and logical thought Take responsibility for and accept outcomes of your own decisions A1.8 Amend your behaviour, when necessary, based on evaluation of your performance by yourself or others A2 Manage work A2.1 A1.3 A1.4 A1.5 A1.6 mini-CEX DOPS CbD Portfolio Standardised Patients OSCE y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y A2.2 Carry out tasks effectively Approach tasks and situations in accordance with the law and with the Regulator's Standards of conduct, ethics and performance y y A2.3 Follow work systems correctly y y A2.4 Use resources effectively y y y y y y y y y 120 y Patient Surveys MRCF MSF A3 Manage Problems A3.1 Recognise and define actual or potential problems A3.2 A3.3 A3.4 Identify workable options to resolve the problem Select the best solution, based on sound analysis and appropriate evidence Suggest and if appropriate implement solutions to problems mini-CEX DOPS CbD y y y y y y y y y y y y y y y y y y y y y y y y y y y A4.1 Demonstrate a commitment to quality Work to an acceptable standard when preparing products and delivering services y y y A4.2 Check your own work effectively y y y A4.3 y y A4.4 Minimise error by others through effective supervision Identify and rectify your own and others' mistakes promptly and effectively y y A4.5 Minimise health and safety risks to yourself and others y y A4.6 Base your actions, advice and decisions on evidence y A4.7 Obtain and process the evidence you need to meet A4.6 Have successfully carried out a small planned audit assignment y y y y y y y y y y y y A4.8 Patient Surveys y y A4 Standardised Patients OSCE Evaluate the outcomes of the solution after implementation and if necessary redefine the problem A3.5 Portfolio y 121 y MRCF A5 MSF A5.2 Demonstrate ongoing learning and development Identify and prioritise your own learning and development needs Develop your own plans to meeting identified needs using SMART learning objectives A5.3 Make full use of learning and development opportunities y A5.4 Evaluate whether your learning objectives have been met y A5.5 y A5.6 Identify your further learning needs Record your own learning and development process and outcomes A5.7 Apply learning to practise y A5.1 mini-CEX DOPS CbD Portfolio Standardised Patients OSCE Patient Surveys y y y y y y y y y y y y Unit B - Interpersonal Skills B1 Communicate effectively B1.1 Communicate effectively in English B1.2 B1.3 B1.4 B1.5 B1.6 B1.7 B1.8 Behave in a polite and helpful manner Sensitively approach people who need or may need assistance Elicit all relevant information by the use of appropriate questions Listen effectively to the whole message Respect and observe confidentiality Act appropriately in response to spoken and unspoken needs of others Behave in a manner which instils confidence y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y B1.9 Behave assertively y y y 122 MRCF B1.10 B1.11 Use appropriate body language Provide information and advice appropriate to the needs of recipients MSF mini-CEX y y y DOPS CbD Portfolio Standardised Patients OSCE y y y y y y y y y y y y B1.12 B2 Handle conflict appropriately y y y y B2.5 Work effectively with others Acknowledge the ideas and opinions of others and act on them when appropriate Present your own ideas and opinions appropriately when speaking and in writing Meet commitments made to others within agreed deadlines Give constructive feedback to others based on accurate evaluation of their performance Secure help from others when necessary in an appropriate manner B2.6 Assist others when necessary y B2.7 Delegate tasks appropriately Supervise others in an appropriate manner to ensure that agreed outcomes are achieved Use your knowledge and skills effectively when helping others learn y B2.1 B2.2 B2.3 B2.4 y y y y y y y y y y y y y y B2.8 B2.9 y y y y y Unit C - Medicines and Health C1 Manage the dispensing process C1.1 Correctly receives prescriptions into the pharmacy y y C1.2 Check the prescription is valid y y y y 123 Patient Surveys y MRCF MSF mini-CEX DOPS CbD Portfolio Standardised Patients OSCE C1.3 Assess the prescription for safety and clinical appropriateness y y y C1.4 Resolve any identified problems appropriately y y y C1.5 Perform calculations correctly y y y C1.6 Assemble the prescription correctly Supply extemporaneously prepared products according to the correct formula Correctly issue dispensed items to patient or representative, with appropriate information and advice y y y y y y y y y y Patient Surveys y C1.7 C1.8 y y y y y y y C1.9 y y C1.10 Ensure stock is managed correctly Respond appropriately to requests to dispense prescription only items without a prescription y y y C1.11 Correctly process necessary documentation y y y C1.12 Effectively check prescriptions dispensed by others y y y y y y y C2.5 Provide additional clinical and pharmaceutical services Provide considered and correct answers to queries, founded on research-based evidence Pro-actively assist patients to obtain maximum benefit from their treatment Identify and take action to minimise risk to patients from their treatment Actively provide information and advice to healthcare professionals Construct medication histories using a range of sources C2.6 Use medication histories correctly C2 C2.1 C2.2 C2.3 C2.4 y y y y y y y y y y y y y y y y y y y y y y y y y y y y y y 124 y y y y y y MRCF C2.7 C2.8 C2.9 Recognise possible adverse drug reactions, evaluate risks and take action accordingly Provide appropriate information and advice on the management of minor and common ailments Effectively use opportunities to promote and support healthy lifestyles and prevent disease MSF mini-CEX y y y y DOPS CbD Portfolio Standardised Patients OSCE y y y y y y y y y Patient Surveys y y y y y y y y C2.10 C2.11 Demonstrate awareness of emergency first aid Refer or direct the person to a more suitable source of help or information when necessary y y y y y y y 125 y y y