rapid-qualitative-evaluation-to-inform-decision-making

Rapid qualitative evaluation to inform decision making C EC I LI A VI N DRO LA N ATA S H A P H I L L I P S D E PA R T M E N T O F A P P L I E D H E A LT H R E S E A R C H A N D R S E T, U C L C H I E F N U R S I N G I N F O R M AT I C S O F F I C E R R A P I D R E S E A R C H E VA LU AT I O N A N D A P P R A I S A L L A B ( R R E A L ) N I AA HS R C S O C I AL S C I EN TI S T C V I N D R O L A @ U C L A C U K U CLH Request for a rapid evaluation •Intervention based on delivering QI training and support to clinicians •Delivered across six NHS Trusts •Based on a training programme and support for the development of local QI projects •Difficulties outlining the scope of the evaluation •Programme theory not fully articulated •Emerging findings required in two months to inform future funding decisions and potential scale-up •Final evaluation findings required in six months “The timeliness of information is no less critical than its accuracy” (McNall et al 2004) Why does some research and evaluation need to be timely? Timeliness influences the utility of research Only findings shared at particular moments can inform decision-making Mismatch between policy and evaluation (Nunns 2009) Some research topics are time-sensitive How would you make an evaluation rapid? ◦ How would you make sure a study is rapid? ◦ Where would you speed up the research process and why? ◦ How would you speed it up? What are rapid methods? Rapid Research and Evaluation Methods (REAM) McNall and Foster-Fishman (2007) How rapid are rapid approaches? 6 weeks (Scrimshaw, et al 1991; Watts et al 1989) 4 to 6 weeks (Beebe 1995, 2014) 3 months (Handwerker 2001) weeks (Wilson and Kimane 1990) 4 to 8 weeks (ERAP 1988) weeks (Pearson, et al 1989) 2-3 months (Bentley, et al 1988) Social Science & Medicine 189 (2017) 63e75 Contents lists available at ScienceDirect Social Science & Medicine How rapid is rapid research? journal homepage: www.elsevier.com/locate/socscimed Review article Rapid qualitative research methods during complex health systematic review the63e75 literature G.A Johnson, emergencies: C Vindrola-Padros / A Social Science & Medicine 189of (2017) Ginger A Johnson Table Main characteristics of articles included in the review a b c a, b, * , Cecilia Vindrola-Padros Anthrologica, Oxford, United Kingdom Department of Anthropology, Southern Methodist University, Dallas, TX, United States Department of Applied Health Research, University College London, United Kingdom Use of research Timeframe Research Research methods Sample size and a b s t r a c t population findings design Type of research for data Article history: The 2013e2016 team Ebola outbreak in West Africa highlighted both the successes and limitations of social collection Received 26 March 2017 Location and Type of complex type of health setting emergency Study aims Cheung, E et al 2003 Afghanistan Setting: Rural community Outbreak Scurvy Brennan and Rimba Indonesia Setting: Rural community Natural disaster Tsunami Identification of scurvy A few days outbreaks and (exact Keywords: Rapid of qualitative monitoring an methods number Complex health emergency Systematic review intervention not Rapid appraisal specified) Epidemic Natural disaster days Determine the public Qualitative health research health impact of a tsunami First author name Year 2005 ~ a2006 Güeren ~ o, F Burguen et al Broz, D et al 2009 Krumkamp, R et al 2010 Bile, K M et al 2010 Brahmbhatt, D 2010 et al 67 c a r t i c l e i n f o Received in revised form 30 July 2017 Accepted 31 July 2017 Available online August 2017 science contributions to emergency response operations An important limitation was the rapid and effective communication of study findings A systematic review was carried out to explore how rapid qualitative methods have been used during global heath emergencies to understand which methods are used,groups; how they are applied, andcommunity the difficulties faced by social science researchers in Mixed commonly Focus Case 120 members Identification ofthe field We also asses their value and benefit for health emergencies The review findings are used to propose methods note reviews in 15infocus groups (groupsarticles high-risk forwere recommendations for qualitative research this context Peer-reviewed and greyareas literature identified through six online databases An initial search was carried out in July 2016 and updated in International and with men and women, targeting February 2017 The PRISMA checklist was used to guide the reporting of methods and findings The articles were assessed for quality using the MMAT of andvillage AACODSleaders) checklist From an initial search yielding national inclusion interventions 1444 articles, 22 articles met the criteria for inclusion Thirteen of the articles were qualitative studies “monitoring” teamsdesign The purpose of the rapid studies included: the identification of and nine used a mixed-methods of the outbreak, and assessment of infrastructure, control strategies, health needs and health Informed the Mixed causes Observations; Focus Survey among 32 facility use The studies varied in duration (from days to month) The main limitations identified by International households methods groups; the authors were:Surveys; the low quality of the collected data, small sample sizes, and little time for crosschecking facts with other data sources to reduce bias Rapid qualitative methods were seen as benefiFocus group with women Rescue Secondary data cial in highlighting context-specific issues that need to be addressed locally, population-level behaviors influencing health service use, and organizational challenges in response planning and implementation Committee's from the community analysis Recommendations for carrying out rapid qualitative research in this context included the early desigresponse size not specified and as a sample nation International of community leaders point of contact, early and continuous sharing of findings, and development of recommendations with local policy makers and practitioners national research © 2017 Elsevier Ltd All rights reserved teams Administrative and clinical Informed US days Mixed Interviews; Rapid health needs staff from 12 hospitals humanitarian methods Observations; assessment to plan assistance Secondary data the most confounding aspects of the outbreak and execute was the staggering Introduction inaccuracies of early disease models which were unable to predict strategies analysis humanitarian how the basic reproduction number of Ebola would react in a In December 2013, a toddler from the Kissi region of Gu eck edou assistance Prefecture died of a sudden and mysterious illnessInternational e months later andregional environment with: 1) governments severely weakened by confirmed as Ebola e in a village near Guinea's border with Sierra national researchdecades of corruption and civil war, 2) failing health care systems, 3) distrust between local populations and governmental figures, 4) Leone and Liberia (Baize et al., 2014; Saez et al., 2014 In the weeks, teams extensive trading networks and patterns of mobility through months and years to follow, the virus would spread throughout the porous borders, 5) spread of the outbreak the from rural loWest African region11 and beyond with over 28,000Interviews; people infected Natural USA Informed 33national staff members days Qualitative Effectiveness of cations to large, densely populated urban centers, and 6) burial and over 11,000 deaths e a case rate nearly 70 times more than that Setting: (clinicians and non-clinical response directed Observations responseofstrategy to disaster rituals involving intimate contact with the deceased (a period in the next largest Ebola outbreak in history (WHO, 2016) One of viral loads are at their highest by peak) Relief center Hurricane the(Abramowitz, Chicago 2015; staff) National researchwhichsupport provide health care to Aylward et al., 2014; Benton and Dionne, 2015; CDC, 2014; Department of team Hurricane* Corresponding Katrina author Department of Anthropology, Southern Methodist UniChowell and Nishiura, 2015; Faye et al., 2015; Leach, 2015; Public Health evacueesversity, Dallas, TX, United States Richards et al., 2014; Wilkinson and Leach, 2015) These were all E-mail address: johnson.ginger@gmail.com (G.A Johnson) Not specified Developed a new Qualitative Interviews; N/A Outbreak Systematic assessment Not http://dx.doi.org/10.1016/j.socscimed.2017.07.029 framework for specified Documentary Influenza of the national health 0277-9536/© 2017 Elsevier Ltd All rights reserved pandemic analysis system capacity to planning respond to pandemic influenza Informed the Government, Survey; Informal Effective coordination, A few days Mixed Natural Pakistan response to humanitarian agencies, methods interviews (exact joint planning, disaster Setting: enhance primary and other partners (described as number Government Earthquake, distribution of roles Sample sizes not specified care and hospital ‘consultations’) not cyclone and and responsibilities, offices and capacities International and specified) and resource floods healthcare national research mobilization between facilities teams partners days Mixed Interviews; Surveys 43 shelter staff members Informed the Evaluate the Natural USA response by (including volunteers, methods National research composition, predisaster Setting: providing a disease nurses, medical team deployment training Hurricane Shelter technicians, and assistants) burden assessment and recognition of and establishing scenarios with Thailand Setting: Healthcare facilities Natural disaster Tsunami How rapid is rapid research? Additional material is published online only To view please visit the journal online http://dx.doi.org/10.1136/ bmjqs-2017-007226) ► Department of Applied Health Research, University College London, London, UK Institute of Archaeology, University College London, London, UK Correspondence to Dr Cecilia Vindrola-Padros, Department of Applied Health ► Similar to other rapid ethnography approaches, it differs in the sense that in FREE there is 62 Focused rapid extensive use of field notes instead of digital recordings.62 ethnographic evaluation (FREE)* ► Short-term video ethnography to create an intensive, complex and rich data set44 44 Short-term focused ► Permits immersion into experience without being intrusive44 video ethnographic case study* *These terms were not used in the search strategy, but emerged from the reviewed articles nuances of care-seeking practices, provide a holistic understanding of service delivery and document activities or interactions as they occur in practice These features of rapid ethnographies allowed the generation of research findings with a close resemblance to the lived realities of service providers and users and were, therefore, deemed SYSTEMATIC suitable to inform service delivery REVIEW For instance, Goepp et al24 argued that rapid assessment processes can ‘close the gap between needs as perceived by planners and by the intended users of services, which in turn increases uptake and adoption of services.’ Quick and dirty? A systematic review of the use of rapid ethnographies in healthcare organisation and delivery The research topics covered by the studies could be Research topics organised in five main categories: (1) an exploration of health attitudes and healthcare seeking practices; Cecilia Vindrola-Padros,1 Bruno Vindrola-Padros (2) the identification of barriers to health service use; (3) the evaluation of the use of services or information ABSTRACT climates of healthcare systems by healthcare staff; organisations (4) an analysisbyof patients’ Background The ability to capture the complexities of adopting a wide range of research experiences of 2–4 treatment andrapid the built environment; healthcare practices and the quick turnaround of findings Various forms of rapid approaches and (5) an assessment of healthcare professionals’ team make rapid ethnographies appealing to the healthcare research have been used, including rapid dynamics These topics were explored in the context sector, where changing organisational climates and evaluations, rapid appraisals, rapid assess- of different types of health services including: end of life care, palliative care, emergency services, maternity services, immunisation, intensive care and surgery The studies were based on the delivery of services to patients diagnosed with HIV/AIDS, stroke and malaria as well as those experiencing multiple conditions Research designs Study time frames The study durations ranged from days to months, and some studies did not specify the length of the study or only included the number of hours of observation Three studies used a series of intensive periods in each of the study sites Ash et al25 and Chesluk and Holmboe26 spent 5–6 days at each site and Wright et al27 used intensive to 2-week periods at each site Data collection Most of the studies combined multiple methods of data collection The most common combination was interviews and observations (used in nine studies) In the case of four studies, focus groups were carried out in addition to interviews and observations.28–31 Three priorities require actionable findings at strategic time ments and rapid ethnographies.5–7 The points Despite methodological advancement, there Vindrola-Padros B Qual Saf 2017;0:1–10 doi:10.1136/bmjqs-2017-007226 development of rapidBMJ research methodolcontinue to be challenges in the implementation of rapidVindrola-Padros C, ethnographies concerning sampling, the interpretation of ogies has been influenced by an acknowlfindings and management of field research The purpose edgement of the importance of generating of this review was to explore the benefits and challenges findings within time frames when they of using rapid ethnographies to inform healthcare can still be actionable and used to inform organisation and delivery and identify areas that require improvements in care As McNall and improvement colleagues have argued, ‘the timeliness Methods This was a systematic review of the literature Different rapid research approaches Research Evaluations Participatory rural appraisal (PRA) Real-time evaluations (RTEs) Rapid ethnographic assessment (REA) Rapid feedback evaluations (RFEs) Rapid appraisal Rapid evaluation methods (REM) Rapid assessment procedures (RAP) Rapid cycle evaluations (RCEs) RARE model Rapid rural appraisal (RRA) Short-term ethnographies Quick ethnographies Focused ethnographies Rapid evaluations “Rapid feedback evaluation is an evaluation model that is focused on a particular issue, problem or information need, where evaluative information is needed in a short timeframe” (McNall et al 2004) Rapid evaluations Shrank (2013) • • Anker et al (1993) Provides timely feedback to funding organizations and program staff and care providers Offers support for continuous quality improvement and allows observations of changes over time How is rapid research used? Quick overview of a situation (exploratory, not in-depth, diagnostic purposes) Inform longer research project (preliminary study) Run in parallel with a longer study (strand of mixed-methods study) Explore the findings of a longer study more in-depth Study on its own RAPID RAPID RAPID LONGER LONGER LONGER RAPID Challenges of rapid research Systematic review Table Thematic framework on potential challenges in rapid ethnographies used to inform the research questions Key literature 10 12 10 12 14 10 14 10 12 14 13 14 12 14 Potential challenges/issues that require more research Description of the challenges Research questions guiding this review Inability to capture changes over time, understand What were the main research designs? all relevant social and cultural factors at stake, or conflict and contradictions Representativeness and sample size and Dependency on most accessible informants and What were the sample sizes used in selection loss of multiplicity of voices the study and selection of groups/ participants? How were these justified? Local research assistants are not always available, Who were the data collectors? Why were Use and training of local research they recruited? Was training provided? assistants (research assistants from the have the required skills or willingness to take Were interpreters used? Were data part Training takes time Research undertaken observed field) collectors fluent in the local language? by researchers without an anthropological background might limit the quality of the study Who are the article authors and Lone researcher versus multimembered Multimembered teams can maximise resources team and cover a wider range of expertise Recruitment what are their affiliations? How were might be an issue and clear roles in the field need research teams defined? How many field researchers were used and what was the to be outlined justification? Did the research team have prior research ‘In and out’ researcher versus long-term New researchers might get more attention, but experience in the study area? Does the engagement lack familiarity with the study area Prolonged research team report the establishment engagement often increases credibility and of relationships with potential research internal validity Prolonged engagement might participants prior to the study? also lead to stronger relationships between research participants and the field researchers Does the article include reflections on Time for reflexivity The rapid study time frames might not allow researchers to critically analyse the position they the authors’ positionality or factors that play in the field site and their role in the collection might have influenced data collection and analysis? and analysis of data Research governance, and ethical Time pressures should not deter researchers from What were the research governance principles undergoing the required governance and informed processes? Was the study approved by an ethics committee? Did the researchers consent processes follow an informed consent process? ‘Breadth’ versus ‘depth’ in data collection The form was changed based on the findings from the RESULTS Study design Choosing the approach ◦ Aim (evaluation, exploratory, diagnostic purposes) ◦ Research questions ◦ How participatory? ◦ How structured? ◦ Resources? ◦ Team or lone researcher? ◦ Who will use the findings? ◦ When are findings needed? (one timepoint or regular feedback?) ◦ Who might be impacted/benefit from the findings? Appraisals and ethnographies Evaluations Participatory rural appraisal (PRA) Real-time evaluations (RTEs) Rapid ethnographic assessment (REA) Rapid feedback evaluations (RFEs) Rapid appraisal Rapid evaluation methods (REM) Rapid assessment procedures (RAP) Rapid cycle evaluations (RCEs) RARE model Rapid rural appraisal (RRA) Short-term ethnographies Quick ethnographies Focused ethnographies Data collection Systematic review Table Types of interviews, observations, focus groups and mapping processes Interviews Observations Focus groups Mapping and surveys Type Articles Type Articles Type Articles Semistructured Ethnographic observations 11 43 60 Focus groups 28–30 30–34 36 37 Health walks 40 41 43 60 35 Structured 11 27 29 30 32 33 35 41 43 50 60 62 28 44 Natural groups 30 Field surveys 27 38 Unstructured 28 Video observations Participant observation 24 30 37 41 62 Informal focus groups Opportunistic or rapid ‘street intercept’ In-depth 11 36 Photographic documentation of spaces Mapping and 36 geocoding Direct observation 28 36 42 24 28 31 33 37 Shadowing 39 40 61 Key informant/ 36 25 Observations expert (specific type not specified) Informal discussions 32 Tour observations Conversational 26 Clinical interviews observations Video-cued 44 interviews Articles 11 26 29 31 27 studies were based on focus group and interviews.32–34 Two studies combined interviews, observations, 35 36 Type researchers or practitioners with prior experience in the area.31 42 43 Sampling in two studies was informed Feedback loops Design the dissemination strategy before the study begins ◦ Decide where the feedback loops will take place based on when findings are needed ◦ Engage with relevant stakeholders When establishing the study timeline, build in time for checking with study participants Ethical approval-potential delays ◦ Study must follow the same ethical guidelines as any other study ◦ Service evaluation ◦ Linked to a wider project ◦ Low risk/streamlined review Feedback loops Rapid Feedback Evaluation (RFE) Zakocs et al (2015) McNall et al (2004) 1 Collect existing data on program performance Schneeweiss et al (2015) Review research findings Collect new data on program performance Translate findings into Collect data (first actions round) Vindrola-Padros et al (2020) Rapid Cycle Evaluation (RCE) Clarify intent: Purpose, questions, study protocol Collect “good enough” data: Collect and analyze data quickly Produce brief memo: Draft concise memo with main findings Engage in reflective debrief: Discuss findings with project team Decide if more information is needed, take action or take no action Repeat feedback loops (steps 2-5) Skillman et al (2019) Develop an analytic framework 3.Evaluate preliminary data Make judgements based on findings Analyze data and develop codebook Share Initiate findings/recommendations implementation with project team Report findings Develop and analyze alternative designs for fullscale evaluation Collect data (second round) adding quantitative data Assist in developing policy and management decisions Make changes in implementation (if needed) Repeat cycle (steps 35) Data analysis Data analysis ◦ Use of tables, mind mapping, drawings, etc ◦ Adapting methods to reduce time for transcription and coding (i.e RITA) ◦ Analysis as a team Tools to reduce time required for data analysis Analysis from interview recordings Interview notes Voice recognition software Selected transcription Mind mapping Using mind mapping in analysis of qualitative data, J Burgess-Allen and V Owen-Smith 409 Figure Sample mind map Data analysis ◦ Will you use other researchers to crosscheck data or analyse data as a team? ◦ Can you streamline data analysis in any way? (data collection and analysis in parallel, targeted transcription) ◦ Can you include participants in the data analysis process? ◦ Can you ‘digest’ data throughout data collection? Rapid evaluation: QI training and support Mixed-methods design • Qualitative • Health economics Data collection • Interviews • Observations • Documentary analysis Data analysis • Framework analysis Fieldwork guide Study stage Time into study Type of dissemination Purpose Scoping/ familiarisation Week 1 Sharing RQs and study outline Agree purpose Face to face of the study meeting Scoping/ familiarisation Week 2 or 3 Sharing final study scope Final agreement on study design and dissemination plan Highlight emerging findings Highlight emerging findings Fieldwork and Month 2 analysis Fieldwork and Month 3-4 analysis Short memos (monthly or weekly) Short memos (monthly or weekly) Final analysis Month 5 Report draft Writing Month 6 Final report and presentation Format Email or face to face meeting Type of stakeholder Intervention designers, implementers and users Intervention designers, implementers and users Email Implementers Face to face Intervention designers, implementers and users Implementers Cross-check Email or face early to face interpretations Final sharing of Face to face findings and development of recommendati ons Intervention designers, implementers and users Strategies used to deliver timely findings 1) Strategies to reduce evaluation duration 2) Strategies to increase engagement 3) Strategies for quality control Data analysis and collection in parallel Multiple stages of coding Synthesizing data in manageable formats Eliminate transcription Use large teams of researchers Establish a ’core’ group of stakeholders to share findings/seek feedback • Cross-checking of data during data collection and analysis • • • • • • Vindrola-Padros et al (2020) Reflections on the Exemplar Ward Programme Evaluation Wave 2 Wave 1 Design Pilot Programme tested on 3 wards Wave 3 Rollout Ward staff Local Ward Ward attend receives meetings to submits launch event data pack answer their questions narrative Ward Debriefing inspection session to or panel share results Ward staff attend learning event to inform rollout Wards develop improvement projects and attend QI training ? Qualitative Health Research Network (QHRN) training Upcoming training March 2020 Dates March 2020 March 2020 4 March 2020 July 2020 Courses Introduction to rapid qualitative research Introduction to rapid evaluations Introduction to ethnography in healthcare Follow us on Twitter at @UCL_QHRN @CeciliaVindrola Dates 1 July 2020 2 July 2020 3 July 2020 Courses Process evaluations in healthcare Participatory research methods Introduction to rapid ethnography Join our mailing list: http://www.ucl.ac.uk/qualitative-health-research-network/contact-us