A systematic review of clinical decision support systems for antimicrobial management Are we failing to investigate these interventions appropriately? Accepted Manuscript A systematic review of clinic[.]
Accepted Manuscript A systematic review of clinical decision support systems for antimicrobial management: Are we failing to investigate these interventions appropriately? Dr Timothy M Rawson, Luke SP Moore, Bernard Hernandez, Esmita Charani, Enrique Castro-Sanchez, Pau Herrero, Benedict Hayhoe, William Hope, Pantelis Georgiou, Alison H Holmes PII: S1198-743X(17)30125-8 DOI: 10.1016/j.cmi.2017.02.028 Reference: CMI 876 To appear in: Clinical Microbiology and Infection Received Date: 17 November 2016 Revised Date: 23 February 2017 Accepted Date: 25 February 2017 Please cite this article as: Rawson TM, Moore LS, Hernandez B, Charani E, Castro-Sanchez E, Herrero P, Hayhoe B, Hope W, Georgiou P, Holmes AH, A systematic review of clinical decision support systems for antimicrobial management: Are we failing to investigate these interventions appropriately?, Clinical Microbiology and Infection (2017), doi: 10.1016/j.cmi.2017.02.028 This is a PDF file of an unedited manuscript that has been accepted for publication As a service to our customers we are providing this early version of the manuscript The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final form Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain ACCEPTED MANUSCRIPT A systematic review of clinical decision support systems for antimicrobial management: Are we failing to investigate these interventions appropriately? RI PT Timothy M Rawson1, Luke SP Moore1, Bernard Hernandez2, Esmita Charani1, Enrique Castro-Sanchez1, Pau Herrero2, Benedict Hayhoe3, William Hope4, Pantelis Georgiou2, Alison H Holmes1 Affiliations: National Institute for Health Research Health Protection Research Unit in Healthcare Associated SC Infections and Antimicrobial Resistance, Imperial College London, Hammersmith Campus, Du Cane Road, Department of Electrical and Electronic Engineering, Imperial College London, South Kensington Campus, London, SW7 2AZ, United Kingdom School of Public Health, Imperial College London, South Kensington Campus, London, SW7 2AZ, United Kingdom Department of Molecular and Clinical Pharmacology, University of Liverpool, Liverpool, L69 3GE, United Kingdom *Corresponding author: TE D M AN U London W12 0NN United Kingdom EP Dr Timothy M Rawson, National Institute for Health Research Health Protection Research Unit in Healthcare Associated Infections and Antimicrobial Resistance, Imperial College London, Hammersmith Campus, Du Cane AC C Road, London W12 0NN United Kingdom Email: timothy.rawson07@ic.ac.uk Telephone: 02033132732 Running Title: Antimicrobial decision support Search terms: Decision algorithms, antimicrobial resistance, antimicrobial stewardship, electronic support ACCEPTED MANUSCRIPT Abstract Objectives Clinical decision support systems (CDSS) for antimicrobial management can support clinicians to optimise antimicrobial therapy We reviewed all original literature (qualitative and quantitative) to understand the current scope of CDSS for antimicrobial management and analyse existing methods RI PT used to evaluate and report such systems Method PRISMA guidelines were followed Medline, EMBASE, HMIC Health and Management, and Global Health databases were searched from 1st January 1980 to 31st October 2015 All primary research studies describing CDSS for antimicrobial management in adults in primary or secondary care were SC included For qualitative studies, thematic synthesis was performed Quality was assessed using Integrated quality Criteria for the Review Of Multiple Study designs (ICROMS) criteria CDSS M AN U reporting was assessed against a reporting framework for behaviour change intervention implementation Results Fifty-eight original articles were included describing 38 independent CDSS The majority of systems target antimicrobial prescribing (29/38;76%), are platforms integrated with electronic medical records TE D (28/38;74%), and have rules based infrastructure providing decision support (29/38;76%) On evaluation against the intervention reporting framework, CDSS studies fail to report consideration of the non-expert, end-user workflow They have narrow focus, such as antimicrobial selection, and use proxy outcome measures Engagement with CDSS by clinicians was poor EP Conclusion Greater consideration of the factors that drive non-expert decision making must be considered when AC C designing CDSS interventions Future work must aim to expand CDSS beyond simply selecting appropriate antimicrobials with clear and systematic reporting frameworks for CDSS interventions developed to address current gaps identified in the reporting of evidence Abstract: 247 Manuscript: 4303 ACCEPTED MANUSCRIPT Introduction In response to the global threat of antimicrobial resistance (AMR),[1] a range of antimicrobial stewardship (AMS) programmes have been developed that tend to focus on reducing high rates of inappropriate antimicrobial use described widely across care pathways and clinical specialties.[2–5] RI PT An important facet of this approach has been the development of decision support mechanisms for those who prescribe antimicrobials These interventions are based on evidence that the majority of antimicrobial prescribing is done by individuals who are not experts in infection management and SC therefore, may have a limited understanding of antimicrobials and the evidence on AMR.[6–9] To address this challenge, electronic clinical decision support systems (CDSS) have been devised with M AN U the aim of providing the prescriber with easy and rapid access to information, which is required to make therapeutic decisions at the point-of-prescription.[10,11] With the expanding use of electronic medical records (EMR) and developments in information technology, the role of CDSS has become an area of great interest with a wide variety of interventions now labelled as such In medicine, CDSS have been demonstrated to reduce medical errors and improve the quality of TE D healthcare provided by promoting the practice of evidence based medicine.[12] Therefore, it seems logical that in a field where we have a need to improve the practice of evidence based antimicrobial management CDSS may be an effective avenue to promote this CDSS were first developed to EP support antimicrobial management in the 1980’s and since then several systematic reviews of AC C experimental and quasi-experimental studies have explored the potential of CDSS to improve antimicrobial management at different levels of care.[11,13,14] However, these reviews have only tended to focus on single care pathways, such as the hospital setting or primary care and fail to include qualitative studies evaluating CDSS Through these reviews, a minor to moderate benefit of CDSS for optimising antimicrobial management has been demonstrated with a number of gaps in knowledge remaining to be answered.[11,13,14] We performed a systematic review of original literature (qualitative and quantitative) to try to understand the current scope of CDSS for antimicrobial management and analyse existing methods used to evaluate and report such systems This will be used to create a pragmatic picture of CDSS for antimicrobial management and produce ACCEPTED MANUSCRIPT recommendations for future research and interventions, which may optimise the effectiveness of AC C EP TE D M AN U SC RI PT CDSS reporting within this field ACCEPTED MANUSCRIPT Method Search strategy This systematic review was performed following PRISMA guidelines.[15] The Medline, EMBASE, HMIC Health and Management, and Global Health databases were searched from 1st January 1980 to RI PT 31st October 2015 using the search criteria described in Supplementary Table Search criteria were broad and intended to capture all information technology products which have been labelled as “clinical decision support systems” for antimicrobial management SC Study selection M AN U Prospective and retrospective articles in English that reporting original research on clinical patient or product outcomes of CDSS for antimicrobial management in primary and secondary care were included Randomised (including cluster), observational (including case-control, cross-sectional, cohort, before-after, and interrupted time series), diagnostic, development reports (including data), mixed-methods, and qualitative (survey, semi-structured interview, or ethnographic) studies were all TE D included Interventions focusing predominantly on critical care were excluded as these CDSS are often used by doctors in a controlled setting, where close working relationships with infection specialists has been demonstrated to significantly improve patient outcomes.[16–20] Therefore, these EP CDSS interventions may not be utilised in a similar way to other areas, where they are often used to supplement this expert support Moreover, CDSS designed specifically for paediatric antimicrobial AC C management were excluded given the differences in prescribing compared to adult antimicrobial management If studies did not present original data, they were not carried forward Two authors (TMR plus either LSPM, EC, or ECS) independently screened study titles and abstracts against the inclusion and exclusion criteria described above and extracted data (described below) On completion of this process, inter-rater reliability was assessed by calculating Cohen’s kappa statistic Where there was disparity between opinions, the authors discussed these to reach a consensus Decision support system grouping & data extraction ACCEPTED MANUSCRIPT Following study selection, two authors (TMR plus either LSPM, EC, or ECS) independently reviewed each study, grouping those for each CDSS described and extracting data Data recorded included the characteristics of the CDSS (decision support provided, platform, and system infrastructure), the study design(s) used to evaluate the CDSS, and any comparator used Primary and secondary outcomes RI PT were recorded when presented in the manuscript, as was the outcome of these Qualitative studies were analysed using a thematic synthesis approach.[21] Qualitative studies were synthesised using an inductive approach with line by line coding of the text to draw out descriptive themes (carried out by one author, TMR) Manuscripts were then re-coded and discussed by the researchers (TMR, LSPM, SC EC, ECS) to agree upon analytical themes from within the text.[21] Finally, the CDSS systems were evaluated against an analytical framework adapted from the Stage Model of Behaviour Intervention M AN U Development[22] and the Medical Research Council’s Developing and Evaluating complex interventions guidance.[23] The framework is outlined in Table The four domains of the framework used to evaluate the CDSS were (i) development; (ii) feasibility and piloting; (iii) evaluation of the system; and (iv) implementation When included within reporting of such systems TE D these criteria will allow the reader to understand holistically the rationale for why and how a CDSS was developed and how its effectiveness was evaluated.[22,23] Quality assessment EP Given the heterogeneity of studies included within this review, we opted to use the Integrated quality AC C Criteria for the Review Of Multiple Study designs (ICROMS) criteria.[24] ICROMS aims to facilitate the review of behaviour change interventions in the field of infection, such as clinical decision support tools It facilitates the review of multiple study designs that includes Randomised Control Trials (RCT’s) (including cluster-RCT’s), cohort, before-after, and interrupted time series studies, as well as qualitative studies.[24] For studies that were not included in ICROMS, we quality assessed these using validated criteria from the literature These were the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) criteria for cross-sectional studies and casecontrol studies;[25] the Consolidated Health Economic Evaluation Reporting Standards (CHEERS) criteria for economic evaluations;[26] and the Standards for Reporting Diagnostic Accuracy Studies ACCEPTED MANUSCRIPT (STARD) criteria for diagnostic studies.[27] For development reports, we were unable to assign a quality criterion (and were therefore labelled as high risk of bias) Using these quality criteria, studies were scored as advised within ICROMS.[24] A study was awarded points if a specific criterion was met, points if the criterion was not met, and point if it RI PT was unclear The sum of the quality criterion was then given to represent a global quality score for each study Based on recommendations from ICROMS scores 80% of the total score for that study type were labelled low risk of bias / high M AN U reliability (“low risk”) Given our objectives were to capture all relevant literature, we did not exclude data based on the quality of evidence provided Summary measures Following extraction and synthesis, data were reviewed by all researchers to identify current barriers TE D and facilitators to success in practice All major primary outcome measures described within the studies were grouped and classified into either patient level, prescriber level, or unit/hospital level outcomes These were tabulated and the level of evidence for overall achievement of each primary EP outcome demonstrated within the literature for these groups was graded using Grading of AC C Recommendations Assessment, Development and Evaluation (GRADE) criteria.[28] ACCEPTED MANUSCRIPT Results Study selection and characteristics Figure describes the screening and eligibility checking process which was undertaken An initial electronic search identified 402 individual titles and abstracts for screening Of these, 131/402 (33%) RI PT abstracts were carried forward for eligibility screening and 58/131 (44%) were included in the review Cohen’s kappa for agreement was 0.88 These 58 studies described 38 different CDSS Table summarises the attributes of the CDSS identified Supplementary Table outlines the full evaluation SC of the 38 CDSS On assessment of the risk of bias of included studies using ICROMS, the majority of studies in primary care were found to be low to medium risk (7/18;39% and 8/18;44%, respectively), and 22/40;55%, respectively) of bias M AN U whereas the majority of studies reported from secondary care were medium to high risk (15/40;38% Decision support systems reported in the literature The majority of CDSS in the literature target antimicrobial prescribing (29/38;76%) The 11 systems TE D focused on antimicrobial prescribing in primary care provided decision support for specific syndromic presentation in adults The conditions targeted were acute respiratory tract infections (ARIs), with two CDSS also including urinary tract infections (UTIs).[29–46] In contrast, systems supporting EP antimicrobial prescribing in secondary care targeted broader populations with interventions tending to focus on empirical and prophylactic antimicrobial prescribing rather than individual syndromes AC C (exceptions included, pneumonia, UTI, MRSA, Clostridium difficile infection).[47–85] Other decision support provided by CDSS for antimicrobial management included; electronic prompts / alerts (7/38; 18%); optimising antimicrobial dosing (3/38; 8%); supporting antimicrobial de-escalation (2/38; 5%); surveillance (2/38; 5%); and prescriber feedback (1/38: 3%) Several platforms for delivering CDSS were reported, including systems being integrated into hospital electronic medical record (EMR) (28/38;74%), via web-based platforms (5/38;13%), via personal digital assistants (3/38;9%), and as standalone software (2/38;5%) The reported infrastructure providing decision support was predominantly rules based (29/38;76%) There were also a number of ACCEPTED MANUSCRIPT machine learning tools reported including; use of neural networks (2/38;5%), association rule learning algorithms (1/38;3%) and predictive models (1/38;3%) These were all reported in secondary care Analysis of CDSS development & pilot and feasibility testing domains On comparison with domains and of our defined analytical framework (Table 1), a paucity of RI PT evidence exists to describe stakeholder involvement in the development processes for CDSS This includes a lack of evidence supporting pre-intervention stakeholder analysis, evidence exploring user decision processes, and how interventions will fit into routine clinical workflow For example, SC Andreassen and colleagues describe the development of an intelligent CDSS using Causal Probabilistic Networks (TREAT) for use in secondary care.[67] Within this report, much detail is M AN U placed on the construction of pathophysiological model for the diagnosis of infection and antimicrobial selection However, no evidence is provided to describe prescriber’s decision pathways and how the system will integrate into this process in clinical practice In contrast, McDermott and colleagues report during the development of the eCRT study engagement with a small number of stakeholders (n=33) in the design of the intervention based on behaviour change theories.[42] TE D However, post implementation review of this intervention identified problems with variations in individuals prescribing behaviours, lack of end-user engagement with implementation, and rigidity of the guidelines incorporated limiting the use of the system.[40] These aspects of the clinician’s EP decision making process were not explored during the development phase This observation is AC C supported by Zaidi and colleagues, who highlighted workflow related issues of their CDSS with junior medical staff during the post-intervention qualitative evaluation of their product.[79] Analysis of evidence domain For analysis of framework domain 3, examination of experimental design studies in primary care reveals primary outcome measures were heterogeneous and tended to focus on rates of prescribing of antibiotics either overall or for a defined syndrome These studies demonstrated zero to minor clinically significant improvements in antimicrobial use.[29–31,37,39,41,42] Failures in demonstrating primary outcome measures were often reported as being due to the intention-to-treat ...ACCEPTED MANUSCRIPT A systematic review of clinical decision support systems for antimicrobial management: Are we failing to investigate these interventions appropriately? RI PT Timothy M Rawson1,... implementation of a CDSS in an Australian hospital.[78,79] However, of note was the paucity of information available describing mechanisms to support implementation and adoption of CDSS as well as a. .. answered.[11,13,14] We performed a systematic review of original literature (qualitative and quantitative) to try to understand the current scope of CDSS for antimicrobial management and analyse existing