Dedicated to our son, Dilip, whose inquisitive nature reminds us that we should never stop asking questions © 2012 PasTest Ltd Egerton Court Parkgate Estate Knutsford Cheshire WA16 8DX Telephone: 01565 752000 All rights reserved No part of this publication may be reproduced, stored in a retrieval system, or transmitted, in any form or by any means, electronic, mechanical, photocopying, recording or otherwise without the prior permission of the copyright owner First edition 2006, Second edition 2009, Third edition 2012 ISBN: 905635 818 ISBN: 978 905635 818 eISBN: 978 908185 846 A catalogue record for this book is available from the British Library The information contained within this book was obtained by the authors from reliable sources However, while every effort has been made to ensure its accuracy, no responsibility for loss, damage or injury occasioned to any person acting or refraining from action as a result of information contained herein can be accepted by the publisher or the authors PasTest Revision Books and Intensive Courses PasTest has been established in the field of undergraduate and postgraduate medical education since 1972, providing revision books and intensive study courses for doctors preparing for their professional examinations Books and courses are available for: Medical undergraduates, MRCGP, MRCP Parts and 2, MRCPCH Parts and 2, MRCS, MRCOG Parts and 2, DRCOG, DCH, FRCA, Dentistry For further details contact: PasTest, Freepost, Knutsford, Cheshire, WA16 7BR Tel: 01565 752000 Fax: 01565 650264 www.pastest.co.uk enquiries@pastest.co.uk Text prepared by Carnegie Book Production, Lancaster Printed and bound by CPI Group (UK) Ltd, Croydon, CR0 4YY About the authors Introduction to the third edition Acknowledgements SECTION A – EVIDENCE-BASED MEDICINE Introducing critical appraisal Formulating a question Search strategies The journal Organisation of the article SECTION B – APPRAISING THE METHODOLOGY Overview of methodology The clinical question Introducing study designs Observational descriptive studies Observational analytical studies Experimental studies Other types of study Research pathway Populations and samples Bias Confounding factors Restriction Matching Randomisation Concealed allocation The placebo effect Blinding Endpoints Validity Reliability SECTION C – INTERPRETING RESULTS Basic statistical terms Epidemiological data Intention-to-treat analysis Risks and odds Types of data Measuring data Data distributions Describing categorical data Describing normally distributed data Describing non-normally distributed data Inferring population results from samples Comparing samples – the null hypothesis Comparing samples – statistical tests Non-inferiority and equivalence trials Correlation and regression Systematic reviews and meta-analyses Heterogeneity and homogeneity Publication bias Interim analysis SECTION D – USING CHECKLISTS Introduction to checklists Aetiological studies Diagnostic or screening studies Treatment studies Prognostic studies Economic studies Qualitative research Guidelines SECTION E – APPLICABILITY The hierarchy of evidence Critical thinking SECTION F – CRITICAL APPRAISAL IN PRACTICE Health information resources Presenting at a journal club Taking part in an audit meeting Working with pharmaceutical representatives Further reading Answers to self-assessment exercises A final thought Index Dr Narinder Kaur Gosall BSc (Hons) PhD Director, Superego Cafe Limited Narinder Gosall studied in Liverpool and gained a PhD in neuropathology after investigating the role of the phrenic nerve in sudden infant death syndrome and intrauterine growth retardation After working as a university lecturer she joined the pharmaceutical industry She worked in a variety of roles, including as a Medical Liaison Executive and as a Clinical Effectiveness Consultant for Pfizer Limited She has extensive experience in teaching critical appraisal skills to healthcare professionals and is an international speaker on the subject She is the editor of the online course at www.criticalappraisal.com Dr Gurpal Singh Gosall MA MB BChir MRCPsych Consultant General Adult Psychiatrist, Lancashire Care NHS Foundation Trust Director, Superego Cafe Limited Gurpal Gosall studied medicine at the University of Cambridge and Guy’s and St Thomas’s Hospitals, London He worked as a Senior House Officer in Psychiatry in Leeds before taking up a post as Specialist Registrar in the North West He now works as a Consultant Psychiatrist, looking after patients in the Psychiatric Intensive Care Units at the Royal Blackburn Hospital and Burnley General Hospital He has a long-standing interest in teaching and runs a popular website for psychiatrists, Superego Cafe, at www.superego-cafe.com Learning the skill of critical appraisal is like learning a foreign language – wherever you start, you come across unfamiliar words and concepts However, persistence pays off and, like speaking a foreign language, the earlier it is mastered and the more it is used, the easier critical appraisal becomes Critical appraisal skills are now as much a part of the clinician’s armoury as the ability to diagnose conditions and prescribe treatments Critical appraisal skills allow clinicians to prioritise evidence that can improve outcomes Such is the importance of acquiring these skills that critical appraisal is now routinely tested in medical, dental and nursing exams We wrote the first edition of this book years ago to explain critical appraisal to the busy clinician Our aim has always been for the book to be the one-stop solution for all clinicians Based on our teaching experience, we took a unique back-to-basics approach that provided a logical and comprehensive review of the subject This new edition expands on the last edition with updated information, new chapters and more help with difficult topics We hope that by reading this book you will start reading and appraising clinical papers with more confidence The language of evidence-based medicine is not as foreign as you might think NKG, GSG 2012 Endnotes Cochrane AL Effectiveness and Efficiency: Random Reflections on Health Services London, Royal Society of Medicine Press, 1999 Sackett DL, Richardson WS, Rosenberg W, Haynes RB Evidence-based Medicine: How to Practise and Teach Evidence-based Medicine London, Churchill Livingstone, 1997 Schwartz D, Lellouch J Explanatory and pragmatic attitudes in therapeutical trials Journal of Chronic Diseases 1967, 20, 637–48 Journal Citation Report (JCR) Philadelphia, USA Thomson Institute for Scientific Information, 2005 JCR provides quantitative tools for ranking, evaluating, categorising and comparing journals Murch SH, Anthony A, Casson DH, et al Retraction of an interpretation Lancet 2004, 363, 750 General Medical Council Good Practice in Research and Consent To Research (Supplementary guidance) London, GMC, 2010 McBride WG Thalidomide and congenital abnormalities Lancet 1961, 2, 1358 Bradford Hill A The environment and disease: association or causation? Proceedings of the Royal Society of Medicine 1965, 58, 295–300 Rothman KJ, Greenland S Causation and causal inference in epidemiology American Journal of Public Health 2005, 95(Suppl 1), S144–50 10 Moher D, Schulz KF, Altman DG The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomised trials Lancet 2001, 357, 1191–4 11 Davidoff F, Batalden P, Stevens D, Ogrine G, Mooney SE Publication guidelines for quality improvement studies in health care: evolution of the SQUIRE project BMJ 2009, 338, a3152 12 Piaggio G, Elbourne DR, Altman DG, Pocock SJ, Evans SJ Reporting of noninferiority and equivalence randomized trials: An extension of the CONSORT statement JAMA 2006, 295, 1152– 60 13 Stroup DF, Berlin JA, Morton SC, et al Meta-analysis of observational studies in epidemiology: a proposal for reporting Meta-analysis Of Observational Studies in Epidemiology (MOOSE) group JAMA 2000, 283, 2008–12 14 Hurwitz B Legal and political considerations of clinical practice guidelines BMJ 1999, 318, 661–4 x tables (contingency tables) ref 1, ref 2, ref 3, ref self-assessment exercises ref 1, ref a priori (primary) hypothesis ref absolute risk (AR) ref absolute risk reduction (ARR) ref 1, ref 2, ref 3, ref negative ref abstracts ref 1, ref accuracy (validity) ref 1, ref 2, ref 3, ref adaptive randomisation ref adverse drug reactions ref 1, ref aetiological studies (causation) ref 1, ref 2, ref AGREE Instrument (for guidelines) ref allocation (randomisation methods) ref 1, ref 2, ref 3, ref alternative hypothesis ref 1, ref see also null hypothesis alternative-form reliability ref analysis of variance (ANOVA) ref 1, ref 2, ref AR see absolute risk ARR see absolute risk reduction articles quality assessment ref 1, ref 2, ref 3, ref structure ref 1, ref attrition bias ref 1, ref 2, ref audit ref 1, ref 2, ref authors ref average see mean; median Bandolier website ref baseline characteristics ref 1, ref Bayes’ theorem ref bell-shaped curve ref Berkson bias ref Bernoulli distribution ref Best Evidence (ACP/BMJ) ref bias in design/execution ref 1, ref 2, ref prevention ref 1, ref 2, ref 3, ref bias in publication ref 1, ref binary data ref 1, ref 2, ref 3, ref binomial distribution ref 1, ref Bland–Altman limits of agreement ref blinding ref 1, ref 2, ref 3, ref block randomisation ref BNI database (British Nursing Index) ref Bonferroni correction ref Boolean operators ref carry-over effects ref case–cohort studies ref case–control studies ref 1, ref 2, ref 3, ref self-assessment exercises ref 1, ref 2, ref case reports ref case series ref CASP (Critical Appraisal Skills Programme) ref categorical (qualitative) data ref 1, ref 2, ref 3, ref 4, ref analytical statistics ref 1, ref 2, ref 3, ref descriptive statistics ref reliability ref causation ref causation studies (aetiological) ref 1, ref 2, ref censored data ref 1, ref censuses ref central limit theorem ref Centre for Evidence-Based Medicine ref CER see control event rate checklists ref 1, ref aetiological studies ref diagnostic studies ref 1, ref economic studies ref prognostic studies ref qualitative research ref treatment studies ref chi-squared ( 2) test ref 1, ref CI see confidence interval CINAHL (Current Index to Nursing and Allied Health Literature) ref citations ref 1, ref 2, ref class effects ref Clinical Evidence (ACP/BMJ) ref clinical questions ref 1, ref 2, ref 3, ref clinical significance ref clinical trials ref 1, ref see also study design cluster analysis ref cluster randomisation ref 1, ref cluster sampling ref Cochrane Collaboration ref 1, ref 2, ref Cochran’s Q ref coefficient of determination (r2) ref coefficient of variation ref 1, ref Cohen’s statistic (κ) ref cohort studies ref 1, ref 2, ref 3, ref self-assessment exercises ref 1, ref 2, ref composite endpoints ref concealed allocation ref concurrent validity ref conferences ref confidence interval (CI) ref 1, ref in forest plots ref conflicts of interest ref confounding factors ref 1, ref 2, ref consent ref CONSORT statement ref constant comparison analysis ref construct validity ref content analysis ref content validity ref contingency tables ref 1, ref 2, ref self-assessment exercises ref 1, ref continuous data ref 1, ref analytical statistics ref descriptive statistics ref 1, ref 2, ref 3, ref reliability ref control event rate (CER) ref 1, ref 2, ref self-assessment exercises ref 1, ref 2, ref 3, ref controlled trials ref see also randomised controlled trials convenience sampling ref convergent validity ref correlation ref intraclass correlation coefficient ref see also regression cost–benefit analysis ref cost–consequences analysis ref cost-effectiveness analysis ref 1, ref cost-minimisation analysis ref cost-of-illness studies ref cost–utility analysis ref covariance ref 1, ref Cox proportional hazards regression ref 1, ref criterion validity ref critical appraisal, overview ref 1, ref Critical Appraisal Skills Programme (CASP) ref Crohnbach’s alpha (α) ref crossover trials ref 1, ref cross-sectional studies ref 1, ref cut-off point optimisation ref data dredging ref databases ref 1, ref interrogation of ref 1, ref death rates ref 1, ref 2, ref degrees of freedom ref diagnostic purity bias ref 1, ref diagnostic studies ref 1, ref 2, ref discrete data (quantitative) ref 1, ref 2, ref distribution ref 1, ref divergent validity ref double-dummy technique ref drop-outs (missing data) ref 1, ref 2, ref ecological studies ref economic analysis ref 1, ref 2, ref EER see experimental event rate effect size ref 1, ref 2, ref 3, ref Effective Health Care bulletins ref effectiveness ref 1, ref efficacy ref 1, ref EMBASE database ref endpoints ref see also missing data epidemiological studies ref 1, ref equivalence studies ref ethics ref 1, ref evidence-based medicine ref 1, ref 2, ref 3, ref levels and grades of evidence ref 1, ref Evidence-Based Medicine Toolkit ref exclusion (attrition) bias ref 1, ref 2, ref exclusion criteria ref 1, ref 2, ref experimental event rate (EER) ref 1, ref 2, ref self-assessment exercises ref 1, ref 2, ref 3, ref experimental studies ref 1, ref 2, ref experts ref 1, ref F value ref face validity ref factor analysis ref factorial studies ref false-negative results (type errors) ref 1, ref 2, ref false-positive results (type errors) ref 1, ref 2, ref 3, ref 4, ref financial (economic) analysis ref 1, ref 2, ref Fisher’s exact test ref fixed randomisation ref fixed-effects model ref focus groups ref forest plots ref 1, ref frequency ref 1, ref 2, ref Friedman’s test ref funnel plots ref Galbraith plots ref Gaussian (normal) distribution ref 1, ref 2, ref geometric mean ref Google Scholar ref GRADE (Grading of Recommendations Assessment, Development and Evaluation) ref grey literature ref 1, ref 2, ref grounded theory ref guidelines ref 1, ref 2, ref hand searching ref 1, ref Hawthorne effect ref hazard rate/ratio ref 1, ref heterogeneity of study results ref hierarchy of evidence ref 1, ref historical control bias ref hospital standardised mortality ratio ref ‘How to read a paper’ series ref hypotheses null/alternative ref 1, ref primary/secondary ref I-squared statistic ref immediacy index ref imputation ref inception cohorts ref incidence ref 1, ref 2, ref inclusion criteria ref 1, ref 2, ref incremental validity ref infectious diseases ref information sources ref 1, ref 2, ref intention-to-treat analysis ref interim analysis ref internet resources ref 1, ref 2, ref 3, ref search strategies ref 1, ref interquartile range ref 1, ref 2, ref inter-rater reliability ref 1, ref interval scales ref 1, ref 2, ref interviewer bias ref interviews, in qualitative research ref intraclass correlation coefficient ref intrarater reliability ref Intute website ref Journal of the American Medical Association (JAMA) ref journal clubs ref journal impact factor ref journals databases ref 1, ref relative quality of ref see also articles Kaplan–Meier survival analysis ref kappa statistic (κ) ref Kendall’s correlation coefficient (τ) ref Koch’s postulates ref Kruskal–Wallis ANOVA test ref kurtosis ref L’Abbé plots ref ‘last observation carried forward’ method ref 1, ref least squares method ref legal status of guidelines ref likelihood ratio ref 1, ref 2, ref 3, ref linear regression ref 1, ref literature review ref 1, ref log–rank test ref log transformation of data ref 1, ref logistic regression ref 1, ref longitudinal studies ref 1, ref Mann–Whitney U test ref MANOVA/MANCOVA (multiple analysis of [co]variance) ref Mantel–Haenszel procedure ref 1, ref manual searching ref 1, ref masking (blinding) ref 1, ref 2, ref 3, ref matching of subjects ref 1, ref McNemar’s test ref mean, arithmetic (x) ref self-assessment exercises ref 1, ref 2, ref 3, ref standard error (SE) ref mean, geometric ref measurement of data scales ref 1, ref types of data ref median ref 1, ref self-assessment exercises ref 1, ref 2, ref 3, ref median survival time ref Medical Subject Headings (MeSH) ref MEDLINE database ref membership bias ref meta-analysis ref 1, ref heterogeneity of data ref publication bias ref meta-regression ref minimisation (adaptive randomisation) ref missing data ref 1, ref 2, ref mode ref 1, ref 2, ref MOOSE group ref morbidity rates/ratios ref mortality rates/ratios ref 1, ref 2, ref multiple analysis ref 1, ref multiple linear regression ref 1, ref multiple testing ref multivariate statistics ref 1, ref 2, ref n-of-1 trials ref National Guideline Clearinghouse (NGC) (USA) ref National Institute for Health and Clinical Excellence (NICE) ref National Library for Health (NLH) ref negative predictive value (NPV) ref 1, ref 2, ref 1, ref negligence ref nested case–control studies ref Neyman (incidence/prevalence) bias ref NHS Centre for Reviews and Dissemination (CRD) ref NHS Economic Evaluations Database (NHS EED) ref NICE ref NNH (number needed to harm) ref 1, ref NNT (number needed to treat) ref 1, ref 2, ref 3, ref 4, ref 5, ref nocebo response ref nominal scales ref 1, ref 2, ref nomograms ref non-inferiority studies ref non-normal distribution ref 1, ref non-parametric tests ref 1, ref normal distribution ref 1, ref 2, ref NPV (negative predictive value) ref 1, ref 2, ref 3, ref null hypothesis ref 1, ref number needed to harm (NNH) ref 1, ref number needed to treat (NNT) ref 1, ref 2, ref 3, ref 4, ref 5, ref observation bias ref 1, ref 2, ref 3, ref 4, ref observational analytical studies ref 1, ref 2, ref 3, ref self-assessment exercises ref 1, ref 2, ref observational descriptive studies ref odds ref 1, ref 2, ref 3, ref 4, ref 5, ref odds ratio (OR) ref 1, ref 2, ref 3, ref 4, ref off-licence data ref one-tailed tests ref online resources ref 1, ref 2, ref 3, ref search strategies ref 1, ref opinion leaders ref 1, ref opportunity cost ref OR (odds ratio) ref 1, ref 2, ref 3, ref 4, ref ordinal scales ref 1, ref 2, ref outcomes (endpoints) ref see also missing data Ovid HealthSTAR database ref P values ref 1, ref 2, ref 3, ref paired t test ref paired/unpaired data ref 1, ref 2, ref 3, ref parametric tests ref 1, ref participant observation ref Pearson’s correlation coefficient (r) ref 1, ref peer review ref PEER value ref per-protocol analysis ref percentiles ref performance bias ref 1, ref period prevalence ref periodicals see journals Peto method ref pharmaceutical company representatives ref pharmacoeconomics ref PICO (analysis of clinical questions) ref pie charts ref placebos ref 1, ref 2, ref point prevalence ref Poisson distribution ref populations in a study see sample population positive predictive value (PPV) ref 1, ref 2, ref 3, ref post-marketing surveillance studies ref post-test probability/odds ref 1, ref 2, ref 3, ref power of a study ref 1, ref PowerPoint presentations ref PPV (positive predictive value) ref 1, ref 2, ref 3, ref pragmatic trials ref precision (reliability) ref 1, ref 2, ref predictive validity ref presentation skills ref pre-test probability/odds ref 1, ref 2, ref 3, ref prevalence ref 1, ref 2, ref primary endpoints ref primary hypothesis ref PRISMA statement ref probability distributions ref probability (risk) ref 1, ref in diagnostic tests ref 1, ref 2, ref 3, ref significance testing ref 1, ref prognostic studies ref 1, ref 2, ref promotional material ref proportional Cox (hazards) regression ref 1, ref prospective studies ref 1, ref PROSPERO register ref PsycNET database ref publication bias ref 1, ref qualitative data see categorical data qualitative research ref 1, ref quality assessment ref 1, ref 2, ref 3, ref quality-adjusted life year (QALY) ref quantitative data ref 1, ref 2, ref analytical statistics ref 1, ref descriptive statistics ref 1, ref 2, ref reliability ref quartiles ref quasi-random allocation ref quasi-random sampling ref random sampling ref random-effects model ref randomisation ref 1, ref 2, ref 3, ref randomised controlled trials (RCTs) ref 1, ref 2, ref 3, ref CONSORT checklist ref range ref 1, ref 2, ref ratio scales ref 1, ref recall bias ref receiver operating characteristic (ROC) curves ref reference ranges ref reflexivity ref regression ref 1, ref 2, ref see also correlation relative benefit increase (RBI) ref relative risk (RR) ref 1, ref 2, ref 3, ref 4, ref relative risk reduction (RRR) ref 1, ref 2, ref 3, ref 4, ref negative ref reliability (precision) ref 1, ref 2, ref reporting bias ref 1, ref research pathway for drug development ref resource allocation see economic analysis response bias (observational) ref 1, ref response bias (sampling) ref retrospective cohort studies ref retrospective studies see case–control studies risk see probability risk factors ref ROC curves ref Rothman’s pies ref RR see relative risk RRR see relative risk reduction safety of drugs ref 1, ref sales representatives ref sample population allocation methods ref 1, ref 2, ref 3, ref critical appraisal of ref matching ref 1, ref restriction criteria ref 1, ref 2, ref sampling bias ref 1, ref 2, ref sampling methods ref 1, ref size of ref 1, ref sampling error ref 1, ref scale types ref 1, ref 2, ref screening studies ref 1, ref SD see standard deviation SE see standard error searching for information ref 1, ref secondary endpoints ref secondary hypothesis ref selection bias ref 1, ref 2, ref 3, ref 4, ref sensitivity (diagnostic tests) ref 1, ref 2, ref 3, ref sensitivity analysis ref 1, ref 2, ref service provision, audit of ref 1, ref 2, ref sham treatment ref SIGLE database (System for Information on Grey Literature in Europe) ref SIGN (Scottish Intercollegiate Guidelines Network) ref 1, ref sign test ref significance testing ref 1, ref simple randomisation ref skewed data ref Spearman’s rank correlation coefficient (ρ) ref specificity ref 1, ref 2, ref 3, ref split-half reliability ref SQUIRE group guidelines ref standard deviation (SD) ref self-assessment exercises ref 1, ref 2, ref 3, ref standard error (SE) ref 1, ref 2, ref 3, ref 4, ref standard normal distribution ref standardisation ref standardised mean difference ref 1, ref STARD statement ref statistical definitions ref statistical significance ref 1, ref 2, ref stratification of data ref stratified randomisation ref stratified sampling ref STROBE checklist ref study design ref bias and ref 1, ref 2, ref 3, ref blinding ref 1, ref and the clinical question ref 1, ref clinical trial phases ref confounders ref 1, ref critical appraisal ref 1, ref hierarchy of evidence ref placebos ref subjects see sample population types ref 1, ref 2, ref subgroup analysis ref 1, ref subjects see sample population superiority trials ref surrogate endpoints ref surveys ref 1, ref survival analysis ref 1, ref systematic reviews ref 1, ref systematic sampling ref t tests ref 1, ref target population ref 1, ref test–retest reliability ref thesauri ref transformation of data ref 1, ref 2, ref treatment studies ref 1, ref 2, ref 3, ref TRIAGE website ref trim and fill method ref TRIP database (Turning Research Into Practice) ref two-tailed tests ref type errors (false positives) ref 1, ref 2, ref 3, ref 4, ref type errors (false negatives) ref 1, ref 2, ref unpaired/paired data ref 1, ref 2, ref 3, ref unpublished data ref 1, ref 2, ref Users’ Guides to the Medical Literature ref validity (accuracy) ref 1, ref 2, ref 3, ref variables ref 1, ref variance (v) ref washout periods ref weighted mean ref Wilcoxon’s matched pairs test ref Wilcoxon’s signed rank test ref worst-case scenario analysis ref Yates continuity correction ref Yellow Card scheme ref z score ref Z statistic ref Z test ref ... allows doctors to assess the research they have found in their search and to decide which research evidence could have a clinically significant impact on their patients Critical appraisal allows doctors. .. nursing exams We wrote the first edition of this book years ago to explain critical appraisal to the busy clinician Our aim has always been for the book to be the one-stop solution for all clinicians... language, the earlier it is mastered and the more it is used, the easier critical appraisal becomes Critical appraisal skills are now as much a part of the clinician’s armoury as the ability to diagnose