This lecture includes these contents: Critical appraisal of systematic reviews, appraise a systematic review for validity, relative risk reduction, number needed to treat, why odds ratios,... Invite you to consult this lecture.
Critical Appraisal of Systematic Reviews Douglas Newberry Systematic Reviews — or How to make a Monkey out of EBM without hardly trying! Systematic Reviews: Objectives: • Appraise a systematic review for validity • Discuss Meta Analysis / use Odds Ratios • Obtain Number Needed to Treat (NNT) from Odds Ratios • Consider clinical implications of a Systematic Review {including when to bin it instead!} We can see further than our forbearers because we stand on the shoulders of Giants {and have better spectacles} • these ideas are cribbed unashamedly from friends, books & previous courses Systematic Reviews: What are your Objectives: What you want to cover? Please interject with helpful questions! Did I really want a systematic review? (but please not pretend) • admit your ignorance — expert review or consensus guidelines > broad introduction, cover many areas (class C evidence) • if the question is important > formulate it! • Systematic review > narrow but rigorous focus Systematic Reviews — Where I start: • Start with your (or 3) part clinical question! • Is a systematic review a sensible approach? • Does THIS systematic review address MY question? • Is it a systematic review at all? Is it a systematic review? does it: • define a four part (answerable) clinical question? • combine Randomized Controlled Trials (RCT’s)? • describe PRE-DEFINED search methods? • PRE-DEFINED inclusion criteria? • PRE-DEFINED methodological exclusion criteria? Sceptical View? Take it with a grain of salt: • transparent declaration of funding of work? • Drug Company sponsorship of Reviews vs Methodological quality>Cochrane review! • who employs the authors? • open discussion of existing controversy & commercial gain? • Don’t waste salt on your food, keep it for your reading! Meta analysis — combine what with what? • Low Molecular Weight Heparin (LMWH) in hip surgery — begin before or after the operation? • meta analysis of placebo controlled RCT’s of heparin in hip surgery >> • pre-op & post-op LMWH vs placebo • post-op LMWH Vs placebo • pre-operative >> less intra-op bleeding?? Our patient wants an absolute Risk Reduction (ARR): • is a 40% reduction in Cardiac Risk worth taking pills daily for 10 years?? >vote! • if I have a 30% risk of MI or death {30 out of 100 people like me will suffer MI or death} in next 10 years > 40% RRR >> only 18 out of 100 will have MI or death ARR = 12 out of 100! >>I like that! • BUT if I have a 1% risk in 10 years, 40% less is a 0.6% risk >> hardly different! Number Needed to Treat (NNT) (very trendy but tricky): • only defined for specific prevalencePatient’s Expected Event Rate=PEER! • only defined for a specific intervention! • only defined for a specific outcome! – eg Pravastatin™ 40 mg nocte x10 years, in a 65 year old male, ex-smoker with high BP and Diabetes, to reduce MI or Death • NNT is the inverse of Absolute Risk Number Needed to Treat (NNT) for previous example: • 12 fewer MI or death in 10 years per 100 persons treated: ARR=12/100 • NNT = 1/(12/100)=100/12= 8.3 • But the same Relative Risk Reduction (RRR) of 40% with a low prevalence: • 0.4 fewer MI/death per 100 treated, ARR=0.4/100 • NNT = 1/(0.4/100) = 100/0.4 = 250! Why Odds Ratios? > compare results of different studies • consider 2x2 table: • RRR is (a-b/a) — but you can only go in rows within same study! • Odds ratio is (a/c)/(b/d) = ad / bc — the individual ratios are in columns, and therefore are independent of the prevalence which is different in different studies • must use odds ratios to combine RCT’s Odds Ratio (OR) to NNT — is the improvement worth the trouble? • 1>OR>0, lower the OR = better the treatment (Rx) >> lower NNT • for any OR, NNT is lowest when PEER=0.5 • estimate the PEER (patient’s risk) • apply the OR to get patient's NNT Convert PEER & OR to NNT: Control Event Rate (CER) {apply PEER here} Odds Ratio (OR) CER 0.9 0.7 0.5 0.1 110 36 21 0.5 38 11 0.9 101 27 12 Formula used in the table: NNT= 1 {PEER * (1OR)} (1PEER)*(PEER)*(1OR) Table induced nausea! • lower OR >> lower NNT • Patient needs to be at risk (non-trivial PEER) in order for risk reduction to be worth the effort • for any OR, NNT lowest when PEER=0.5 • more effective treatment > lower NNT • BUT are your patient’s values satisfied by the intervention and its sequelae? Subgroup analysis: Sceptical unless: • the subgroups make biological and clinical sense? • the differences are both clinically & statistically significant? • was a-priori hypothesis (before this data)? • other evidence supports these sub-groups? • few (OK) or many (nix) sub-group analyses? Any Questions? Summary 1: Set your goals • define your (or 3) part question • you want a true systematic review? • does this narrow review address my question? • PRE-DEFINED search, inclusion, exclusion! Summary 2: Be Sceptical! • • • • • look for bias, conflict of interest critical appraisal of primary studies? consistent results? if not, why not? does our patient fit the groups studied? does it matter to our patient? Summary 3: Risks that matter • Absolute risk > estimate the Patient Expected Event Rate (PEER) • obtain Relative Risk Reduction (RRR) or Odds Ratio (OR) from a Meta-analysis • plug into a table to estimate Number Needed to Treat (NNT) Summary 4: Sceptical & common sense! • beware of post-hoc sub-group analysis, especially if multiple • step back and consider if the systematic review really related to our patient’s situation (PEER), culture and expectations? • not loose sight of common sense! Coffee Now! • Small Groups Afterwards .. .Systematic Reviews — or How to make a Monkey out of EBM without hardly trying! Systematic Reviews: Objectives: • Appraise a systematic review for validity • Discuss... grain of salt: • transparent declaration of funding of work? • Drug Company sponsorship of Reviews vs Methodological quality>Cochrane review! • who employs the authors? • open discussion of existing... Systematic review > narrow but rigorous focus Systematic Reviews — Where I start: • Start with your (or 3) part clinical question! • Is a systematic review a sensible approach? • Does THIS systematic