This lecture includes these contents: The systematic reviews, methods and procedures, features of narrative reviews and systematic reviews, steps of a cochrane systematic review, synthesis of data,... Invite you to consult this lecture.
Systematic Reviews: Methods and Procedures George A Wells Editor, Cochrane Musculoskeletal Review Group Department of Epidemiology and Community Medicine University of Ottawa Ottawa, Ontario, Canada Meta-analysis: • Meta-analysis is a statistical analysis of a collection of studies • Meta-analysis methods focus on contrasting and comparing results from different studies in anticipation of identifying consistent patterns and sources of disagreements among these results • Primary objective: • Synthetic goal (estimation of summary effect) vs • Analytic goal (estimation of differences) • Systematic Review: – the application of scientific strategies that limit bias to the systematic assembly, critical appraisal and synthesis of all relevant studies on a specific topic • Meta-Analysis: – a systematic review that employs statistical methods to combine and summarize the results of several studies Features of narrative reviews and systematic reviews QUESTION NARRATIVE SYSTEMATIC Broad Focused SOURCES/ SEARCH Usually unspecified Comprehensive; Possibly biased explicit SELECTION Unspecified; biased?Criterion-based; uniformly applied APPRAISAL Variable SYNTHESIS Usually qualitative Quantitative INFERENCE Sometimes evidence-based Rigourous Usually evidencebased Steps of a Cochrane Systematic Review • Clearly formulated question • Comprehensive data search • Unbiased selection and extraction process • Critical appraisal of data • Synthesis of data • Perform sensitivity and subgroup analyses if appropriate and possible • Prepare a structured report • What is the study objective to validate results in a large population to guide new studies Pose question in both biologic and health care terms specifying with operational definitions population intervention outcomes (both beneficial and harmful) Inclusion Criteria • Study design • Population • Interventions • Outcomes Steps of a Cochrane Systematic Review • Clearly formulated question • Comprehensive data search • Unbiased selection and extraction process • Critical appraisal of data • Synthesis of data • Perform sensitivity and subgroup analyses if appropriate and possible • Prepare a structured report • • • • Need a well formulated and co-ordinated effort Seek guidance from a librarian Specify language constraints Requirements for comprehensiveness of search depends on the field and question to be addressed • Possible sources include: computerized bibliographic database review articles abstracts conference proceedings dissertations books experts granting agencies trial registries industry journal handsearching • Procedure: usually begin with searches of biblographic reports (citation indexes, abstract databases) publications retrieved and references therein searched for more references Published Reports (publication bias ie. tendency to publish statistically significant results) as a step to elimination of publication bias need information from unpublished research databases of unpublished reports clinical research registries clinical trial registries unpublished theses conference indexes Correlation coefficient (cont’d) • meta-analysis is carried out on Z-transformed measures and final results are transformed back to the scale of correlation using r 2Z e 2Z e Numerical Example • Source: Fleiss J., Statistical Methods in Medical Research 1993; 2: 121 -145 • correlation coefficients reported by independent studies in education are included in the meta-analysis • Comparison: association between a characteristic of the teacher and the mean measure of his or her student’s achievement Example: Fleiss (1993) Study n r Z* W** WZ WZ2 ============================================================== 15 -0.073 -0.073 12 -0.876 0.064 16 0.308 0.318 13 4.134 1.315 15 0.481 0.524 12 6.288 3.295 16 0.428 0.457 13 5.941 2.715 15 0.180 0.182 12 2.184 0.397 17 0.290 0.299 14 4.186 1.252 15 0.400 0.424 _ 12 _5.088 2.157 Sum 88 26.945 11.195 =================================================== * Z = Fisher’s Z-transformation of r ** W = n-3 Q Wi ( Z i Z ) 2 Wi Z i ( Wi Z i ) / Wi 11.195 (26.945) /88 2.94 Q = 2.94 on df is not statistically significant Results and discussions • No evidence for heterogeneous association across studies • Fixed effect analysis may be undertaken • Questions: – Would a random effect analysis as shown earlier produce a different numerical value for the combined correlation coefficient? – How would the weights be modified to carry out a REM? Results and discussions (cont’d) • the weighted mean of Z is Z Wi Z i / Wi 26.945/ 88 0.306 • the approximate standard error of the combined mean is SE ( Z ) Wi 88 0.107 Results and discussions (cont’d) • Test of significance is carried out using z Z SE ( Z ) 0.306 0.107 2.86 – this value exceeds the critical value 1.96 (corresponding to 5% level of significance), so we conclude that average value of Z (hence the average correlation) is statistically significant Results and discussions (cont’d) • 95% confidence interval for Z 1.96 0.096 is SE ( Z ) 0.516 • Transforming back to the original scale, a 95% CI for the parameter of interest, , is 0.096 0.474 – again confirming a significant association Critical Appraisal of a Systematic Review (A) The Message • Does the review set out to answer a precise question about patient care? – Should be different from an uncritical encyclopedic presentation (B) The Validity • Have studies been sought thoroughly: Medline and other relevant bibliographic database Cochrane controlled clinical trials register Foreign language literature "Grey literature" (unpublished or un-indexed reports: theses, conference proceedings, internal reports, non-indexed journals, pharmaceutical industry files) Reference chaining from any articles found Personal approaches to experts in the field to find unpublished reports Hand searches of the relevant specialized journals Validity (cont’d) • Have inclusion and exclusion criteria for studies been stated explicitly, taking account of the patients in the studies, the interventions used, the outcomes recorded and the methodology? Validity (cont’d) • Have the authors considered the homogeneity of the studies: the idea that the studies are sufficiently similar in their design, interventions and subjects to merit combination – this is done either by eyeballing graphs like the forest plot or by applications of chi-square tests (Q test) (C) The Utility • The various studies may have used patients of different ages or social classes, but if the treatment effects are consistent across the studies, then generalisation to other groups or populations is more justified Utility (cont’d) • Be wary of sub-group analyses where the authors attempt to draw new conclusions by comparing the outcomes for patients in one study with the patients in another study – Be wary of "data-dredging" exercises, testing multiple hypotheses against the data, especially if the hypotheses were constructed after the study had begun data collection Utility (cont’d) • One may also want to ask: Were all clinically important outcomes considered? Are the benefits worth the harms and costs? ... Meta-Analysis: – a systematic review that employs statistical methods to combine and summarize the results of several studies Features of narrative reviews and systematic reviews QUESTION NARRATIVE SYSTEMATIC. .. randomization was appropriate ( e Random numbers were computer generated) g Steps of a Cochrane Systematic Review • Well formulated question • Comprehensive data search • Unbiased selection and. .. collection of studies • Meta-analysis methods focus on contrasting and comparing results from different studies in anticipation of identifying consistent patterns and sources of disagreements among