Critical Appraisal of the Literature Given the exponential growth in the number of scholarly journals and reports, it is important for clinicians to have sufficient skills in critically appraising the quality and relevance of studies being published These reports constitute the evidence that is incorporated into the practice of evidence-based clinical decision making For many reasons, as noted earlier, studies and their reports vary widely in their quality Levels of Evidence Often recommendations on a clinical practice as seen in a guideline are assigned a “level of evidence.” This is a guide for the practitioner on how much medical evidence supports the recommendation When producing a recommendation for clinical practice, whether it is for broader dissemination or for personal practice, the appraisal and incorporation of multiple reports have an important impact on the level of evidence that is assigned to that recommendation Several schemes have been devised with the purpose of grading the evidence (Table 24.3) A grade should be assigned to any recommendation and should influence the strength of that recommendation Recommendations supported by high or rigorous levels of evidence should carry the greatest weight in influencing clinical practice Table 24.3 System for Grading the Quality of Evidence and Strength of Recommendations Evidence Quality for Grades of Evidence Grade Evidence A Well-designed randomized controlled trials or diagnostic studies performed on a population similar to the guideline's target population B Randomized controlled trials or diagnostic studies with minor limitations; genetic natural history studies; overwhelmingly consistent evidence from observational studies C Observational studies (case-control and cohort design) D Expert opinion, case reports, or reasoning from first principles (bench research or animal studies) Definitions for Evidence-Based Statements Statement Type Definition Implication Strong The reporter(s) believe that the benefits of the recommended Clinicians should follow a recommendation approach clearly exceed the harms and that the quality of the supporting evidence is excellent (grade A or B) In some clearly defined circumstances, strong recommendations may be made on the basis of lesser evidence when high-quality evidence is impossible to obtain and the anticipated benefits clearly outweigh the harms Recommendation The reporter(s) feel that the benefits exceed the harms, but the quality of the evidence is not as strong (grade B or C) In some clearly defined circumstances, strong recommendations may be made on the basis of lesser evidence when high-quality evidence is impossible to obtain and the anticipated benefits clearly outweigh the harms Option Either the quality of the evidence that exists is suspect (grade D) or well-performed studies (grade A, B, or C) show little clear advantage to one approach vs another No There is both a lack of pertinent evidence (grade D) and an recommendation unclear balance between benefits and harms strong recommendation unless a clear and compelling rationale for an alternative approach is present Clinicians should generally follow a recommendation but remain alert to new information and sensitive to patient preferences Clinicians should be flexible in their decision making regarding appropriate practice, although they may set boundaries on alternatives; patient preference should have a substantial influencing role Clinicians should feel little constraint in their decision making and be alert to new published evidence that clarifies the balance of benefit vs harm; patient preference should have a substantial influencing role Modified from American Academy of Pediatrics, Steering Committee on Quality Improvement and Management Classifying recommendations for clinical practice guidelines Pediatrics 2004;114:874–877 Critical Appraisal Several guides are available for assisting with critical appraisal, with specifics depending on the perspective of the report being appraised The British Medical Journal published a very complete series of articles regarding specific methodologies to be used in critically assessing the quality of different types of publications.51–58 Additionally, the Journal of the American Medical Association has also published a series relevant to critical appraisal of the literature.59–65 Critical appraisal begins by identification of the underlying question being addressed and the appropriateness of the design and methods of the study Next, the quality of the design and its conduct are assessed to determine if the investigation has succeeded in achieving valid findings Then the findings are examined to determine if they are both clinically important and relevant as well as statistically significant or reliable Finally, the report is appraised as to its relevance to personal practice or a specific clinical situation relevant to the reader Critical appraisal can sometimes be challenging when the investigators do not provide sufficient information in their report regarding various aspects of their study The following section outlines a brief guide to reading different types of reports Therapy Often of greatest interest to clinicians are reports of results of specific therapies In reviewing a study of a therapy, the reader should consider the following in appraising the report Was the study conducted in manner that minimized bias? ■ Were the subjects randomly assigned to the study interventions? ■ Of those subjects randomized, what proportion in each group completed the study according to the protocol? ■ Was blinding as to study group feasible, attempted, and achieved for subjects, care providers, personnel, data analysts, and investigators? ■ Were assessments conducted, evaluated, and interpreted in a rigorous and standardized manner? Are the results reported appropriately? ■ Are both absolute and relative differences in outcomes reported? ■ Are confidence limits provided around differences? ■ Are benefits interpreted in light of associated risks and adverse outcomes? ... Modified from American Academy of Pediatrics, Steering Committee on Quality Improvement and Management Classifying recommendations for clinical practice guidelines Pediatrics 2004;114:874–877 Critical Appraisal