EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1.0 EPI Case Study 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening Time to Complete Exercise: 30 minutes LEARNING OBJECTIVES At the completion of this Case Study, participants should be able to: Know how to construct 2-by-2 tables Compare screening techniques using the Kappa Statistic Distinguish between reliability and validity Identify groups at high risk for TB infection Identify methods used to screen for TB infection ASPH EPIDEMIOLOGY COMPETENCIES ADDRESSED C.2 Identify the principles and limitations of public health screening programs C Describe a public health problem in terms of magnitude, person, place, and time C Apply the basic terminology and definitions of epidemiology C Calculate basic epidemiologic measures C Draw appropriate inference from epidemiologic data C 10 Evaluate the strengths and limitations of epidemiologic reports This material was developed by the staff at the Global Tuberculosis Institute (GTBI), one of four Regional Training and Medical Consultation Centers funded by the Centers for Disease Control and Prevention It is published for learning purposes only Permission to reprint excerpts from other sources was granted Case study author(s) name and position: George Khalil, MPH (work done as MPH candidate) Marian R Passannante, PhD Associate Professor, University of Medicine & Dentistry of New Jersey, New Jersey Medical School and School of Public Health Epidemiologist, NJMS, GTBI For further information please contact: New Jersey Medical School Global Tuberculosis Institute (GTBI) 225 Warren Street P.O Box 1709 Newark, NJ 07101-1709 or by phone at 973-972-0979 Suggested citation: New Jersey Medical School Global Tuberculosis Institute /Incorporating Tuberculosis into Public Health Core Curriculum./ 2009: Epidemiology Case Study 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT Version 1.0 EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1.0 Introduction In the United States, no vaccines are given to prevent the transmission of tuberculosis (TB) due to their current lack of efficacy.1 Because scientists are still working to create a more efficient vaccine, the generally accepted approach to TB control relies on screening, surveillance, and contact investigations.2 Identifying and treating persons with latent TB infection (LTBI) at high risk for developing TB is part of the current TB elimination strategy in the United States Screening is essential in the identification phase of this strategy (2) The most common method of screening is with purified protein derivative (PPD), a type of tuberculin skin test (TST) Infection with Mycobacterium tuberculosis (M tuberculosis) produces a delayed-type hypersensitivity reaction to certain antigenic components of the organism that are contained in extracts of culture filtrates called tuberculins The skin test is injected in the forearm and read by a trained clinician after 48 to 72 hours The size of the reaction is measured in millimeters and interpreted according to its size, using cutoff points corresponding to the degree of induration.3 Another type of screening test is called interferon gamma assay (IGA), which measures the production of the cellular interferon gamma by T-cells after sensitization with M tuberculosis antigens Although researchers believe that interferon tests are preferable to the TSTs, they are much more expensive This exercise will be based on the following study Gerald H Mazurek; Philip A LoBue; Charles L Daley; John Bernardo; Alfred A Lardizabal; William R Bishai; Michael F Iademarco; James S Rothel, Comparison of a Whole-Blood Interferon Assay With Tuberculin Skin Testing for Detecting Latent Mycobacterium tuberculosis Infection JAMA 2001;286:1740-1747.5 Sections of this document have been reprinted with permission of the journal (permission pending) Context Identifying persons with latent tuberculosis infection (LTBI) is crucial to the goal of TB elimination A whole-blood interferon (IFN- ) assay, the Quanti-FERONTB test, is a promising in vitro diagnostic test for LTBI that has potential advantages over the tuberculin skin test (TST) Objectives To compare the IFN- assay with the TST and to identify factors associated with discordance between the tests Design and Setting Prospective comparison study conducted at university affiliated sites in the United States between March 1, 1998 and June 30, 1999 Participants A total of 1226 adults (mean age, 39 years) with varying risks of Mycobacterium tuberculosis infection or documented or suspected active TB, all of whom underwent both the IFN- assay and the TST Main Outcome Measure Level of agreement between the IFN- assay and the TST EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1.0 METHODS The study was conducted at sites: Boston University School of Medicine, Mass; Johns Hopkins School of Hygiene and Public Health, Baltimore, Md; University of California at San Francisco; New Jersey Medical School, Newark; and University of California at San Diego, using a common protocol These sites were randomly coded as A-E in the analysis Ethical approval for the study was obtained from the institutional review boards at the Centers for Disease Control and Prevention (CDC), which supported the study, and the study sites prior to enrolling any subjects All participants provided written informed consent Persons recruited for the study were 18 years or older and included persons requesting a preemployment or preschool enrollment TST; persons being screened with a TST because they were considered to be at high risk for LTBI; persons in whom TB was clinically suspected and who had received fewer than weeks of anti-TB therapy; and persons who previously had active TB, confirmed by a positive culture, and who had completed a course of multidrug anti-TB therapy within the prior years Subjects were excluded from the study if they self-reported as pregnant or HIVpositive; had a history of severe reaction to tuberculin; were immunocompromised due to leukemia, lymphoma, or Hodgkin disease; or had taken immunosuppressive drugs (eg,orticosteroids,methotrexate, azathioprine) during the preceding months After providing written informed consent, enrolled persons completed a detailed questionnaire about possible risk factors for exposure to M tuberculosis Subjects were also asked to indicate results of any prior TST, whether they had received BCG vaccination, details of any contact with a person having TB, any risk factors associated with HIV infection, and whether they had any other medical conditions When applicable, data were also collected from medical records about findings on chest radiography, results and dates of cultures for mycobacteria, and details of treatment for TB Data were collected on subjects’ age, race, place of birth, residence outside of the United States, and residence or work (paid or unpaid) in a health care setting, prison, homeless shelter, drug rehabilitation unit, or other group housing Based on responses to the questionnaire and a review of available medical records, persons were categorized into study groups: (1) low-risk for LTBI, subjects receiving preemployment or preschool enrollment TST with no identified risks for LTBI; (2) highrisk for LTBI, asymptomatic subjects with risk of LTBI including contacts of patients with TB; persons from countries where tuberculosis is prevalent (>10 cases per 100000 population)26; intravenous drug users; persons who lived, worked, or volunteered on a regular basis in a homeless shelter, prison, drug rehabilitation unit, hospital, or nursing home; and persons determined to be at increased risk by prior local investigations;(3) TB suspects, subjects being evaluated for active TB who had received fewer than weeks of anti-TB therapy; and (4) culture-confirmed TB, subjects who completed treatment for culture-confirmed TB within the prior years Persons enrolled during preemployment or preschool enrollment examinations were assigned to group if risk factors for LTBI were identified during questioning However, EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1.0 to maintain the integrity of group as truly low risk for LTBI, persons considered to be at high-risk for LTBI at enrollment were assigned to group even when risk factors were denied Reference 26 in Mazurek et al: World Health Organization Global Tuberculosis Control Geneva, Switzerland: WHO; 1999.WHO/ CDS/CPC/TB/99.259 For the purposes of this exercise, we will examine the data from Group (low-risk group for TB infection) and Group (high-risk group) Table compares the responses to both TST and IFN tests for Groups (low risk) and (high risk) Table Response to TST and IFN- tests in high- and low-risk Groups5 Question A For Group 1, create a 2X2 table to test the association between TST and IFNreadings Show percent for marginal totals B For Group 2, create a 2X2 table to test the association between TST and IFNreadings Show percent for marginal totals Use the following 2x2 tables: A TST IFNPositive Positive Negative Total B TST IFNPositive Positive Negative Total assay Negative Total assay Negative Total EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1.0 Question A For Group 1, calculate an overall percent agreement by TST and IFNinterpret B Do the same for Group assay and Note: Percent agreement can be calculated as (a+d)/(a+b+c+d) x 100 and is called p o (or proportion of agreement observed) where a,b,c,d are shown in this table: Outcome Independent variable Yes No Total Yes No Total a c a+c b d b+d a+b c+d a+b+c+d=n A po or % agreement for Group = B po or % agreement for Group = Question For Groups and 2, create2X2 tables of percent agreement by screening test expected by chance alone and calculate its percent agreement To this you can calculate the expected values for cells a and d and then calculate the percent agreement expected by chance using the formula (a expected value + d expected value)/(a+b+c+d) x 100 This is called pe (or proportion of agreement expected) Remember: Expected values= (row total x column total)/grand total Use the following 2x2 tables: A Percent agreement expected by chance alone for Group 1= TST IFN- assay Total Positive Negative Positive Negative Total B Percent agreement expected by chance alone for Group 2= EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1.0 TST IFNPositive assay Negative Total Positive Negative Total Question A What is the kappa statistic of the tests for Group (low risk for TB)? B And for Group (high risk for TB)? κ = (percent observed agreement) – (percent agreement by chance alone) 100%- (percent agreement expected by chance alone) Table 26 Source: Understanding interobserver agreement: the kappa statistic Viera AJ, Garrett JM Fam Med 2005;37:362 Permission granted to reprint table Question EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1.0 Rate the overall reliability of the screening tests Does prevalence seem to have an impact on the kappa value? Question Is there a limitation in comparing the IFN- assay with the TST? If so, what is it? Validity and Reliability A useful screening test is both reliable (reproducible or accurate) and valid (precise) In Figure 16, assuming the goal is to hit the middle of the target, B and D are reliable (also called reproducible) but only D is both reliable and accurate Unfortunately, to test for validity the truth has to be known or there has to be a “gold standard” and in TB screening there is no gold standard So, in this exercise we are only looking at reliability of two screening tests Source: Understanding interobserver agreement: the kappa statistic Viera AJ, Garrett JM Fam Med 2005;37:361 Permission granted to reprint Figure EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1.0 Works Cited Development of new vaccines for tuberculosis MMWR Recomm Rep 1998;47 (RR-13):16 The role of BCG vaccine in the prevention and control of tuberculosis in the United States: A joint statement by the Advisory Council for the Elimination of TB and the Advisory Committee on Immunization Practices MMWR Recomm Rep 1996;45 (RR-4):1-18 Ayub A, Yale SH, Reed KD, Nasser RM, Gilbert SR Outpatient practice management tips Testing for Latent Tuberculosis Accessed February 8, 2008 http:www.clinmedres.org/cgi/content/full/2/3/191 Madariaga MG, Jalali Z, Swindells S Clinical utility of interferon gamma assay in the diagnosis of tuberculosis J Am Board Fam Med 2007;20:540-547 Mazurek GH, LoBue PA, Daley CL, et al Comparison of a whole-blood interferon gamma assay with tuberculin skin testing for detecting latent mycobacterium tuberculosis infection JAMA 2001;286:1740-1747 Viera J, Garrett JM Understanding interobserver agreement: the kappa statistic Fam Med 2005;37:360-363 ... assay and the TST Main Outcome Measure Level of agreement between the IFN- assay and the TST EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT. .. during questioning However, EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1. 0 to maintain the integrity of group as truly low... expected by chance alone for Group 2= EPIDEMIOLOGY CASE STUDY 2: Reliability, Validity, and Tests of Agreement in M Tuberculosis Screening STUDENT VERSION 1. 0 TST IFNPositive assay Negative Total