This document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of RAND intellectual property is provided for non-commercial use only. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. Permission is required from RAND to reproduce, or reuse in another form, any of our research documents for commercial use. For information on reprint and linking permissions, please see RAND Permissions. Limited Electronic Distribution Rights Visit RAND at www.rand.org Explore RAND Education View document details For More Information Purchase this document Browse Books & Publications Make a charitable contribution Support RAND This PDF document was made available from www.rand.org as a public service of the RAND Corporation. 6 Jump down to document The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world. THE ARTS CHILD POLICY CIVIL JUSTICE EDUCATION ENERGY AND ENVIRONMENT HEALTH AND HEALTH CARE INTERNATIONAL AFFAIRS NATIONAL SECURITY POPULATION AND AGING PUBLIC SAFETY SCIENCE AND TECHNOLOGY SUBSTANCE ABUSE TERRORISM AND HOMELAND SECURITY TRANSPORTATION AND INFRASTRUCTURE WORKFORCE AND WORKPLACE This product is part of the RAND Corporation monograph series. RAND monographs present major research findings that address the challenges facing the public and private sectors. All RAND monographs undergo rigorous peer review to ensure high standards for research quality and objectivity. Ending Social Promotion Without Leaving Children Behind The Case of New York City Jennifer Sloan McCombs, Sheila Nataraj Kirby, and Louis T. Mariano, Editors Prepared for the New York City Department of Education EDUCATION The RAND Corporation is a nonprofit research organization providing objective analysis and effective solutions that address the challenges facing the public and private sectors around the world. RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors. R ® is a registered trademark. © Copyright 2009 RAND Corporation Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND documents to a non-RAND Web site is prohibited. RAND documents are protected under copyright law. For information on reprint and linking permissions, please visit the RAND permissions page (http://www.rand.org/ publications/permissions.html). Published 2009 by the RAND Corporation 1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138 1200 South Hayes Street, Arlington, VA 22202-5050 4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665 RAND URL: http://www.rand.org To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) 451-7002; Fax: (310) 451-6915; Email: order@rand.org Cover design by Eileen Delson La Russo The research described in this report was prepared for the New York City Department of Education and conducted within RAND Education, a unit of the RAND Corporation. Library of Congress Cataloging-in-Publication Data McCombs, Jennifer Sloan, 1970- Ending social promotion without leaving children behind : the case of New York City / Jennifer Sloan McCombs, Sheila Nataraj Kirby, Louis T. Mariano. p. cm. Includes bibliographical references. ISBN 978-0-8330-4778-6 (pbk. : alk. paper) 1. Promotion (School)—New York (State)—New York—Case Studies. 2. Grade repetition—New York (State)—New York—Case Studies. I. Kirby, Sheila Nataraj, 1946– II. Mariano, Louis T. III. Title. LB3063.M285 2010 371.2'8097471—dc22 2009032280 iii Preface Many states and districts are moving toward test-based requirements for promotion at key transitional points in students’ schooling careers, thus ending the practice of “social promotion”—promoting students without regard for how much they have learned. e rationale for retention is that repetition of the grade will give students an additional year to master the academic content that they failed to master the previous year, and, thus, students should be less at risk for failure when they go on to the next grade. Opponents of grade retention argue that prior research has shown that grade retention disproportionately affects low-income and minority children and is associated with low self-esteem, problem behaviors, and an increased risk of dropping out of school. In 2003–2004, the New York City Department of Education (NYCDOE) implemented a new promotion and retention policy for 3rd-grade students in New York City (NYC) public schools. e policy was extended to grade 5 in 2004–2005, grade 7 in 2005–2006, and grade 8 in 2008–2009. NYCDOE asked the RAND Cor- poration to conduct an independent longitudinal evaluation of the 5th-grade social promotion policy and to examine the outcomes for two cohorts of 3rd-grade students. is study—conducted between March 2006 and August 2009—examined (1) policy implementation, factors affecting implementation, and implementation progress over time; (2) the impact of the policy on student academic and socioemotional outcomes; and (3) the links between implementation and desired outcomes. is monograph presents the results of the study. Two other reports (Marsh et al., 2009, and Xia and Kirby, 2009) document the results of two additional tasks that were part of the overall study. e first reviews lessons learned regarding the design and implementation of promotion policies in a selection of states and districts with promo- tion policies similar to that of NYC, and the second presents a detailed and compre- hensive review of the literature on grade retention. All three publications should inter- est policymakers, practitioners, and researchers involved in designing, implementing, or studying interventions to improve outcomes for low-performing students. is research was conducted by RAND Education, a unit of the RAND Corporation. v Contents Preface iii Figures xiii Tables xvii Summary xxi Acknowledgments xxxi Abbreviations xxxiii CHAPTER ONE Introduction 1 Sheila Nataraj Kirby, Jennifer Sloan McCombs, and Louis T. Mariano e Current Study 2 Contribution of the Study 4 Terminology 5 Organization of is Monograph 5 CHAPTER TWO What We Know About the Effects of Grade Retention and Implementation of Promotion Policies 7 Nailing Xia and Sheila Nataraj Kirby Methods 7 Characteristics of Retained Students 10 Effects of Grade Retention on Students’ Academic and Nonacademic Outcomes 11 Effect on Academic Outcomes 11 Effect on Socioemotional Outcomes 12 Effect on the Propensity to Drop Out of School 13 Effect of Supportive Components of Promotion Policies on Student Achievement 14 Summer School Programs and Student Achievement 14 Findings on Other Supportive Components of Retention Policies 15 Summary 16 vi Ending Social Promotion Without Leaving Children Behind: The Case of New York City CHAPTER THREE Context and Conceptual Framework for Understanding New York City’s Promotion Policy 17 Jennifer Sloan McCombs, Sheila Nataraj Kirby, Julie A. Marsh, and Catherine DiMartino New York City’s Reform Initiative 17 e Children First Initiative 17 e Promotion Policy 19 Criteria for Promotion 19 Key Components 21 Support Structures for the Promotion Policy 22 Conceptual Framework 24 Major Research Questions 30 Summary 30 CHAPTER FOUR Data and Methods 33 Sheila Nataraj Kirby, Louis T. Mariano, and Jennifer Sloan McCombs Data 33 Interviews with Regional Directors of Instructional Services and School Support Organization Leaders 33 Case-Study Site Visits 34 Selection of Case-Study Sites 34 Administrator Surveys 35 Student Surveys 38 Data on School Characteristics 41 Longitudinally Linked Student Data 41 Methods 42 Analysis of Implementation 42 Analysis of the Effects of Supportive Interventions and Retention on Academic Outcomes 45 Analyzing Differences in Students’ Socioemotional Attitudes and eir Association with Future Achievement 50 Study Limitations 51 Summary 53 Contents vii CHAPTER FIVE School-Provided Support for Students: Academic Intervention Services 55 Jennifer Sloan McCombs, Scott Naftel, Gina Schuyler Ikemoto, Catherine DiMartino, and Daniel Gershwin Profile of NYC Schools with a 5th Grade 56 Students Needing Services Across Schools 57 Selection and Monitoring of Students 60 Process for Selection and Monitoring of Students for AIS 60 Students Targeted for Services 61 Academic Intervention Service Provision 63 Providers 63 Types of Academic Intervention Service Programs 64 Focus of Intervention Services 66 Perceived Effectiveness 67 Additional School Strategies to Improve Student Performance 68 Support for Promotion Policy Implementation 70 Hindrances to Improving the Performance of In-Need 5th-Grade Students 72 Perceptions Regarding Student Retention and the Promotion Policy 74 Exploring the Relationships Among School Context, Implementation Factors, and Students’ Outcomes on the Spring Assessments 77 Summary 78 CHAPTER SIX Implementation of the Policy: Saturday and Summer Schools 81 Gina Schuyler Ikemoto, Jennifer Sloan McCombs, Catherine DiMartino, and Scott Naftel Saturday Preparatory Academies 82 Profile of Saturday Preparatory Academies and Students 82 Student Participation 83 Schedule 84 Staffing 86 Curriculum 87 Professional Development 88 Communication Between SPAs and Schools 88 Perceived Effectiveness of SPAs 89 Summer Success Academies 90 Profile of Summer Success Academies 91 Student Participation 91 Schedule 93 Staffing 93 Curriculum 94 viii Ending Social Promotion Without Leaving Children Behind: The Case of New York City Intervention 94 Time for Professional Development 94 Portfolios 95 Factors Perceived to Affect Implementation of SPAs and SSAs 95 Factors Hindering Implementation 95 Factors Enabling Implementation 97 Summary 98 CHAPTER SEVEN Performance of 5th Graders in New York City and Overall Performance Trends in New York State 101 Sheila Nataraj Kirby, Scott Naftel, Jennifer Sloan McCombs, Daniel Gershwin, and Al Crego Performance of the NYC 5th-Grade Cohorts 101 Students Who Were Subject to the Promotion Policy 101 Overall Performance on the 5th-Grade Spring Assessments 102 Snapshots of the 5th-Grade Cohorts 104 Profile of Retained Students 108 Performance of At-Risk Students in Later Years 111 Performance of Retained Students in Later Years 112 Performance of At-Risk Promoted Students in Later Years 115 Achievement Trends in Grades 3–8: New York City and the Rest of New York State 119 Performance on the Spring ELA Assessments, Grades 3–8 119 Performance on the Spring Mathematics Assessments, Grades 3–8 121 Discussion 122 Summary 125 Students Held to the Promotion Policy 125 Students Who Needed Services at the Beginning of the School Year 125 Student Performance on the Proximal-Year Spring Assessments 125 Fifth-Grade Promotion/Retention Outcomes for Students at Risk of Retention 125 Retained Students 126 Performance of At-Risk Students on Higher-Grade Spring Assessments 126 Comparing Performance Trends in NYC and the Rest of the State 126 CHAPTER EIGHT Measuring the Effect of Supportive Interventions on Proximal-Year Student Achievement 127 Louis T. Mariano, Sheila Nataraj Kirby, Al Crego, and Claude Messan Setodji Analytic Methods 128 Propensity Score Weighting and Doubly Robust Regression 128 Modeling the Relationship Between Intensity of Participation and Outcomes 131 [...]... the promotion benchmark) Performance at or above Level 3 is considered “proficient” under the No Child Left Behind Act (NCLB), a higher standard than the promotion benchmark The policy places considerable emphasis on identifying struggling students early, providing them with additional instructional time, and continuously monitoring their xxi xxii Ending Social Promotion Without Leaving Children Behind: ... Sixth- and 7th-Grade Outcomes for Students Attending SSA and Retained, P1 and P2 Cohorts 144 144 145 148 151 151 155 156 160 162 162 163 163 x Ending Social Promotion Without Leaving Children Behind: The Case of New York City ChAPTer Ten The Impact of new York City’s Promotion Policy on Students’ Socioemotional Status ... 5th-Grade Cohort One to Three Years After the Retention Decision 232 xx Ending Social Promotion Without Leaving Children Behind: The Case of New York City D.3 D.4 D.5 E.1 E.2 E.3 E.4 E.5 E.6 E.7 E.8 E.9 E.10 E.11 E.12 E.13 Standardized Regression Coefficients by Promotion Type, One to Four Years After the Retention Decision ... Ending Social Promotion Without Leaving Children Behind: The Case of New York City estimate the effects of supportive interventions and of being retained in grade, we used two quasi-experimental methods to define plausible comparison groups that could be used as controls in our models These models were coupled with a “difference-indifference” approach that allowed us to estimate the effect of the promotion. .. NYC’s promotion policy, and approximately three-quarters of AIS leaders and principals agreed that the policy focused their schools’ instruction efforts in a positive way and that it made parents and guardians more concerned about their child’s progress However, the majority of respondents thought that the promotion policy relied too heavily on state xxviii Ending Social Promotion Without Leaving Children. .. test-based promotion policies Continue early identification of students and provision of academic intervention services Our findings suggest that the process of early identification and support helped students meet promotion standards and had positive effects on student achievement in 3 See Marsh et al (2009) for a full description of these and related findings xxx Ending Social Promotion Without Leaving Children. .. Organization MSE mean square error NAEP National Assessment of Educational Progress NCLB No Child Left Behind Act NYC New York City NYCDOE New York City Department of Education NYSED New York State Education Department OLS ordinary least squares xxxiii xxxiv Ending Social Promotion Without Leaving Children Behind: The Case of New York City PIP personal intervention plan PK prekindergarten PSO Partner Support... exception (7th-grade ELA in 2008, which was 5 percent) In 2009, the probability of scoring Level 2 by “guessing” increased, but this occurred after the period of our study xxvi Ending Social Promotion Without Leaving Children Behind: The Case of New York City have additional resources for students, such as mathematics specialists, reading coaches, and mathematics coaches Schools provided additional... 111 Percentage of At-Risk Promoted and Retained Students Scoring Level 1 on the 6th-Grade ELA Assessment 116 xiii xiv Ending Social Promotion Without Leaving Children Behind: The Case of New York City 7.8 7.9 7.10 7.11 7.12 7.13 7.14 7.15 7.16 8.1 8.2 8.3 8.4 8.5 9.1 9.2 10.1 10.2 10.3 B.1 Percentage of At-Risk Promoted and Retained Students... Attended and 3rd-Grade Summer Mathematics Assessment Scale Score, 2003–2004 3rd-Grade Cohort 251 xvi Ending Social Promotion Without Leaving Children Behind: The Case of New York City E.7 Third-Grade Students’ Socioemotional Responses One, Two, and Three Years After the Retention Decision . Outcomes for Students Attending SSA and Retained, P1 and P2 Cohorts 163 x Ending Social Promotion Without Leaving Children Behind: The Case of New York. Cataloging-in-Publication Data McCombs, Jennifer Sloan, 1970- Ending social promotion without leaving children behind : the case of New York City / Jennifer Sloan