This document and trademark(s) contained herein are protected by law as indicated in a notice appearing later in this work. This electronic representation of RAND intellectual property is provided for non-commercial use only. Unauthorized posting of RAND PDFs to a non-RAND Web site is prohibited. RAND PDFs are protected under copyright law. Permission is required from RAND to reproduce, or reuse in another form, any of our research documents for commercial use. For information on reprint and linking permissions, please see RAND Permissions. Limited Electronic Distribution Rights This PDF document was made available from www.rand.org as a public service of the RAND Corporation. 6 Jump down to document THE ARTS CHILD POLICY CIVIL JUSTICE EDUCATION ENERGY AND ENVIRONMENT HEALTH AND HEALTH CARE INTERNATIONAL AFFAIRS NATIONAL SECURITY POPULATION AND AGING PUBLIC SAFETY SCIENCE AND TECHNOLOGY SUBSTANCE ABUSE TERRORISM AND HOMELAND SECURITY TRANSPORTATION AND INFRASTRUCTURE WORKFORCE AND WORKPLACE The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. Visit RAND at www.rand.org Explore RAND Education View document details For More Information Browse Books & Publications Make a charitable contribution Support RAND This product is part of the RAND Corporation technical report series. Reports may include research findings on a specific topic that is limited in scope; present discus- sions of the methodology employed in research; provide literature reviews, survey instruments, modeling exercises, guidelines for practitioners and research profes- sionals, and supporting documentation; or deliver preliminary findings. All RAND reports undergo rigorous peer review to ensure that they meet high standards for re- search quality and objectivity. EDUCATION Incorporating Student Performance Measures into Teacher Evaluation Systems Jennifer L. Steele, Laura S. Hamilton, Brian M. Stecher Sponsored by the Center for American Progress This work was sponsored by the Center for American Progress with support from the Bill and Melinda Gates Foundation. The research was conducted in RAND Education, a unit of the RAND Corporation. The RAND Corporation is a nonprofit institution that helps improve policy and decisionmaking through research and analysis. RAND’s publications do not necessarily reflect the opinions of its research clients and sponsors. R ® is a registered trademark. © Copyright 2010 RAND Corporation Permission is given to duplicate this document for personal use only, as long as it is unaltered and complete. Copies may not be duplicated for commercial purposes. Unauthorized posting of RAND documents to a non-RAND website is prohibited. RAND documents are protected under copyright law. For information on reprint and linking permissions, please visit the RAND permissions page (http://www.rand.org/publications/ permissions.html). Published 2010 by the RAND Corporation 1776 Main Street, P.O. Box 2138, Santa Monica, CA 90407-2138 1200 South Hayes Street, Arlington, VA 22202-5050 4570 Fifth Avenue, Suite 600, Pittsburgh, PA 15213-2665 RAND URL: http://www.rand.org To order RAND documents or to obtain additional information, contact Distribution Services: Telephone: (310) 451-7002; Fax: (310) 451-6915; Email: order@rand.org Library of Congress Control Number: 2011927262 ISBN: 978-0-8330-5250-6 iii Preface Research tells us that teachers vary enormously in their ability to improve students’ perfor- mance on standardized tests but that many existing teacher evaluation and reward systems do not capture that variation. Armed with this knowledge and with improved access to longitu- dinal data systems linking teachers to students, reform-minded policymakers are increasingly attempting to base a portion of teachers’ evaluations or pay on student achievement gains. However, systems that incorporate student achievement gains into teacher evaluations face at least two important challenges: generating valid estimates of teachers’ contributions to student learning and including teachers who do not teach subjects or grades that are tested annually. is report summarizes how three districts and two states have already begun or are planning to address these challenges. In particular, the report focuses on what is and is not known about the quality of various student performance measures school systems are using and on how the systems are supplementing these measures with other teacher performance indicators. is report should be of interest to educational policymakers and practitioners at the fed- eral, state, and local levels and to families and communities interested in policy strategies for evaluating and improving teacher eectiveness. e research was carried out by RAND Education, a unit of the RAND Corporation, on behalf of the Center for American Progress, with support from the Bill and Melinda Gates Foundation. v Contents Preface iii Tables vii Summary ix Acknowledgments xiii Abbreviations xv CHAPTER ONE Introduction 1 e Problem: Teachers’ Evaluations Do Not Typically Reect eir Eectiveness in Improving Student Performance 1 A Growing Movement to Use Student Learning to Evaluate Teachers 2 Purpose, Organization, and Scope of is Report 3 CHAPTER TWO Using Multiple Measures to Assess Teachers’ Eectiveness 5 Technical Considerations in Selecting Quality Measures of Student Performance 6 Reliability Considerations 6 Validity Considerations 7 Vertical Scaling 8 Measuring Student Performance in Grades and Subjects at Are Not Assessed Annually 8 Assigning Teachers Responsibility for Students’ Performance 10 CHAPTER THREE How Are New Teacher Evaluation Systems Incorporating Multiple Measures? 11 Denver ProComp 12 Hillsborough County’s Empowering Eective Teachers Initiative 13 e Tennessee Teacher Evaluation System 15 Washington, D.C., IMPACT 16 e Delaware Performance Appraisal System II 17 CHAPTER FOUR How Are the New Teacher Evaluation Systems Addressing Key Measurement Quality Challenges? 21 Reliability Considerations 21 Promoting Reliability of Value-Added Estimates 23 Validity Considerations 23 vi Incorporating Student Performance Measures into Teacher Evaluation Systems Vertical Scaling 23 Measuring Growth in Nontested Subjects 23 Assigning Responsibility for Student Performance 24 CHAPTER FIVE Policy Recommendations and Conclusion 27 References 29 About the Authors 35 vii Tables 3.1. Key Components of Denver ProComp 12 3.2. Key Components of Hillsborough County’s Empowering Eective Teachers Initiative 14 3.3. Key Components of the Tennessee Teacher Evaluation System 15 3.4. Key Components of the D.C. IMPACT Program 16 3.5. Key Components of Delaware’s Performance Appraisal System II 18 4.1. Test Information, Including Range of Internal Consistency Reliability Statistics for the Principal Standardized Test in Each System, Reported Across All Tested Grades, by Subject 22 [...]... incorporated value-added measures of student learning (National Center on Performance Incentives, 2008) 4 Incorporating Student Performance Measures into Teacher Evaluation Systems policymakers about factors to consider when incorporating student achievement measures into teacher evaluation systems This report focuses primarily on the use of student performance measures to evaluate teachers’ effectiveness... teachers are held accountable for the performance gains of their own students and other teachers are held accountable for the performance of a broader set of students than those they currently teach 10 Incorporating Student Performance Measures into Teacher Evaluation Systems Assigning Teachers Responsibility for Students’ Performance As educational systems begin attaching high stakes to student performance. .. supplemental ways of holding teachers accountable for these students, such as requiring them to set individual performance goals on teacher- chosen measures for those students CHAPTER THREE How Are New Teacher Evaluation Systems Incorporating Multiple Measures? We selected five systems to examine how new teacher evaluation systems are incorporating measures of student performance beyond the tested... Considerations in Selecting Quality Measures of Student Performance As states and districts seek multiple measures of student performance to incorporate into their evaluation systems, they must find student performance measures that can support inferences about teacher effectiveness in a variety of grades and content areas When using student achievement measures to evaluate teachers’ performance, the technical... generating valid estimates of teachers’ contributions to student learning and including teachers who do not teach subjects or grades that are tested annually This report considers these chal- ix x Incorporating Student Performance Measures into Teacher Evaluation Systems lenges in terms of the kinds of student performance measures that educational systems might use to measure teachers’ effectiveness in... when incorporating student achievement measures into teacher evaluation systems, are as follows: • Create comprehensive evaluation systems that incorporate multiple measures of teacher effectiveness • Attend not only to the technical properties of student assessments but also to how the assessments are being used in high-stakes contexts • Promote consistency in the student performance measures that teachers... decided 16 Incorporating Student Performance Measures into Teacher Evaluation Systems teacher s evaluation will be based on principal observations similar to those already conducted as part of the teacher s evaluation process; these may be supplemented with personal conferences and evaluation surveys from supervisors, peers, and students (Zelinski, 2010) If another recommendation by the Teacher Evaluation. .. schoolwide student performance, and individual value added 2 However, some programs described here are based on the older value-added or pay-for -performance systems on which the new evaluation systems are built How Are New Teacher Evaluation Systems Incorporating Multiple Measures? 13 ponents based on (1) teacher- selected assessments, (2) schoolwide student performance, and (3) individual teacher value... in teachers’ performance evaluations; under MAP, student performance is weighted at 60 percent The new Empowering Effective Teachers program, in contrast, will base 60 percent of each teacher s evaluation rating on classroom observations Half of this classroom observation rating will be based on the principal’s observations and the other 14 Incorporating Student Performance Measures into Teacher Evaluation. .. students’ prior learning One study in Florida reported that fewer than 31 percent of teachers in the state teach these tested subjects and grades (Prince et al., 2009) Thus, a critical policy question is how to develop evaluation systems that incorporate measures of student learning for the other teachers in the system as well 5 6 Incorporating Student Performance Measures into Teacher Evaluation Systems . these chal- x Incorporating Student Performance Measures into Teacher Evaluation Systems lenges in terms of the kinds of student performance measures that. websites of systems incorporating some type of student performance measures into their teacher evaluations according to media reports, prior studies, and teacher- quality