Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized Public Disclosure Authorized 52099 Handbook on Impact Evaluation Handbook on Impact Evaluation Quantitative Methods and Practices Shahidur R Khandker Gayatri B Koolwal Hussain A Samad © 2010 The International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved 13 12 11 10 This volume is a product of the staff of the International Bank for Reconstruction and Development / The World Bank The findings, interpretations, and conclusions expressed in this volume not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent The World Bank does not guarantee the accuracy of the data included in this work The boundaries, colors, denominations, and other information shown on any map in this work not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries Rights and Permissions The material in this publication is copyrighted Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law The International Bank for Reconstruction and Development / The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750-4470; Internet: www.copyright.com All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org ISBN: 978-0-8213-8028-4 eISBN: 978-0-8213-8029-1 DOI: 10.1596/978-0-8213-8028-4 Library of Congress Cataloging-in-Publication Data Khandker, Shahidur R Handbook on impact evaluation : quantitative methods and practices / Shahidur R Khandker, Gayatri B Koolwal, Hussain A Samad p cm Includes bibliographical references and index ISBN 978-0-8213-8028-4 — ISBN 978-0-8213-8029-1 (electronic) Economic development projects—Evaluation Economic assistance—Evaluation I Koolwal, Gayatri B II Samad, Hussain A., 1963- III Title HD75.9.K52 2009 338.90072—dc22 2009020886 Cover design by Patricia Hord.Graphik Design Contents Foreword xiii Preface xv About the Authors xvii Abbreviations xix Part Methods and Practices 1 Introduction References .6 Basic Issues of Evaluation Summary Learning Objectives .7 Introduction: Monitoring versus Evaluation .8 Monitoring .8 Setting Up Indicators within an M&E Framework Operational Evaluation .16 Quantitative versus Qualitative Impact Assessments .18 Quantitative Impact Assessment: Ex Post versus Ex Ante Impact Evaluations 20 The Problem of the Counterfactual 22 Basic Theory of Impact Evaluation: The Problem of Selection Bias .25 Different Evaluation Approaches to Ex Post Impact Evaluation .27 Overview: Designing and Implementing Impact Evaluations 28 Questions .29 References .30 Randomization .33 Summary 33 Learning Objectives .33 v Contents Setting the Counterfactual 34 Statistical Design of Randomization 34 Calculating Treatment Effects .35 Randomization in Evaluation Design: Different Methods of Randomization 38 Concerns with Randomization 38 Randomized Impact Evaluation in Practice .39 Difficulties with Randomization .47 Questions .49 Notes .51 References .51 Propensity Score Matching 53 Summary 53 Learning Objectives .53 PSM and Its Practical Uses 54 What Does PSM Do? 54 PSM Method in Theory .55 Application of the PSM Method 58 Critiquing the PSM Method .63 PSM and Regression-Based Methods 64 Questions .66 Notes .67 References .68 Double Difference .71 Summary 71 Learning Objectives .71 Addressing Selection Bias from a Different Perspective: Using Differences as Counterfactual 71 DD Method: Theory and Application 72 Advantages and Disadvantages of Using DD .76 Alternative DD Models 78 Questions .82 Notes .84 References .84 vi Contents Instrumental Variable Estimation 87 Summary 87 Learning Objectives .87 Introduction 87 Two-Stage Least Squares Approach to IVs 89 Concerns with IVs .91 Sources of IVs 95 Questions .99 Notes 100 References 100 Regression Discontinuity and Pipeline Methods 103 Summary 103 Learning Objectives 103 Introduction .104 Regression Discontinuity in Theory 104 Advantages and Disadvantages of the RD Approach .108 Pipeline Comparisons .110 Questions 111 References 112 Measuring Distributional Program Effects 115 Summary 115 Learning Objectives 115 The Need to Examine Distributional Impacts of Programs 115 Examining Heterogeneous Program Impacts: Linear Regression Framework 116 Quantile Regression Approaches 118 Discussion: Data Collection Issues 124 Notes 125 References 125 Using Economic Models to Evaluate Policies 127 Summary 127 Learning Objectives 127 Introduction .127 vii Contents Structural versus Reduced-Form Approaches 128 Modeling the Effects of Policies 130 Assessing the Effects of Policies in a Macroeconomic Framework 131 Modeling Household Behavior in the Case of a Single Treatment: Case Studies on School Subsidy Programs .133 Conclusions 135 Note 136 References 137 10 Conclusions 139 Part Stata Exercises 143 11 Introduction to Stata 145 Data Sets Used for Stata Exercises .145 Beginning Exercise: Introduction to Stata 146 Working with Data Files: Looking at the Content 151 Changing Data Sets 158 Combining Data Sets 162 Working with log and Files 164 12 Randomized Impact Evaluation 171 Impacts of Program Placement in Villages .171 Impacts of Program Participation 173 Capturing Both Program Placement and Participation 175 Impacts of Program Participation in Program Villages 176 Measuring Spillover Effects of Microcredit Program Placement 177 Further Exercises 178 Notes 179 13 Propensity Score Matching Technique 181 Propensity Score Equation: Satisfying the Balancing Property .181 Average Treatment Effect Using Nearest-Neighbor Matching 185 Average Treatment Effect Using Stratification Matching 186 Average Treatment Effect Using Radius Matching 186 Average Treatment Effect Using Kernel Matching 187 Checking Robustness of Average Treatment Effect 187 Further Exercises 188 Reference 188 viii Contents 14 Double-Difference Method .189 Simplest Implementation: Simple Comparison Using “ttest” 189 Regression Implementation 190 Checking Robustness of DD with Fixed-Effects Regression 192 Applying the DD Method in Cross-Sectional Data 193 Taking into Account Initial Conditions 196 The DD Method Combined with Propensity Score Matching 198 Notes 201 Reference 201 15 Instrumental Variable Method 203 IV Implementation Using the “ivreg” Command 203 Testing for Endogeneity: OLS versus IV 205 IV Method for Binary Treatment: “treatreg” Command 206 IV with Fixed Effects: Cross-Sectional Estimates .207 IV with Fixed Effects: Panel Estimates 208 Note 209 16 Regression Discontinuity Design 211 Impact Estimation Using RD 211 Implementation of Sharp Discontinuity 212 Implementation of Fuzzy Discontinuity 214 Exercise .216 Answers to Chapter Questions 217 Appendix: Programs and Files for Chapter 12–16 Exercises 219 Index 231 Boxes 2.1 Case Study: PROGRESA (Oportunidades) in Mexico 10 2.2 Case Study: Assessing the Social Impact of Rural Energy Services in Nepal .13 2.3 Case Study: The Indonesian Kecamatan Development Project 15 2.4 Case Study: Monitoring the Nutritional Objectives of the FONCODES Project in Peru .17 2.5 Case Study: Mixed Methods in Quantitative and Qualitative Approaches .19 2.6 Case Study: An Example of an Ex Ante Evaluation 21 ix Appendix: Programs and Files for Chapter 12–16 Exercises ****IV with FE implementation in panel data; use \data\hh_9198,clear; gen lexptot=ln(1+exptot); gen lnland=ln(1+hhland/100); gen vill=thanaid*10+villid; egen villmmf=max(dmmfd), by(vill year); gen mchoice=villmmf==1 & hhland