2006 regression analysis by example

Regression Analysis by Example WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A SHEWHART and SAMUEL S WILKS Editors: David J Balding, Noel A C Cressie, Nicholas I Fisher, Iain M Johnstone, J B Kadane, Geert Molenberghs, Louise M Ryan, David I$? Scott, Adrian F M, Smith, Jozef L Teugels Editors Emeriti: Vic Barnett, J Stuart Hunter, David G Kendall A complete list of the titles in this series appears at the end of this volume Regression Analysis by Example Fourth Edition SAMPRIT CHATTEFUEE Department of Health Policy Mount Sinai School of Medicine New York, NY ALI S HAD1 Department of Mathematics The American University in Cairo Cairo, Egypt WILEY- INTERSCl ENCE A JOHN WILEY & SONS, INC., PUBLICATION Copyright 02006 by John Wiley & Sons, Inc All rights reserved Published by John Wiley & Sons, Inc., Hoboken, New Jersey Published simultaneously in Canada No part of this publication may be reproduced, stored in a retrieval system, or transmitted in any form or by any means, electronic, mechanical, photocopying, recording, scanning, or otherwise, except as permitted under Section 107 or 108 of the 1976 United States Copyright Act, without either the prior written permission of the Publisher, or authorization through payment of the appropriate per-copy fee to the Copyright Clearance Center, Inc., 222 Rosewood Drive, Danvers, MA 01923, (978) 750-8400, fax (978) 750-4470, or on the web at www.copyright.com Requests to the Publisher for permission should be addressed to the Permissions Department, John Wiley & Sons, Inc., 11 River Street, Hoboken, NJ 07030, (201) 748-601 I, fax (201) 748-6008, or online at http://www.wiley.com/go/permission Limit of Liability/Disclaimer of Warranty: While the publisher and author have used their best efforts in preparing this book, they make no representations or warranties with respect to the accuracy or completeness of the contents of this book and specifically disclaim any implied warranties of merchantability or fitness for a particular purpose No warranty may be created or extended by sales representatives or written sales materials The advice and strategies contained herein may not be suitable for your situation You should consult with a professional where appropriate Neither the publisher nor author shall be liable for any loss of profit or any other commercial damages, including but not limited to special, incidental, consequential, or other damages For general information on our other products and services or for technical support, please contact our Customer Care Department within the United States at (800) 762-2974, outside the United States at (3 17) 572-3993 or fax (3 17) 572-4002 Wiley also publishes its books in a variety of electronic formats Some content that appears in print may not be available in electronic format For information about Wiley products, visit our web site at www.wiley.com Library of Congress Cataloging-in-Publieation Data: Chatterjee, Samprit, 1938Regression analysis by example - 4th ed / Samprit Chatterjee, Ah S Hadi p cm Includes bibliographical references and index ISBN-I3 978-0-471-74696-6 (cloth : acid-free paper) ISBN-I0 0-471-74696-7 (cloth : acid-free paper) Regression analysis Title QA278.2.C5 2006 519.5’36dc22 Printed in the United States of America 10 2006044595 Dedicated to: Allegra, Martha, and Rima - S C My mother and the memory of my father - A S H It’s a gift to be simple Old Shaker hymn True knowledge is knowledge of why things are as they are, and not merely what they are Isaiah Berlin CONTENTS Preface Introduction 1.1 1.2 1.3 1.4 1.5 What Is Regression Analysis? Publicly Available Data Sets Selected Applications of Regression Analysis 1.3.1 Agricultural Sciences 1.3.2 Industrial and Labor Relations 1.3.3 History 1.3.4 Government 1.3.5 Environmental Sciences Steps in Regression Analysis 1.4.1 Statement of the Problem Selection of Potentially Relevant Variables 1.4.2 1.4.3 Data Collection 1.4.4 Model Specification 1.4.5 Method of Fitting 1.4.6 Model Fitting 1.4.7 Model Criticism and Selection 1.4.8 Objectives of Regression Analysis Scope and Organization of the Book Exercises xiii 1 3 6 11 11 11 12 14 14 16 16 17 18 vii Viii CONTENTS Simple Linear Regression 21 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.1 2.12 21 21 26 28 29 32 37 37 39 42 44 45 45 Multiple Linear Regression 53 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 53 53 54 57 58 60 61 62 64 66 3.10 3.1 Introduction Covariance and Correlation Coefficient Example: Computer Repair Data The Simple Linear Regression Model Parameter Estimation Tests of Hypotheses Confidence Intervals Predictions Measuring the Quality of Fit Regression Line Through the Origin Trivial Regression Models Bibliographic Notes Exercises Introduction Description of the Data and Model Example: Supervisor Performance Data Parameter Estimation Interpretations of Regression Coefficients Properties of the Least Squares Estimators Multiple Correlation Coefficient Inference for Individual Regression Coefficients Tests of Hypotheses in a Linear Model 3.9.1 Testing All Regression Coefficients Equal to Zero 3.9.2 Testing a Subset of Regression Coefficients Equal to Zero 3.9.3 Testing the Equality of Regression Coefficients Estimating and Testing of Regression Parameters 3.9.4 Under Constraints Predictions Summary Exercises Appendix: Multiple Regression in Matrix Notation 69 71 73 74 75 75 82 Regression Diagnostics: Detection of Model Violations 85 4.1 4.2 4.3 4.4 4.5 85 86 88 90 93 Introduction The Standard Regression Assumptions Various Types of Residuals Graphical Methods Graphs Before Fitting a Model CONTENTS 4.6 4.7 4.8 4.9 4.10 4.1 4.12 4.13 4.14 4.5.1 One-Dimensional Graphs 4.5.2 Two-Dimensional Graphs 4.5.3 Rotating Plots 4.5.4 Dynamic Graphs Graphs After Fitting a Model Checking Linearity and Normality Assumptions Leverage, Influence, and Outliers Outliers in the Response Variable 4.8.1 4.8.2 Outliers in the Predictors 4.8.3 Masking and Swamping Problems Measures of Influence 4.9.1 Cook’s Distance 4.9.2 Welsch and Kuh Measure 4.9.3 Hadi’s Influence Measure The Potential-Residual Plot What to Do with the Outliers? Role of Variables in a Regression Equation 4.12.1 Added-Variable Plot 4.12.2 Residual Plus Component Plot Effects of an Additional Predictor Robust Regression Exercises iX 93 93 96 96 97 97 98 100 100 100 103 103 104 105 107 108 109 109 110 114 115 115 Qualitative Variables as Predictors 121 5.1 5.2 5.3 5.4 121 122 125 128 130 137 138 139 140 141 143 5.5 5.6 5.7 Introduction Salary Survey Data Interaction Variables Systems of Regression Equations 5.4.1 Models with Different Slopes and Different Intercepts 5.4.2 Models with Same Slope and Different Intercepts 5.4.3 Models with Same Intercept and Different Slopes Other Applications of Indicator Variables Seasonality Stability of Regression Parameters Over Time Exercises Transformationof Variables 151 6.1 6.2 6.3 151 153 155 156 158 Introduction Transformations to Achieve Linearity Bacteria Deaths Due to X-Ray Radiation 6.3.1 Inadequacy of a Linear Model 6.3.2 Logarithmic Transformation for Achieving Linearity X CONTENTS 6.4 6.5 6.6 6.7 6.8 6.9 6.10 179 7.1 7.2 179 180 180 182 183 185 194 196 Introduction Heteroscedastic Models 7.2.1 Supervisors Data 7.2.2 College Expense Data Two-Stage Estimation Education Expenditure Data Fitting a Dose-Response Relationship Curve Exercises The Problem of Correlated Errors 8.1 8.2 8.3 8.4 8.5 8.6 8.7 8.8 8.9 8.10 159 164 66 167 168 169 173 174 Weighted Least Squares 7.3 7.4 7.5 Transformations to Stabilize Variance Detection of Heteroscedastic Errors Removal of Heteroscedasticity Weighted Least Squares Logarithmic Transformation of Data Power Transformation Summary Exercises Introduction: Autocorrelation Consumer Expenditure and Money Stock Durbin-Watson Statistic Removal of Autocorrelation by Transformation Iterative Estimation With Autocorrelated Errors Autocorrelation and Missing Variables Analysis of Housing Starts Limitations of Durbin-Watson Statistic Indicator Variables to Remove Seasonality Regressing Two Time Series Exercises 197 197 198 200 202 204 205 206 210 21 I 214 216 Analysis of Collinear Data 22 9.1 9.2 9.3 9.4 9.5 22 222 228 233 239 240 24 243 246 9.6 9.7 Introduction Effects on Inference Effects on Forecasting Detection of Multicollinearity Centering and Scaling Centering and Scaling in Intercept Models 9.5.1 Scaling in No-Intercept Models 9.5.2 Principal Components Approach Imposing Constraints ... ,X p , , X , can be approximated by the regression Regression Analysis by Example, Fourth Edition By Samprit Chatterjee and Ali S Hadi Copyright @ 2006 John Wiley & Sons, Inc 2 INTRODUCTION... guided by the principles and concepts of exploratory data analysis Our presentation of the various concepts and techniques of regression analysis relies on carefully developed examples In each example, .. .Regression Analysis by Example WILEY SERIES IN PROBABILITY AND STATISTICS Established by WALTER A SHEWHART and SAMUEL S WILKS Editors: David