Research methods for piblic administrators gail johnson, routledge, 2014 scan

Research Methods for Public Administrators This page intentionally left blank Research Methods for Public Administrators Third Edition Gail Johnson First published 2014 by M.E Sharpe Published 2015 by Routledge Park Square, Milton Park, Abingdon, Oxon OX14 4RN 711 Third Avenue, New York, NY 10017, USA Routledge is an imprint of the Taylor & Francis Group, an informa business Copyright © 2014 Taylor & Francis All rights reserved No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, now known or hereafter invented, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers Notices No responsibility is assumed by the publisher for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use of operation of any methods, products, instructions or ideas contained in the material herein Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information, methods, compounds, or experiments described herein In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility Product or corporate names may be trademarks or registered trademarks, and are used only for identification and explanation without intent to infringe Library of Congress Cataloging-in-Publication Data Johnson, Gail, 1947– Research methods for public administrators / by Gail Johnson.—Third edition pages cm Includes bibliographical references and index ISBN 978-0-7656-3714-7 (pbk : alk paper) Public administration—Research—Methodology I Title JF1338.A2J64 2014 351.072—dc23 2013038332 ISBN 13: 9780765637147 (pbk) Contents List of Illustrations Preface and Acknowledgments xi xv Introduction: Research Methods for Public Administrators Overview Goals: Research as a Critical Thinking Tool Research in the Public Sector What Is Research? Types of Research Ethics and Principles of Good Research Overview of This Book Exercises 3 11 13 15 16 Basic Research Concepts Overview The Secret Language of Social Science Theory Hypothesis in Its Many Forms Variables Values Levels of Measurement Determining Causality Independent and Dependent Variables Control Variables Direction of Relationships Program Evaluation: Research in the Public Sector Using Models for a Holistic View of Relationships The Logic Model Applying the Logic Model 17 17 18 18 18 20 20 20 21 22 23 23 24 24 27 29 v CONTENTS vi Conclusion Exercises 30 31 What Is the Question? Overview Determining the Research Question Learning from Others Engaging the Stakeholders Working Together Types of Questions Descriptive Questions Normative Questions Relationship Questions Conclusion Exercises 34 34 34 36 37 39 40 41 42 44 45 46 Identifying Measures and Measurement Strategy Overview Defining Key Terms Conceptual Definitions Operational Definitions Setting Boundaries Valid and Reliable Measures Validity Reliability Why Measurement Matters Conclusion Exercises 48 48 48 48 50 53 55 55 58 59 61 62 Designs for Research: The Xs and Os Framework Overview Designing an Experiment Applying the Design Elements: The Xs and Os Framework Nonexperimental Design Quasi-Experimental Design Classic Experimental Design Design Variations Using Statistical Controls to Create Comparison Groups Longitudinal Studies Internal Validity Why Validity Matters External Validity Conclusion Exercises 64 64 65 65 66 67 68 70 71 73 75 77 79 81 82 CONTENTS vii Other Research Approaches Overview Secondary Analysis of Data Evaluation Synthesis (Meta-Analysis) Content Analysis Survey Research Case Studies Cost-Benefit Analysis Conclusion Exercises 84 84 84 85 86 90 90 92 95 96 Data Collection I: Available Data and Observation Overview Data Collection: The Degree of Structure Available Data Data Collection Instruments Observation Obtrusive and Unobtrusive Data Collection The Design Matrix Conclusion Exercises 97 97 98 101 102 105 106 109 109 110 Data Collection II: Interviews and Focus Groups Overview General Guidelines About Choosing the Appropriate Method Encouraging Participation In-Person Interviews Focus Groups Other Group Data Collection: Expert Panels, Public Hearings Conclusion Exercises 112 112 112 114 115 118 122 125 125 Data Collection III: Surveys Overview Basic Methods Response Rates Telephone Surveys Mail Surveys Cyber-Research: E-mail and Web-Based Surveys Developing Closed-Ended Questions Using One-Way and Two-Way Intensity Scales Ranking Questions Demographic Questions Conclusion Exercises 127 127 128 128 130 133 134 134 137 140 141 142 142 viii CONTENTS 10 Sampling Demystified Overview Sampling Jargon Random and Nonrandom Samples Random Samples Nonrandom Samples Random Samples: The Options Simple Random Sample Systematic Random Sample Stratified Random Sample Proportional Stratified Sample Disproportionate Stratified Sample Cluster Sample Nonrandom Samples: The Options Determining Sample Size Nonsampling Errors Conclusion Exercises 145 145 146 147 147 148 148 148 150 150 151 151 153 155 156 159 160 161 11 Qualitative Data Analysis Overview Analyzing Qualitative Data Identifying Themes and Quotes Working with Qualitative Data Conclusion Exercises 162 162 162 164 166 169 169 12 Data Analysis for Description Overview Simple Descriptive Statistics in Public Administration Commonly Used Descriptive Statistics Counts Percents Rates Ratios Rates of Change Distributions Measures of Central Tendency Which Measure to Use? Comparison of Means Measures of Dispersion Conclusion Exercises 171 171 171 175 175 176 177 177 177 178 179 180 181 182 184 185 CONTENTS ix 13 Analyzing Survey Scales Overview Handling Exits and the Middle of a Five-Point Scale Setting Benchmarks and Extreme Analysis Handling the Middle Category in One-Way Intensity Scales Should Means Be Used with Nominal and Ordinal Scales? The Analytical Tool: Cross Tabulations Conclusion Exercises 186 186 186 189 191 193 193 194 195 14 Data Analysis: Exploring Relationships Overview Using Crosstabs to Examine Relationships Controlling for a Third Variable Exploring Relationships: Comparison of Means and Medians Measures of Association Frequently Used Measures of Association Working with Interval or Ratio Data Conclusion Exercises 198 198 198 201 202 205 208 209 210 212 15 Data Analysis: Regression Overview Bivariate Regression: Key Elements Using Bivariate Regression Analysis: Sunshine and Tourism Multiple Regression Beta Weights: Relative Predictive Strength Regression in the News Why Did the Violent Crime Rate Drop After 1991? Conclusion Exercises 216 216 216 218 219 222 224 225 228 228 16 Data Analysis Using Inferential Statistics Overview Statistical Significance: Basic Concepts The Logic of Statistical Significance Testing Errors in Tests for Statistical Significance Common Tests for Statistical Significance Chi-Square t-Tests: Analyzing Difference in Means Analysis of Variance Tests for Statistical Significance in Regression Analysis Reporting Results of Statistical Significance 230 230 230 232 233 234 235 237 238 239 240 CHAPTER 70 Congress would randomly assign some states to a program and others to a comparison group that does not get the program When laws are implemented in every county or every state at the same time, there is no way to form comparison groups Random assignment might also pose a challenge to political deal making For example, it is difficult to randomly assign cities to receive a program that will bring in millions of dollars from the federal government Elected officials from other cities are likely to negotiate to have their cities included in the experiment As a result, the cities are no longer randomly assigned (because political influence has gotten them included) and the pot of money might be divided up to the point where it is insufficient to be effective in any city There may be situations, however, when an experimental design can be used For example, when a program is not large enough to accommodate all those who apply, random assignment is possible (see Application 5.2 on pages 72–73) Since relatively few can participate, random assignment not only provides a strong design, it is also more equitable because it rules out bias and favoritism in selection In the mid-1980s, for example, a public-private partnership funded a huge evaluation of welfare training programs (Gueron 1988) Eight states were selected, and over 35,000 people who volunteered to participate in the training program were randomly assigned to one of three groups: the full training program, a program with limited job search services, or the no-service control group In states with a typical unemployment rate, the programs worked; those who participated in the full training program earned more than those who did not, although the differences were generally small This is a good example of how an experimental design can be used in the public sector DESIGN VARIATIONS Every situation is different and therefore constrains the ability to use one or more of the design elements In some situations, random assignment is possible but there is no way to get a preprogram measure For example, suppose researchers are testing a training program for people on welfare The researchers go through the welfare rolls in their town and randomly assign clients to the training program or not The outcome measure is whether they get jobs and keep them for six months Since they are on welfare when the program begins, there is no premeasure Some people may have had prior work experience, but random assignment equalizes the groups since people with and without prior work experience are in both groups My point, however, is that there will be no premeasure in this experimental design The notation using Xs and Os would look like this: Random assignment Random assignment Training program Comparison group X O O Sometimes researchers stumble on a situation that is naturally a quasi-experimental design For example, until Indiana’s state legislature decided to put the entire state on daylight saving time in 2006, only fifteen of its ninety-two counties turned their clocks forward in the spring and back in the fall (Lahart 2008) This created a natural DESIGNS FOR RESEARCH 71 experiment by which the energy use in the counties could be compared Using data from monthly meter readings for three years, researchers were able to compare energy use before and after daylight saving went into effect as well as compare the usage of residents living in counties that were using daylight saving before 2006 with those who were not The result? The researchers concluded that daylight saving time does not save energy or money: residential energy usage increased between percent and percent, amounting to an additional $8.6 million a year that consumers paid Another quasi-experimental variation is to find a group that is matched on key characteristics Researchers might, for example, select two schools that share very similar demographic characteristics to compare a specific program offered in one school but not the other The assumption is that if the demographics are the same, the schools are relatively comparable, so any observed difference is likely to be due to the program USING STATISTICAL CONTROLS TO CREATE COMPARISON GROUPS Creating comparison groups by using statistical controls is another very common quasi-experimental design In the jargon, this type of design is typically called correlational with statistical controls, but variations are called an ex post facto or causal comparative design Basically, they use analytical techniques utilizing computer software to make comparisons Suppose researchers want to determine whether the Head Start program has a lasting impact on the reading abilities of the children who participate in the program Assuming the data is available in school records, the researchers design a study to gather information on all the eighth graders in an inner-city school district This data includes whether or not they attended Head Start, various test scores, grades, and demographic information Using statistical software, the researchers separate the students into two groups: those who attended Head Start and those who did not The software then performs various statistics procedures to determine whether there is a noticeable difference in reading scores between the former Head Start and the non–Head Start students Specifically how this is done will be presented in the analysis chapters What else might affect a child’s reading scores? Maybe the educational level of the mother, family income, or having attended nursery school makes a difference If these data are included in the files, they can be used to test the possible rival explanations, and the researchers use statistical controls to examine reading scores associated with these factors The computer software sets up comparison groups For example, the computer can look at reading scores for children who attended nursery school, Head Start, or those who did not attend any type of preschool If Head Start makes a difference, we would expect to see higher reading scores for that group as compared to those who did not attend any preschool groups Ideally, the Head Start group would have scores at least as high as those who attended nursery schools Using statistical controls is a strong quasi-experimental design to employ in the field We often see statistical controls used in analyzing polling data For example, a national exit poll conducted by NBC was used to analyze how young people voted in CHAPTER 72 !PPLICATION Measuring the Impact of Medicaid $IFFERENT$ESIGNS $IFFERENT-EASURES !STHE!FFORDABLE(EALTH#ARE!CT/BAMA#ARE ISPOISEDTOEXPAND-EDICAIDTOCOVERMANY PEOPLECURRENTLYWITHOUTHEALTHINSURANCE SOMERESEARCHERSUNDERTOOKSTUDIESTODETERMINE -EDICAIDSIMPACT/NEISTHE/REGON(EALTH)NSURANCE%XPERIMENT"AICKERETAL AND THESECONDISASTUDYTHATCOMPARESTHEHEALTHOUTCOMESOFTHREESTATESTHATEXPANDED-ED ICAIDTOTHOSEOFNEIGHBORINGSTATESTHATDIDNOT3OMMERSETAL  !CCORDINGTOTHE/REGON(EALTH)NSURANCE%XPERIMENT7EBSITETHE EXPERIMENTISALANDMARK RANDOMIZEDSTUDYOFTHEEFFECTOFEXPANDINGPUBLICHEALTHINSURANCE ONTHEHEALTHCAREUSE HEALTHOUTCOMES lNANCIALSTRAIN ANDWELL BEINGOFLOW INCOMEADULTS)T REPRESENTSTHElRSTUSEOFARANDOMIZEDCONTROLLEDDESIGNTOEVALUATETHEIMPACTOF-EDICAIDIN THE5NITED3TATES 4HISWASPOSSIBLEBECAUSETHESTATEOF/REGONDECIDEDTOOPENUPITS-EDICAIDPROGRAMTO LOW INCOME NONELDERLYADULTSINANDUSEDALOTTERYTOSELECTTHOSEWHOWOULDBEELIGIBLE TOAPPLY%XPERIMENTALDESIGNSARENOTTYPICALLYUSEDINGOVERNMENTPROGRAMSREQUIRINGTHAT ALLELIGIBLEPEOPLERECEIVESERVICES$ENYINGSERVICESINTHOSESITUATIONSRAISESBOTHLEGALAND ETHICALCONCERNS(OWEVER /REGONSEXPANSIONDIDNOTHAVETOPROVIDESERVICESTOEVERYONE WHOWASELIGIBLETHEYLIMITEDAVAILABILITY ANDWITHMOREPEOPLETHANAVAILABILITY ALOTTERYWAS AFAIRMETHOD 7HILEEXPERIMENTALDESIGNSAREOFTENTOUTEDASTHEhGOLDSTANDARDvINRESEARCH ALLTHATGLIT TERSISNOTNECESSARILYGOLD"YTHETIMETHERESEARCHERSCONDUCTEDTHESECONDPHASEOFTHE STUDY .. .Research Methods for Public Administrators This page intentionally left blank Research Methods for Public Administrators Third Edition Gail Johnson First... This page intentionally left blank Research Methods for Public Administrators This page intentionally left blank Introduction Research Methods for Public Administrators OVERVIEW With the tsunami... Methods for Public Administrators Overview Goals: Research as a Critical Thinking Tool Research in the Public Sector What Is Research? Types of Research Ethics and Principles of Good Research