Design of the evaluation

George C Homas (1949): People who write about methodology often forget that it is a matter of strategy, not of morals There are neither good nor bad methods but only the methods that are: More effective • • Design of the evaluation includes here: • in s ce n a The design st s al o m u i ve t g c r Designing the process i and n ct taoutcome c e j r la ob dis evaluation u tic ing to a r a ch y p Important concepts: r rea wa validity and e d e unit of analysis th Un On • Design informal: self- evaluation, expert judgment design Less • Design formal: one group and extended effective onePresenters: group design ++ R.M Kaphle Modestus Uzokwe The design: indicates • Which people or unit will be studied? • How they are to be selected? • Which kind of comparison will be drawn? • What is the timing of the investigation? So key in design are: Status Method Comparison Change/Longitudinal Process Evaluation: Is program is being conducted & producing output as planned? How can process can be improved? • In fact not very different from monitoring- similar kind of inquiry on both But some differences: • Often conducted for the benefit of the program • Help to understand the program: what it has been doing and how? • Lead to reflect on how it might improves its operations? • More systematic and relay more on data and less on intuitive judgments A program evaluation has to be designed- whether it is Q or Q method of investigation First: evaluator has to decide -which sites to study (large or small, few or several(class room with in the school) and which people to query - which sample strategy to use - Time period: which data, what interval, how intensively In qualitative(Yin): make design an integral feature So, major advantage in process evaluation= “opportunity to find the unexpected” - The evaluator can follow the trail wherever it leads - Also learns which aspects of the program matter to staff and to participants - What elements of the program are relevant to whom at which time? When should an evaluator conduct a process evaluation? • When she knows little about the nature of the program & its activities • When the program represents a marked departure from ordinary method of service? • When the theories underlying the program – implicitly or explicitly are doubtful, disputable or problematic • …in such cases a wider gauge inquiry is worthwhile However, again there is the option of quantitative measures of program process: design forms, interviews, survey So, whichever method or combination of methods is used for process evaluation the evaluator has to decide the frequency with which information will be collected Key methodological components to consider in process evaluation Methodologi cal component General definition Example – qualitative and quantitative methods Design Timing of data collection: when and how often data will be collected Observe classroom activities at least twice per semester with at least weeks of observation Conduct focus groups with participants in the last month of the intervention Data sources Source of information (for example, who will be surveyed, observed, interviewed) Both qualitative and quantitative – data sources include participants, teachers/staff delivering sessions records, the environment etc Data collection tools/ measures Instruments, tools and guides used for gathering process-evaluation data Both qualitative and quantitative – tools include surveys, checklists, observations forms, interview guides etc Data collection procedures Protocols for how the data collection tool will be administered Detailed description of how to quantitative/ qualitative classroom observation, face-to-face or phone interview, mailed survey, focus group etc Data managemen t Procedures for getting data from field and entered – plus quality checks Staff turn in participant sheets weekly, evaluation coordinator collects and checks surveys and gives them to data entry staff Interviews transcribed and tapes submitted at the end of the month Data analysis Statistical and/or qualitative methods used to analyse or summarise data Statistical analysis and software that will be used to analyse the quantitative data Types of qualitative analysis used Outcome evaluation: to what extent have a program’s goals have been met? fold: Before –program- after Diagram of an experiment Program Random participants assignment Control group Before After a b c d b-a=y d-c=z y-z= net outcome of the program If +, positive net outcome b-a= out comes for participants= y d-c= out comes for the control group=z y-z= the net outcome of the program More on chapter- Selection Attrition Outside effects Maturation Testing Instrumentatio n If there is no reason to suspect that selection or attribution or outside events or any other threats are getting in the way, then experimental design is a luxury Evaluator is doing an evaluation to inform the policy community about the program & to help decision makers make wise decisions Her task is to provide best information possible What the evaluator needs to be concerned about is: countering credible threats to valid conclusions In the absence of experimental design, conditions other than the program can be responsible for observed outcomes chief among them are: • • • • • • Important concepts • Validity: the extent to which the indicator actually captured the concept that it aimed to measure • Valid findings describe the way things actually are • Internal: the causal link between independent(program inputs: describes the participants or features of the service they receive) and dependent( observed out comes of the program) variables • External/ generalizability: whether the findings of one evaluation can be generalized to apply to other programs of similar type Unit of analysis and Unit of sampling UOA: the unit that the evaluation measures and enters in to statistical analysis or it is the unit that figures in data analysis The unit that receives a program is usually a person but it can be: a group, an organization, or a community or any other definable and observable unit Researchers used to worry about the appropriate UOA e.g analysis of higher level of department of health not possible to rich sound conclusion about individuals with in those departments.= Ecological fallacy UOS: is the entity i.e selected in to the program e.g “ study of a nutrition education program carried out in supermarkets” In program conducted in school classroom are usually the unit sampled There may be stages of sampling 1st schools can be sampled followed by choice of classrooms within the selected school However, techniques of multi-level analysis make it possible to analyze data at several levels simultaneously A matter that requires continuing care is the unit of measurement Formal: i)One group design • Does not include any comparison with units • categories of evaluation: after only(AO) and before and after(BA) • A) After only: the evaluator finds out what the situation is after the program by examine records, if unavailable the evaluator can use historical comparisons E.g compare test result at the end semester(3rd) with history of test result(1st and 2nd) This adds useful information but there are drawbacks Changing situation of school, diff population, test system, teachers… Question of validity of data, reliability of peoples memory ,= real difficulty inducing a reliable baseline with out records BUT certain matter of facts such as age, school complete time, whether they were un/employed, what kind of work they are doing such responses are fairly trustworthy Another way through the “ use of experienced judgment” B)Before and after: Look at program recipients before they enter the program and then again in the end Cohort Cohort Cohort Cohort Before After Before After Before One group After Before After Result: was the change in their skills or health or income due to the program? May be or may not be ??? But if the data collected with care and systematically, they offer considerable information about the program and the people who participate in it Then the evaluation will be able to say much about how the program is working Advantages and disadvantages of one group design Advantaged They provide a preliminary look at the effectiveness of a program Disadvantages Program effects may be underestimated if outside events operates to counter act program efforts Evaluators mind eye Higher expectations Can not provide quick result Some agencies demand one-time ex-post facto investigation: short term needs quick results Evaluators will have to exploit every opportunity… ii)Extending one group design: one group design can be directions elaborated in basic During the program Collecting more data on what happens ? That is Much before & after the program MDDP: More data during the program DRD: Dose-Response- Design MDDP: More Data During the Program DRD: Dose Response Design • One way: During measure, or series of 3D “During-During- During” measure • Q&Q data can be collected in program services and on participant’s progress as they move through the program • The data can be analyzed to elaborate the picture of what happens during the program • Can identify the association between program events and participant outcomes • It can probe in to the relationships, among rules, recruitments, strategies, mode of organization, auspices, services, leadership, communication, feedback loops… & whatever else people are concerned about • Evaluation can compared participants who received a large quantity of service ( a high doge) with those who receive a lower doge • The notion that more is better may not always be right • In social programs, evaluator can make many internal comparisons • It is a highly versatile design • It can be used not only in one group designs but also when the evaluation includes comparison or control group…in ch- The assumed link between tenant participation in the mgmt of a public housing project and resident security Program Theory Model: MDDP Greater demand for security staff More security staff hired More patrol surveillance and arrests Tenant/resident representatives P.Theory hypothesizes Greater sense of individual responsibility for the project More neighborliness, social networks More people’s eyes on the street Tenant watch, Fear of troublemaker Link is not obvious Culture of mutual concern Evaluator then designs methods for collecting data Assumptions actually comes to pass, She locates sources, time intervals Inhabitation of violence Improved security We are discussing a variety of research design for evaluating the process and outcome of social programs Evaluator will want to use designs that are less vulnerable to bias and incompleteness… TIME-SERIES DESIGNS Presented by: MODESTUS TIME-SERIES DESIGNS Time-series data enable the evaluator to interpret the pre-to-post changes in the light of additional evidence They show whether the measures immediately before and after the program or whether they mark a decisive change This type of evaluation has the ability to follow participants for more than a year after their departure from the program; these can be apply on crime, welfare caseloads, hospital stays, medical costs, births, refugee resettlement etc The main criticism is that some outside events coincided with the program and were responsible for whatever change was observed between before and after For example, the program that accounted for the decline in smoking; it could have been the television series on the risks of smoking that came along at the same time Another way to cope with the possibility of contamination with outside events is by studying what happened over the same time period to people who did not receive the program but were exposed to the same external influences Comparison Group When these groups are not selected randomly from the same population as program recipients, we call them comparison groups The main purpose for the comparison is to see whether receiving the program adds something extra that comparison units not get these shows what program recipients would have been like if they had not been in the program Before-and-After with Comparison Group The comparison group is selected to be as much like the client group through any of a variety of procedures The evaluator can locate a similar site where the program not take place, compare the status of both group before the program began & test how similar they were However, each will compensate for differences that the other comparison leaves uncontrolled Multiple Time Series These are especially useful for evaluation policy changes, when jurisdiction passes a law; Example evaluator of the crackdown on highway speeding Evaluator collected reports of traffic fatalities for several periods before & after the new program went into effect They found that fatalities went down after police began strict enforcement of penalties for speeding Constructing a Comparison Group Effect can be assumed (Program – effect) No effect Initial differences persists However, the most important is to compare the actual situation with the predicted situation The difference between the two should be the effect of the program Forecasting the situation without the program To estimate the counterfactual – the condition that would have occurred if no program had been operating Thus, for example, an evaluation of a dropout prevention program can use past school data to project the rate of future school dropouts, on the assumption that past trend continues into the future (Assumes that any observed difference is due to the program) Lesson Learned There are two main goals of time series analysis: (a) identifying the nature of the phenomenon represented by the sequence of observations, and (b) forecasting (predicting future values of the time series variable) [...]... Result: was the change in their skills or health or income due to the program? May be or may not be ??? But if the data collected with care and systematically, they offer considerable information about the program and the people who participate in it Then the evaluation will be able to say much about how the program is working Advantages and disadvantages of one group design Advantaged They provide... to compare the actual situation with the predicted situation The difference between the two should be the effect of the program Forecasting the situation without the program To estimate the counterfactual – the condition that would have occurred if no program had been operating Thus, for example, an evaluation of a dropout prevention program can use past school data to project the rate of future school... whether the measures immediately before and after the program or whether they mark a decisive change This type of evaluation has the ability to follow participants for more than a year after their departure from the program; these can be apply on crime, welfare caseloads, hospital stays, medical costs, births, refugee resettlement etc The main criticism is that some outside events coincided with the. .. the program that accounted for the decline in smoking; it could have been the television series on the risks of smoking that came along at the same time Another way to cope with the possibility of contamination with outside events is by studying what happened over the same time period to people who did not receive the program but were exposed to the same external influences Comparison Group When these... Inhabitation of violence Improved security We are discussing a variety of research design for evaluating the process and outcome of social programs Evaluator will want to use designs that are less vulnerable to bias and incompleteness… TIME-SERIES DESIGNS Presented by: MODESTUS TIME-SERIES DESIGNS Time-series data enable the evaluator to interpret the pre-to-post changes in the light of additional evidence They... like the client group through any of a variety of procedures The evaluator can locate a similar site where the program do not take place, compare the status of both group before the program began & test how similar they were However, each will compensate for differences that the other comparison leaves uncontrolled Multiple Time Series These are especially useful for evaluation policy changes, when jurisdiction... reliable baseline with out records BUT certain matter of facts such as age, school complete time, whether they were un/employed, what kind of work they are doing such responses are fairly trustworthy Another way through the “ use of experienced judgment” B)Before and after: Look at program recipients before they enter the program and then again in the end Cohort 1 Cohort 2 Cohort 3 Cohort 4 Before After... the rate of future school dropouts, on the assumption that past trend continues into the future (Assumes that any observed difference is due to the program) Lesson Learned There are two main goals of time series analysis: (a) identifying the nature of the phenomenon represented by the sequence of observations, and (b) forecasting (predicting future values of the time series variable) ... randomly from the same population as program recipients, we call them comparison groups The main purpose for the comparison is to see whether receiving the program adds something extra that comparison units do not get these shows what program recipients would have been like if they had not been in the program Before-and-After with Comparison Group The comparison group is selected to be as much like the client... Response- Design c) Time Series Design 3) Comparison groups… 1.Self evaluation • Ask the people who are involved with it and what they think: For instant: Staffs: d2d, Administrators: +&- ,Clients: services, Outsiders: press, community, other org like or dislike about the program • Shortcomings; if people know it is for evaluation: most favorable reporting • Despite the weaknesses, these kind of judgments ... generalizability: whether the findings of one evaluation can be generalized to apply to other programs of similar type Unit of analysis and Unit of sampling UOA: the unit that the evaluation measures... However, the most important is to compare the actual situation with the predicted situation The difference between the two should be the effect of the program Forecasting the situation without the. .. and after the program or whether they mark a decisive change This type of evaluation has the ability to follow participants for more than a year after their departure from the program; these can