Not-Getting-Lost-in-the-Numbers-Program-and-Outcome-Evaluation-Basics-Larwin-Dr.-K

Not Getting Lost in the Numbers: Program and Outcome Evaluation Basics Youngstown State University Define “evaluation” and why it is conducted; Discuss the importance of a map or “logic model”; Explore what can be measured; And, develop ideas for evidence based conclusions Dictionary definition: “to establish the worth of value of” Specific definition: It is a systematic application of scientific methods to assess the design, implementation, improvement or outcomes of a program (Short, Hennessy, & Campbell, 1996) A program is defined as an organized action such as a media campaign, service provision, educational service, public policy, research project, etc (CDC, 1999) In an effort to … demonstrate program effectiveness; identify strengths and weaknesses of program delivery; provide evidence of resource management; track the positive impacts of program delivery; justify current funding; to support the acquisition of future funding or continued funding; provide a framework for program replication; to satisfy ethical responsibilities to ALL stakeholders (Short et al., 1996) Evaluation is not just an EVENT that occurs at the end of a project… “but it is an ongoing process which helps decision makers to: • Better understand the project • Better understand how the project is impacting the participants; • Better understand how the project is impacting the partners and the community; • And better understand how the program is being impacted by internal and external factors (W.K.Kellogg Foundation Handbook on Program Evaluation, p.3 While research seeks to prove… Evaluation seeks to IMPROVE! M.Q Patton So, STEP #1 is to develop a plan… Wisconsin Flow Chart Logic Model Short Term Medium Term Long Term “Learning” “Action” “Conditions” Awareness Behavior Social Knowledge Practice Economic Attitudes Decision-Making Civic Skills Policies Environmental Opinions Social Action Examples: Aspirations *increase achievement Motivations *improve health Wisconsin Flow Chart Logic Model What is being done What is being done poorly and how we correct? What is being done well and why? How is that being accomplished? How well is X being accomplished? • Utility • Who needs evaluation results? • What they need? • Feasibility • Are proposed measures feasible – based on timeline and resources? • Propriety • Does the evaluation protect identities of participants? • Does the project fulfill deliverables? • Accuracy • Will the data provide reliable and valid information? Measuring Impact? Reporting that all of the attendees were happy with the presentation does not measure IMPACT STEP ONE: WHAT IS THE GOAL OF THE PRESENTATION? Measuring Impact? • Measure awareness before and after training; • Measure knowledge with a pre and post testing; • Measure attitudes with pre- and post-training assessment, etc …whether that target is knowledge, awareness, attitudes, et cetera! Questions? How “Action” can be measured? Following the logic of the model, needed professional development or training has been provided (short term) …and what was gained at this training should result in measurable actions STEP 2: What ACTIONable response is seen? Measuring “Action”? Think about the target (or goal)! What would demonstrate that the mid-term goals are being met? Measuring “Action”? ACTION measures should be observable, either directly (observed) or indirectly ( such as through indications from stakeholders or productivity) Measuring “Action”? How can behavior be measured? How can practice be evaluated? How can decision making be assessed? Et cetera? Your organization does not want to work hard just so they can say they are making things spin! If you are not seeing a measurable impact…STOP and find out why! If the TEAM is turning the right gears… …a MEASUREABLE PROGRAM IMPACT should occur! What are you doing? Training? Activity? Programming? Why are you doing the training/ activity/ programming? Who are your target participants? Who are your Stakeholders? What is the expected impact? How are you measuring this impact? How close are you to the expected impact? How can you improve? Karen H Larwin, PhD khlarwin@ysu.edu