EVALUATION AND SOCIO-ECONOMIC DEVELOPMENT
Evaluations that make a difference
Investing in evaluation is essential for enhancing the success of policies and programs in socio-economic development, as it aims to improve the social and economic prospects of individuals and communities Each intervention has its unique rationale, focusing on aspects such as productivity in small and medium-sized enterprises, modern infrastructure development, urban regeneration, integration of disadvantaged groups, or rural diversification, all of which are reflected in European Cohesion Policy programs The key justification for evaluation across these diverse priorities is to determine how evaluation methods can enhance the quality of life and opportunities for citizens To achieve meaningful outcomes, evaluations must address pertinent questions that benefit all stakeholders, including managers, policymakers, and beneficiaries.
Evaluation plays a crucial role in innovative policy areas where success is uncertain and implementation can be challenging It facilitates sophisticated management and planning, helping to navigate the complexities and unavoidable uncertainties inherent in socio-economic development As this field is not an exact science, it necessitates careful goal-setting, program design, and policy implementation Effective evaluation requires thorough analysis, anticipation of outcomes, the establishment of feedback systems, and the engagement of various institutions, agencies, and community groups to sustain development efforts.
Evaluation expertise and its application play a crucial role in enhancing socio-economic development initiatives, making it an essential element in these processes.
There are two important implications if we justify evaluation in these terms:
For evaluation to be effective and impactful, it must be embedded within decision-making processes and management, serving as a key component of democratic accountability A robust evaluation system should be seamlessly integrated into the policy cycle This GUIDE emphasizes the importance of designing effective evaluation systems and fostering evaluation capacity within public agencies and professional networks.
Evaluators and stakeholders must balance optimal evaluation methods with practical constraints, as socio-economic development often lacks the time, resources, and data for comprehensive evaluations This GUIDE emphasizes strategic decision-making in evaluation, addressing questions such as when to invest more in evaluations, the circumstances that necessitate advanced methods, and how evaluations can address knowledge gaps that ideally would have been resolved prior to intervention planning.
Improving policies over time
One important organising principle that runs through the GUIDE is the time-line of policy.
The policy cycle encompasses the formulation of policies and programs, progressing through planning, resource allocation, program design, implementation, and the delivery of results Evaluation terminology aligns with this cycle, evident in the use of terms like ex ante, interim, and ex post evaluation within European Cohesion Policy Additionally, it is important to note that the programming and evaluation life cycles intersect across various programming periods.
Understanding the three key time cycles in evaluation is essential for effective assessment The first is the evaluation cycle, which takes place at various stages within the second cycle The second cycle, known as the programme cycle, creates a demand for these evaluation moments Lastly, the policy cycle, which is generally longer than the programme cycle, plays a crucial role in shaping and influencing both programmes and their evaluation requirements.
Figure 1 Policy, programme and evaluation cycles
Designing programmes
A key competency in evaluation is the ability to collect information from various stakeholders, which is crucial during the programme design phase Ensuring that programmes align with the needs of the community is vital, and evaluation plays a significant role in achieving this relevance Furthermore, the process of gathering feedback is not limited to a single point in time; in many socio-economic development initiatives, ongoing input from both potential and actual users, as well as other stakeholders, is essential for maintaining programme effectiveness.
Evaluation plays a crucial role in program design by guiding stakeholders in making informed decisions about program targeting It helps identify specific needs that require attention and outlines the potential changes that could occur if the program achieves its goals By addressing these critical questions, evaluation supports policymakers and planners in their strategic planning efforts.
Choosing between instruments
A well-structured evaluation system, especially ex ante evaluation, plays a crucial role in selecting appropriate instruments or interventions This process often includes an economic appraisal to evaluate the potential costs and benefits of various alternatives, including large projects Additionally, it involves assessing eligibility to align specific interventions with relevant criteria within a broader program or regulatory framework A key aspect of this evaluation is examining the clarity and credibility of the proposed intervention, which helps gauge its likelihood of success.
Improving management and delivery
A comprehensive evaluation process significantly enhances programme management and delivery by analyzing monitoring data and identifying the root causes of challenges faced This evaluation provides valuable feedback for programme management, facilitating timely adjustments Early outputs can be identified, particularly when a clear implementation framework and intervention logic are established, highlighting the importance of assessing implementation from the outset Many initial challenges revolve around processes, including stakeholder interactions, decision-making, and the establishment of new organizational structures and partnerships Evaluating these processes, even through simple descriptions, benefits all stakeholders, including sponsors and managers.
Identifying outputs and results and analysing impacts
Socio-economic development programs must showcase early achievements, typically through measurable outputs like the number of firms utilizing subsidies for equipment upgrades or unemployed individuals receiving training However, policymakers ultimately seek more significant outcomes, such as enhanced competitiveness for firms or successful job placements for the unemployed To determine the true impact of these policies, thorough evaluations are essential, addressing critical questions like whether the growth of regional firms is sustainable and if the long-term unemployed have experienced lasting improvements in their employment prospects.
Identifying unintended consequences and perverse effects
Even when programs achieve their intended goals, they can result in unintended consequences, which may be either positive or negative For instance, support for rural entrepreneurs can inadvertently benefit urban entrepreneurs in the same sector Conversely, certain interventions aimed at enhancing employment opportunities for one group may adversely affect another In extreme cases, initiatives can produce perverse outcomes, such as a tourism promotion effort that misinterprets local demand, ultimately harming the existing tourism sector without generating new markets.
Levels of evaluation: policies, themes, programmes, priorities and projects
Linking policies, programs, and specific interventions poses a persistent challenge in evaluation Effective programs do not always equate to sound policies, and well-crafted program documents do not guarantee successful projects Additionally, successful projects do not automatically lead to overall program success Nonetheless, program evaluation serves as a crucial component of policy evaluation, just as project evaluation contributes to program evaluation.
Thematic evaluations and criteria derived from policies when applied to programme material are common ways of introducing a policy level dimension into evaluation.
There is a growing trend for evaluations to focus more on the policy level, indicating that policymakers are increasingly receptive to incorporating evaluation findings This shift poses challenges for evaluators, as they must contextualize their results within a broader framework Additionally, considering the policy level can enhance program evaluations by establishing results-oriented criteria for assessing program success.
Cohesion Policy programmes are often extensive and intricate, encompassing various policies and intervention areas Effective programme evaluation relies on accessible evidence regarding specific interventions, such as the success of aid to SMEs, the benefits of enhancements in transport corridors, and the impact of human resource development initiatives Consequently, evaluations must focus on both the overarching policy framework and the detailed evidence derived from individual intervention areas.
Project evaluation is primarily the responsibility of project promoters and intermediaries, especially for smaller-scale initiatives However, large-scale projects, such as infrastructure developments, require a different approach due to their complexity and size, resembling program evaluations.
Project managers and promoters are often required to perform self-evaluations, which, despite lacking external credibility, play a vital role in enhancing program effectiveness These self-assessments foster a culture of feedback and learning within projects, ultimately benefiting the entire program.
When conducting evaluations, it is essential to understand the connections between policy, program, priority, project, and thematic evaluations The principles outlined in this guide apply universally to all evaluation types, ensuring a coherent and effective evaluation process.
History and purpose of evaluation
Evaluation became a distinct professional practice in North America during the 1950s and 1960s, focusing on three key areas: assessing educational innovations, connecting evaluation with resource allocation, and evaluating anti-poverty programs These foundational strands established enduring evaluation traditions, incorporating quantitative and experimental studies with control groups for educational testing, as well as cost-benefit analyses and participatory methods that engage intended beneficiaries in the evaluation process.
Underpinning these different traditions are four main groups whose interests sometimes compete with each other in defining evaluation priorities These include:
policy makers, e.g., elected officials and politicians;
professional and specialist interests, e.g., teachers in education or scientists in research;
managers and administrators, e.g., civil servants and managers of public agencies;
citizens and those affected by public action, e.g., the presumed beneficiaries of planned interventions
Different stakeholders perceive evaluation in distinct ways: policymakers view it as a means to ensure accountability and justify policy choices; citizens see it as a tool for democratic accountability that allows them to influence public interventions; managers and administrators focus on the effective delivery and organization of policies and programs; while professionals view evaluation as a chance to enhance the quality of their work and assert the autonomy of their profession.
Evaluation, encompassing systematic social and economic research, has been present in Europe, particularly thriving in Northern Europe and regions with strong ties to the U.S and Canada Since the 1970s, evaluation has gained traction in European countries, each developing unique traditions and focuses In Scandinavia, evaluation aligns with democratic governance principles, while France exhibits a dual approach—centralized and structured at the government level, yet diverse and dynamic regionally Notably, French evaluation practices adapted significantly post-2000 due to budgetary reforms Additionally, the scope and focus of evaluation activities in various countries have evolved in response to shifting governmental policies, as seen in the UK's substantial expansion of evaluation following the 1997 government change.
European Structural Funds have significantly promoted the practice of evaluation across the EU, with clearly defined goals and responsibilities at each stage of the programming cycle, including ex-ante, interim, and ex-post evaluations The requirement for evaluation has notably influenced Southern European countries and those that joined the EU in 2004 and 2007 Since its inception in 1988, the approach to evaluating Structural Funds has evolved into a comprehensive framework.
a legal obligation for programme sponsors and managers to evaluate;
shared responsibility between different tiers of government for the overall evaluation process;
a linked multi-stage evaluation process (ex-ante, interim, ex-post);
the involvement of many partners in programmes and in their evaluation;
clear links between evaluation on the one hand and programming and resource allocation on the other
Some of the main transitions have been:
from externally imposed evaluation obligations to internally driven demand for evaluation coming from programme managers and policy makers themselves;
from evaluation that is bolted on to programmes at the end of a programme cycle to evaluation that is fully integrated into programmes from the beginning;
from the expectation that evaluation results need to be disseminated largely for accountability purposes to a concern for the systematic use of evaluation throughout the implementation of a programme;
from a view that the management of evaluation was essentially a matter of contract administration to an interest in the way evaluation can contribute to knowledge management
Based on the experience of the 2000-2006 period, the major innovations of the 2007-
The 2013 evaluation provisions introduced the principle of proportionality and promoted a needs-based approach to ongoing evaluations for Member States and regions This shift reinforces the use of evaluations as a management tool, aimed at enhancing program design and implementation while emphasizing accountability and a strong focus on achieving results.
During the 2014-2020 period, a comprehensive revision of evaluation articles was implemented, emphasizing results and intervention logic Impact evaluation has become a key component, requiring managing authorities to conduct assessments aligned with an evaluation plan This shift redefines impact as the direct contribution of policy to change, rather than merely a long-term statistical evolution potentially influenced by policy.
The Evolution of Structural Fund Regulations:
Ex ante assessment required to be undertaken by the
Member State for each programme
Commission may undertake its own assessment.
An ex-ante evaluation must be carried out in partnership by the Commission and the Member State; it must include environmental impact
Member States hold primary responsibility for conducting ex-ante evaluations, which are essential for assessing the potential impacts of policies These evaluations must prioritize considerations related to environmental sustainability, labor market effects, and gender equality.
As for 2007-2013 with a stronger focus on the results orientation and the intervention logic.
Mid term assessment required for programmes with a duration longer than 3 years, to be carried out by an independent assessor.
Include are a critical analysis of monitoring data and measurement of the extent to which objectives are being achieved.
The managing authority is responsible for the mid-term evaluation in partnership with the Commission; the
Commission assesses the evaluation's relevance The evaluation is carried out by an independent evaluator by end 2003.
An update of the mid-term evaluation is carried out by the end of 2005 to prepare the ground for the future (also known as the final evaluation).
Member States are tasked with continuous evaluation in collaboration with the Commission, shifting from a compliance-focused approach to a needs-based strategy It is recommended that they develop evaluation plans to effectively guide this process, ensuring that evaluations are conducted based on their specific informational needs and timelines.
An evaluation is required for each priority during the programming period to assess its contribution to its objectives.
A report is required by end 2021 for each programme, summarising the findings of evaluations carried out during the programming period.
Supposed to be carried out at national level, but not done in many cases
An ex-post evaluation must be carried out in partnership by the Commission and the Member State, to assess the impact of measures in terms of intended objectives.
The Commission is primarily responsible for conducting ex-post evaluations in collaboration with the Member State This evaluation aims to assess the outcomes of the program and is performed by an independent evaluator within three years following the conclusion of the programming period.
No change compared to the 2000-2006 programming period, except that it is to be completed a year earlier – 2015, the same time as spending is to finish.
No change, although the availability of the report by programme summarising evaluations will provide new material for the evaluation.
Choosing methods and techniques
When designing evaluations, it is crucial to consult stakeholders, map out intervention logics, identify evaluation questions, select criteria, and assess the evaluability of the proposed actions Only after these foundational steps can methods and techniques be appropriately chosen, as illustrated in the accompanying diagram that contextualizes this decision-making process.
Box Choosing Methods in a Wider Context:
The choice of evaluation design directly influences the methods used, as they must effectively address specific questions Despite this, a common issue in evaluation practice is the preference for certain methods by evaluators and commissioners, regardless of their suitability for the questions at hand Often, those commissioning evaluations do not allow evaluators the freedom to select their preferred methods or to conduct an inception report that refines the design and data collection techniques However, this chapter assumes that evaluators will have the opportunity to choose appropriate methods and that commissioners will be guided by a clear understanding of which methods are best suited for their evaluation purposes.
3) Obtaining and using data and evidence
Evaluators hold diverse perspectives on the nature of evidence, with positivists favoring objective observations and constructivists emphasizing the influence of perceptions and theory Adopting a pragmatic approach, we recognize the value of a realist perspective that prioritizes empirical investigation while considering the contexts and theories surrounding phenomena Additionally, we acknowledge the significance of constructivist thinking in prioritizing the experiences and judgments of program participants, while still appreciating the systematic inquiry and careful interpretation advocated by positivist research.
Scientists differentiate between data, the raw material of research, and information, which requires processing to become useful Evidence further refines this information into a reliable form that is credible enough for stakeholders like policymakers and program managers to consider seriously.
Evaluators utilize a variety of data sources, including pre-existing administrative records such as local public employment bureau data and tax returns Additionally, programs generate data through monitoring activities, making the quality of these systems essential for effective evaluation Furthermore, evaluators may need to create their own data by adapting monitoring systems, conducting interviews with local SME managers, or analyzing advertisements in trade magazines.
Effective implementation methods focus on detailing processes and interim outcomes to offer valuable feedback to program managers Drawing from organizational and policy studies, these techniques often compare the performance of various administrative units, such as different regions or municipalities, to assess progress Case studies exploring organizational and partnership arrangements reveal the strengths and weaknesses of diverse implementation strategies Typically, formative evaluation methods are employed, emphasizing the evaluator's responsibility to deliver actionable feedback that enables program managers to convert emerging evidence into practical solutions.
Knowledge production in socio-economic research closely mirrors academic methodologies, emphasizing rigor, representativeness, and careful interpretation of findings, particularly when inconsistencies arise Evaluators primarily seek to answer the question, "What works?" which often leads to the use of experimental methods However, the diverse and context-specific nature of socio-economic interventions complicates the application of traditional experiments Therefore, integrating realist thinking is essential, as it considers how context influences outcomes and prompts a more nuanced inquiry into "What works, for whom, how, and in what circumstances?" Effective methods for this research typically involve comparative analysis of different cases to highlight various interventions and contexts, utilizing case studies, structured databases, and other techniques to effectively capture the complexities of socio-economic development.
In the evaluation community, it is recognized that dependable insights rarely stem from a single assessment Consequently, there is an increasing focus on synthesis studies and meta-analyses that consolidate findings from multiple evaluations The robustness of these analyses is enhanced when evaluations are designed to include standardized structures and data items, facilitating their inclusion in future meta-analyses.
Evaluations are increasingly recognized as a collective responsibility, extending beyond the needs of program managers and sponsors to include a diverse range of stakeholders The success of program delivery relies heavily on the capacities of these stakeholders' institutions and broader civil society networks Therefore, employing participatory methods is crucial, fostering close collaboration between evaluators and involved organizations Such approaches are vital not only for formulating evaluation questions but also for data generation and result utilization In community settings, where diverse interests may complicate consensus, evaluators must engage with community representatives to ensure that evaluation outcomes are effectively implemented.
Choices for different programme/policy stages
The significance of the time-cycle in programs and policies is a central theme in this GUIDE, particularly emphasized in European Cohesion Policy through ex-ante, interim, and ex-post evaluations Beyond these specific terms, the evaluation process must address various stages of a program, including policy formulation, program design, implementation, delivery, and ultimately assessing results and impact, which presents distinct evaluation demands for major programs.
At the design stage, there will be an emphasis on identifying appropriate interventions and the organisation management arrangements able to deliver them;
At the implementation stage, there will be an emphasis on feedback processes, first outputs and results and providing feedback in a way that supports learning;
In the conclusions or results stage, the focus shifts to the outcomes and impacts on intended beneficiaries or regions, alongside any unintended consequences related to the set objectives For the 2014-2020 period, this stage begins prior to the completion of programs, as authorities must evaluate how interventions are meeting their goals during the programming phase at a time they deem suitable Additionally, the European Commission typically mandates that the ex post evaluation be finalized concurrently with the cessation of funding.
Effective policy and program formulation involves identifying current conditions and desired goals, but often lacks clarity in the connections between them during the design phase Developing program theories or logic models for socio-economic initiatives can help outline the implementation pathways from baseline situations to long-term changes Additionally, incorporating techniques like evaluability assessments engages program managers and policymakers in evaluating feasible deliverables, enhancing the overall planning process.
Recognizing the role of various stakeholders is crucial in socio-economic development programs Combining program design with participatory methods can enhance evaluation processes Actively engaging stakeholders in developing their own program theories, rather than solely depending on policymakers, is an effective strategy Approaches like the theory of change are specifically designed to be participatory, allowing stakeholders to express their understandings and implicit theories as active participants rather than passive recipients of program inputs.
Evaluation techniques should be implemented both at the programmatic level and for individual interventions Project appraisal methods, such as cost-benefit analysis, can guide decision-making among various interventions aimed at achieving the same goals Additionally, assessing the trade-offs between different measures and interventions can provide valuable insights for optimizing outcomes.
Evaluating interventions at this stage is essential for clarifying the underlying logic and assessing whether the resources allocated can effectively produce specific outputs through designated policy actions, ultimately contributing to desired results.
Obtaining and using data and evidence
Evaluators hold diverse perspectives on what constitutes valid evidence, influenced by their philosophical orientations Positivists prioritize objective observations and empirical data, while constructivists emphasize the role of perceptions and theories in shaping observations Adopting a pragmatic approach, we acknowledge a realist framework that values empirical investigation and measurement, while also considering the contextual factors and theoretical frameworks that explain phenomena Additionally, we recognize the significance of constructivist perspectives, particularly when prioritizing the experiences and judgments of program participants Our approach also respects the lessons learned from positivist research, highlighting the importance of systematic inquiry and careful interpretation of evidence.
Scientists often differentiate between data, the raw material for investigations, and information, which requires processing to become useful Evidence further refines this information into a reliable form that is credible enough for users like policymakers and program managers to take seriously.
Evaluators utilize a variety of data sources for effective assessments, including pre-existing administrative data such as local public employment records and tax returns Additionally, programs generate data through their monitoring activities, which are essential for successful evaluations and rely heavily on the quality of monitoring systems managed by program managers Furthermore, evaluators may need to create their own data by adapting monitoring systems, conducting interviews with local SME managers, or analyzing advertisements in trade magazines.
Implementation methods typically focus on describing processes and interim outcomes to provide valuable feedback to program implementers These approaches, often rooted in organizational and policy studies, may include comparisons of performance across various administrative units, such as regions or municipalities Case studies examining organizational and partnership arrangements can reveal the strengths and weaknesses of diverse implementation strategies Frequently, these methods employ formative evaluation techniques, which emphasize the evaluator's responsibility to deliver actionable feedback that assists program managers in translating emerging evidence into effective practices.
Knowledge production in socio-economic interventions aligns closely with academic research methods, emphasizing rigor, representativeness, and careful interpretation of findings, particularly when results are inconsistent Evaluators typically seek to answer "what works?" which often necessitates experimental methods However, the complexity and variability of socio-economic interventions across different contexts pose challenges for traditional experiments Therefore, integrating realist thinking is essential, as it focuses on understanding outcomes in relation to context This approach prompts a more nuanced inquiry: "what works, for whom, how, and in what circumstances?" Suitable methods for this exploration include comparative analyses of various cases, utilizing case studies, structured databases, and other techniques that effectively capture the multifaceted nature of socio-economic development.
In the evaluation community, it is generally acknowledged that dependable knowledge typically emerges from multiple evaluations rather than a single one Consequently, there is an increasing focus on synthesis studies and various forms of meta-analysis aimed at consolidating knowledge from diverse evaluations The robustness of these analyses is enhanced when evaluations are designed with standardized structures and data items, facilitating their inclusion in future meta-analyses.
Evaluations are increasingly recognized as essential not only for programme managers and sponsors but also for a diverse range of stakeholders, emphasizing the importance of ownership among all involved The successful implementation of programs relies heavily on the capabilities of the institutions and organizations representing these stakeholders, as well as the broader civil society networks In many cases, participatory methods are most effective, fostering close collaboration between evaluators and the involved institutions and networks These approaches are crucial not only for formulating evaluation questions but also for data generation and result utilization For instance, in community settings with varying interests and perspectives, evaluators may need to collaborate with community representatives to build consensus, ensuring that evaluation results are effectively applied.
Choices for different programme/policy stages
The time-cycle is crucial in the evaluation of programmes and policies, particularly within European Cohesion Policy, which includes ex-ante, interim, and ex-post evaluations This structured approach emphasizes the need for thorough evaluation at each stage of a programme, from policy formulation and design to implementation, delivery, and assessment of results and impact.
At the design stage, there will be an emphasis on identifying appropriate interventions and the organisation management arrangements able to deliver them;
At the implementation stage, there will be an emphasis on feedback processes, first outputs and results and providing feedback in a way that supports learning;
In the conclusions or results stage, the focus is on assessing the outcomes and impacts for the intended beneficiaries or regions, considering both intended objectives and unintended consequences For the 2014-2020 programming period, this stage begins early, as program authorities are required to evaluate the effectiveness of various interventions in relation to their objectives before the programs conclude The European Commission mandates that the ex post evaluation be completed simultaneously with the cessation of spending.
Design: Interventions and organisation
Policy and program formulation involves identifying current conditions and desired outcomes, yet the connections between these elements often remain unclear during the design phase Developing program theories or logic models for socio-economic initiatives can effectively illustrate the implementation pathways that lead from baseline conditions to long-term changes Additionally, incorporating techniques like evaluability assessment allows for a more comprehensive approach, engaging program managers and policymakers in evaluating feasible delivery options.
Recognizing the vital role of various stakeholders in socio-economic development programs is essential To enhance program design and subsequent evaluations, it's beneficial to incorporate participative methods that engage these stakeholders actively Involving groups in creating their own program theory, rather than depending solely on policymakers, fosters a more inclusive approach Participatory methods like the theory of change are specifically designed to capture the insights and implicit theories of stakeholders as active participants, rather than passive recipients of program inputs.
Evaluation techniques should be implemented both at the programmatic level and for individual interventions Utilizing project appraisal methods, such as cost-benefit analysis, can guide decision-making when selecting between various interventions aimed at achieving similar goals Additionally, assessing the trade-offs among different measures and interventions can provide valuable insights for optimizing outcomes.
Evaluating interventions at this stage is essential, as it clarifies the underlying logic and assesses the feasibility of resource allocation in achieving specific outputs This process is vital for understanding how particular policy actions contribute to desired outcomes.
At this stage, it is essential to conduct synthesis studies of prior implementation mechanisms, focusing on suitable organizational and administrative arrangements Understanding the most effective decision-making processes and partnership structures is crucial Comparative case studies and literature reviews of existing evaluations are likely the best methods to address these questions comprehensively.
Implementation: Feedback and intermediate outcomes
Effective program implementation requires ongoing feedback for managers to pinpoint issues and initiate corrective measures Monitoring systems play a crucial role in delivering this feedback, while also highlighting specific areas that may need further examination For instance, if certain projects experience slow start-ups and resulting budget under-spending, or if key stakeholder support diminishes, these situations warrant thorough evaluation to address potential challenges.
When planning an innovative or experimental intervention, it is essential to track the implementation process closely Formative evaluation techniques, such as participant observation, should be complemented by systematic feedback While feedback is valuable at this stage, it can also feel intimidating, necessitating strong communication and consultation skills to handle it constructively Additionally, program managers may need to engage in self-evaluations as part of the overall evaluation process.
Monitoring systems are essential for tracking intermediate outcomes in programs When logical frameworks and program theories are well-developed during the design phase, a clear template can outline the expected milestones at various stages of the program.
Conclusions: Results and impacts
Policymakers and key stakeholders utilize evaluation to obtain information on outcomes and impact estimates at the conclusion of a program cycle, ensuring accountability and addressing their own needs Evaluation methods compare actual achievements against intended goals and assess endpoints against baselines, while also evaluating the policy's contribution to the results A variety of techniques can be employed in this process.
Econometric and statistical models are essential tools for illustrating variations in economic performance relative to forecasted outcomes By analyzing trends within a development context and contrasting them with other environments, these models, initially established at the onset of a development cycle, provide valuable insights into economic dynamics.
Literature review of similar policies,
Participatory methods including workshops and focus groups,
Indicators based on contextual data or administrative data provided by public authorities
Acquiring and using data in evaluation
All data are "produced"
Evaluators rely on data, which serves as the essential raw material for their analysis Once collected, this data is organized, described, grouped, counted, and manipulated using various methods and techniques It is important to distinguish between primary data, which is generated directly from a program or intervention, and secondary data, which is created for different purposes and exists prior to the program Examples of secondary data sources may include existing research studies, government reports, or historical records.
Statistical sources such as national and regional statistics and EUROSTAT 3 ,
Annual reports of development authorities or federations of enterprises, and
Administrative records of public employment agencies, taxation returns, qualifications and training data
3 http://epp.eurostat.ec.europa.eu/portal/page/portal/eurostat/home/
Before utilizing secondary data, evaluators must understand the methodology behind its collection Key considerations include the sample selection, the definition of outcomes, the timescale involved, and the unit of analysis By addressing these critical questions, evaluators can effectively assess the data's relevance and applicability for their specific evaluations.
It is easier for an evaluator to understand the provenance of primary data These can include:
Monitoring data produced by a programme as part of its reporting obligations to funding authorities,
Usage data generated by the use or uptake of services, funds or facilities provided by a programme, and
Data collected from development sites and intended beneficiaries by evaluators through surveys of beneficiaries, counts of those using a consultancy fund, focus groups and stakeholder consultations
Data collection is not a straightforward process; it requires specific protocols and techniques that determine their applications It raises questions about whether usage data can distinguish between various user types and if the monitoring is limited to financial information Additionally, the representativeness of beneficiary samples is a crucial consideration.
Understanding the decisions made during data collection is crucial for evaluators, particularly when dealing with secondary data This task can be challenging, as evaluators must grasp the context of these decisions In contrast, when it comes to primary data generated by or closely associated with a program, the evaluation team typically has a clearer understanding of the decision-making process It's important to differentiate between data produced directly by the evaluation and that generated by the program itself Even when data is collected by the evaluation team, its strengths, limitations, scope, and relevance must be carefully considered to inform future analyses and support the intended arguments effectively.
Careful consideration of evaluation during the design stage is crucial, as it directly impacts the effectiveness of data collection and the selection of appropriate analysis methods.
Accessing data as a planned activity
The management of data across multiple interventions and partners presents significant challenges, as relevant information is often dispersed across various locations Mapping the availability of this data is a complex undertaking, compounded by the need to negotiate the terms for access, particularly concerning administrative data, which is frequently subject to strict confidentiality and data protection regulations In many cases, such data can only be shared after removing identifying information like names and postcodes Furthermore, administrative bodies are typically protective of their data sources, making the negotiation process for data access time-consuming and requiring careful planning.
Accessing data sources can yield valuable insights for evaluations, as the willingness of partners to share information serves as an indicator of the partnership's coherence and strength Conversely, a consistent refusal to share data may highlight a lack of trust within the partnership.
When evaluators participate in data generation, particularly with primary data sources, access challenges must be carefully addressed These access issues can manifest in various ways, impacting the overall evaluation process.
To ensure high response rates in sample surveys, it is crucial to address the frequent issue of low participation, which can undermine the validity of conclusions and judgments Implementing various strategies can significantly enhance response rates, thereby strengthening the evidence base for evaluations.
Communicating (perhaps in the local or trade press) clearly what is the purpose of an evaluation in general and of surveys in particular;
Designing survey instruments in simple non technical language and keeping the surveys short;
Devoting time to follow-up activities reminder letters, phone calls and re-issue of survey instruments after an elapse time to known non-respondents
Accessing disadvantaged or hard-to-reach groups is essential for effective evaluations, as these populations may harbor distrust towards official actions, which can affect their participation To address these challenges, strategies must be implemented to build trust and facilitate engagement with these communities.
Making links with community gatekeepers so that they can act as local advocates of the evaluation
Producing instruments in different languages (when minority languages are used) or in multiple formats - Braille or audio tapes for those with disabilities
Employing local people from these groups to collect information, run focus groups and explain the evaluation within their own networks
Beneficiaries and stakeholders often hesitate to cooperate in evaluations due to a perceived lack of benefits for their involvement This reluctance can be mitigated by clearly communicating the advantages and value they will gain from participating in the evaluation process.
There is adequate involvement of beneficiaries and stakeholders in the design stages of the overall evaluation and in designing and piloting particular instruments
All participants will receive feedback, which may include a publicly available report, a feedback letter with an executive summary, or an invitation to a feedback meeting after the evaluation is completed.
Data quality is crucial for effective evaluation, heavily influenced by the cooperation of gatekeepers, stakeholders, and beneficiaries Therefore, prioritizing data quality and strategic planning is essential when collecting data and selecting appropriate methods.
Quantitative and Qualitative data
Utilizing a comprehensive array of evaluation methods, both quantitative and qualitative, is crucial for effective analysis This article outlines key characteristics that differentiate qualitative data from quantitative data, emphasizing their unique contributions to a robust evaluation process.
It is crucial to differentiate between raw data and analyzed data, as virtually all data must be produced and processed to be useful Despite employing various analytical methods, quantitative data will still exhibit distinct characteristics and strengths.
What is called quantitative data can take very different forms, for example:
Categoric or nominal data serves as a method for categorizing information without assigning numeric values Instead, numbers are utilized merely as labels to differentiate various categories For instance, Group 1, 2, 3, and 4 can represent distinct sets of SMEs classified by their respective operating sectors.
Ordinal data provides a stronger means of quantification by indicating that certain items are ranked as more or less significant than others For instance, in a business context, some companies may be experiencing growth while others face decline This information can serve as valuable data, even in the absence of precise measurements of the differences in performance among these firms.
A more robust form of quantification involves placing relative differences on a known scale with defined intervals, as seen in various scoring and ranking systems or questionnaires For instance, an expert may assess a project's environmental impact on a scale from -3 to +3, while a questionnaire respondent might indicate their satisfaction on a scale from 1 to 5 Although these quantification methods are stronger than earlier approaches, they still possess limitations regarding their numerical and calculative capabilities.
Ratio data represents the highest level of quantification, characterized by a defined zero point on a scale, allowing for independent measurement Common examples include monetary values, population age profiles, export flows, and productivity indices derived from annual output records While ratio data is often synonymous with quantitative data, its usage in evaluations is less frequent than one might think.
In socio-economic development, much of what is labeled as quantitative data is often weak, as it relies on categoric, ordinal, and interval data to assess reductions in social exclusion, improvements in human resource quality, and diversification of rural economies While some data, such as that concerning local firm competitiveness or labor market participation, can undergo more rigorous analysis through ratio data, these instances are less common than is frequently assumed.
Quantitative and qualitative data exist on a continuum, ranging from the most quantitative to the most qualitative forms Categorical and ordinal data are generally more qualitative in nature Pure qualitative data is diverse and often consists of unique instances or reports that require case-by-case descriptions For example, a case study or life history compiled without prior categorization aligns with this qualitative ideal However, once multiple instances or biographies are gathered, they can be categorized similarly to quantitative data.
Qualitative data can be analyzed by comparing and categorizing related phenomena, such as individuals, firms, or households Ranking these data along a scale is effective in these cases However, when examining unique examples like a specific rural community or manufacturing sector, comparisons are often made over time or against external standards or criteria.
The distinction between quantitative and qualitative data often blurs, particularly in terms of analytic intent While individual opinions may be unique and qualitative, the underlying raw data remains constant The processing of this data for analysis is crucial, as quantitative data is typically used for aggregation and generalization, whereas qualitative data is essential for describing complexity and specific needs Ultimately, the choice between these strategies should align with the evaluation's objectives, and most evaluations will benefit from incorporating both types of data.
Creating indicators and indicator systems
What are indicators?
Definition and characteristics of an indicator
An indicator is a measurable element that reflects an objective, mobilized resources, achieved effects, quality levels, or contextual variables It generates quantified information to assist stakeholders in public interventions with communication, negotiation, and decision-making In evaluation contexts, the most critical indicators are those associated with the success criteria of public initiatives.
In order to be useful it is preferable if an indicator has the following characteristics:
Indicators are defined in relation to specific policy objectives, and they become most effective when these objectives are clearly outlined with measurable targets or milestones that correspond to the indicator's definition.
Regular measurement of the indicator is essential, ideally using consistent definitions over time Access to historical data prior to the intervention's adoption enhances analysis, although new data collection is often necessitated by the interventions themselves.
To ensure the reliability of gathered data, it is essential to implement specific steps For output indicators, data provided directly by projects should undergo sample checks for verification Meanwhile, for result indicators, collecting data independently is recommended to enhance accuracy and trustworthiness.
In practice indicators rarely exhibit all of these characteristics and it is likely to be necessary to gather evidence from a variety of disparate sources including:
The inputs to and timing of the programming process;
Primary sources, including Stakeholder surveys;
Much of this information may have been gathered for purposes other than evaluation.
An indicator serves as a measurable metric that assesses key aspects of a program's effectiveness For instance, it can highlight that "1,200 long-term unemployed individuals received training funded by the program," or reveal that "75% of training course participants report being satisfied or highly satisfied."
An effective indicator should convey clear and straightforward information that is easily understood by both suppliers and users Examples of such indicators include the budget absorption rate, the percentage of regional firms receiving assistance, the total number of jobs supported, and the unemployment rate in the eligible area.
An indicator can exhibit multiple values over time, such as the unemployment rate, which may differ at the beginning of a program compared to its midpoint These fluctuations over time reflect underlying trends.
There are several typologies of indicators:
In relation to variables: Complete, partial and complex indicators
In relation to the processing of information: Elementary, derived and compound indicators
In relation to the comparability of information: Specific and common indicators
In relation to the scope of information: Context and programme indicators
In relation to the phases of completion of the programme: Resource and output indicators
In relation to evaluation criteria: Relevance, efficiency, and effectiveness indicators
The most useful of these typologies for socio-economic programmes is the distinction between resources (inputs); outputs and results indicators
Resource indicators are essential for assessing the financial, human, material, organizational, and regulatory resources utilized in program implementation These indicators reflect the shared responsibility of financing authorities and operators, who work together to allocate and use these resources effectively Regular monitoring systems quantify various resource indicators, such as total budget, annual budget absorption rate, expected over/under spending percentages, the share of European financing in public funding, and the number of personnel and organizations involved in program execution.
Output indicators reflect the results of program activities and encompass all goods and services produced as a result of public spending These outputs are primarily managed by operators who track and report them through a monitoring system Examples of output indicators include the number of kilometers of roads constructed, hectares of urban wasteland rehabilitated, capacity of newly built purification plants, and the number of trainees funded by the program.
Result indicators signify the desired changes that a program aims to achieve The evaluation literature highlights the evolving definitions of "results" and "impacts." For additional insights, refer to the guidance provided by the Directorate Generals for Regional Policy and Employment and Social Affairs.
Impact indicators have been eliminated from Structural Funds programs, with the exception of those related to fisheries and rural development For the Directorate Generals of Regional Policy and Employment and Social Affairs, the significance of impact lies in the intervention's ability to drive change, which cannot be tracked through monitoring and necessitates thorough evaluation.
Indicators and evaluation
Indicators play a crucial role in evaluation by providing measurable data that aligns with clearly defined objectives To draw accurate evaluative conclusions, it is essential to interpret the information they offer in conjunction with other evidence Furthermore, indicators can significantly enhance the assessment of socio-economic programs by offering insights into their effectiveness and impact.
The analysis of the indicator scores can be used to provide support for a rationale for intervention and resource allocation
Indicators can be used to compare inputs and outputs in order to measure efficiency
Indicators can be used to compare actual outcomes with expectations in order to assess how needs have been addressed
Indicators can be used to compare inputs relative to impacts and hence allow the assessment of the value (value added) of policy, legislation or initiatives
The system of indicators and the programme cycle
At the outset of the program cycle, indicators are essential for identifying territories eligible for assistance, analyzing regional contexts, diagnosing economic and social issues, and assessing program needs Key indicators, including the number of start-ups, unemployment rates, and infrastructure disparities, significantly influence these evaluations.
4 See: http://www.dfid.gov.uk/Documents/publications1/design-method-impact-eval.pdfAnd http://evi.sagepub.com/content/16/2/153.abstract
The second stage of the programming cycle involves selecting and validating the intervention strategy, where program designers must clearly define and quantify the objectives Each priority should have a specific objective accompanied by a result indicator that reflects the desired change It is essential to establish known baselines, and the intervention logic should convincingly illustrate how the allocated resources and outputs will achieve the intended results.
Once a program is defined and adopted, its implementation begins, with ongoing monitoring and evaluation Key indicators track budget expenditure, adherence to schedules, reach among the eligible population, and progress towards planned outputs Program managers conduct evaluations to determine how effectively the interventions are meeting specific objectives and achieving desired results.
The programming cycle concludes with an ex post evaluation, which primarily focuses on assessing the outcomes of a program or policy and determining the degree to which its objectives have been met At this stage, the utilization of indicators is crucial for effectively conveying the program's achievements.
Organisational aspects: Involving users and suppliers of information
Involving both suppliers and users in the creation of a system of indicators significantly increases its chances of success Conversely, when a closed group of specialists designs the system, they may focus on developing an expensive, technically perfect solution that ultimately fails to operate effectively.
To ensure user satisfaction, it is crucial to have explicit support from the highest authority overseeing the program Additionally, forming a group of prospective users to define key performance indicators is highly recommended.
Key information providers are the operators executing projects in the field, as their involvement helps create a pragmatic system Their familiarity with the practical possibilities and limitations of data collection enhances the effectiveness of the information gathered.
Selecting indicators
Selection of the most relevant indicators
Each program actor has distinct responsibilities, decision-making areas, and information needs, leading to the conclusion that not all indicators are relevant at every level It is widely recognized that each actor requires a concise operating report featuring a limited selection of the most pertinent indicators tailored to their specific decision-making context Research indicates that individuals can effectively process only about ten indicators simultaneously; an overload of information can hinder decision-making.
Cohesion Policy experience highlights the challenges in selecting essential indicators for effective program monitoring and evaluation Given the multi-sectoral and multi-objective nature of these programs, there is a tendency to attempt measuring everything, leading to the creation of overly complex indicator systems that are impractical to implement Consequently, producing and regularly utilizing such vast amounts of information becomes unfeasible.
During the 2014-2020 period, there is a significant focus on minimizing the number of indicators to streamline resource allocation The goal is for indicators to effectively contribute to policy discussions, as an excess of them can obscure the evaluation of policy achievements.
Avoiding the adverse effects of indicators
The use of indicators is often hindered by the fear of provoking adverse effects There are several types of adverse affect:
The skimming-off or creaming effect,
Unanticipated effects where results are subordinated to indicator scores
Skimming-off effects arise when training and employment services prioritize placement rates, leading organizations to favor beneficiaries with higher employability This practice results in assistance being directed toward individuals who are already in better situations, rather than those who are in greater need of support Such an approach undermines the effectiveness of these services by neglecting the most vulnerable populations.
Indicators can lead to a reduction in disparities by promoting convergence towards the average; however, they may also incentivize undesirable behaviors that result in sub-standard performance This happens when the indicators reward poor outcomes or when operators focus on meeting the indicator criteria instead of achieving the desired results.
Convergence towards the average rather than excellence:
The British Audit Commission's system of indicators quantifies around two hundred output and result indicators for municipal services, allowing for annual comparisons between towns published in local media Poor performance in any service can significantly impact a town's reputation, prompting municipalities to boost budgets for underperforming services, often at the expense of more effective ones Consequently, this reliance on indicators has led to a convergence towards average performance rather than fostering excellence, highlighting an unintended adverse effect of the system.
Adverse effects inevitably appear after two or three years of functioning of a system of indicators, no matter how well it is designed These undesirable effects are generally not foreseeable.
The potential for adverse effects should not deter performance measurement, as these effects can be minimized through adjustments to problematic indicators or by establishing expert panels for interpretation It is crucial to monitor for any adverse effects and to make necessary corrections to the system when they arise.
It's essential to recognize that indicators alone do not provide a complete picture of performance Evaluators must focus on understanding and effectively communicating the reasons and processes behind various phenomena, as indicators cannot convey this information This understanding is fundamental to the evaluation process.
1 Choosing methods and techniques follows directly from the kind of questions one wants to ask and these questions are part of an extensive design exercise that includes consulting stakeholders and assessing programmes characteristics.Choosing methods and techniques first and trying to make them fit with questions for which they have not been specifically chosen will create problems The techniques chosen need to reflect the purpose and focus of the evaluation
2 Most techniques have strengths and weaknesses; these need to be recognised and where possible different techniques need to be applied together to strengthen the analysis and make the evaluation results and conclusions more reliable
3 Because of the distinctive character of socio-economic development: bottom-up, using different combinations of interventions and tailored to territorial and sectoral needs, it is difficult to measure and compare outcomes across programme settings This doesn't mean that measurement, quantification and statistics are not relevant They can be powerful tools when comparisons are at the level of the particular development programme and do not attempt to compare non comparable settings
4 Qualitative methods and techniques are well suited to socio-economic development because of the subtlety and holistic nature of what is being attempted and because of the differences in contexts which need to be described in qualitative ways
5 Thematic priorities which are very common in European programmes pose real difficulties for evaluators Because policy makers want to understand how far their policies are successful as a whole, there is often pressure to aggregate results and find a common way of describing or even measuring what is happening This often cannot be done Sometimes only qualitative descriptions will work Take care not to add up apple and pears
6 There is often a tension between choosing evaluators who know a lot about a particular policy area and those whose evaluation skills are more generic Ideally in an evaluation team one tries to span both of these sets of knowledge and experience Commissioners need to be aware of the dangers of contracting evaluators who have lived most of their professional lives working in one specialised area and using a limited set of methods and techniques This is another argument for looking at the balance of skills in the evaluation team
7 It is important to distinguish between methods and techniques for gathering data, for analysing data and for informing evaluative judgements This distinction is not always made partly because those who undertake evaluations may be more preoccupied with one stage of the process rather than another As in all things there needs to be a balance
8 Data are never pure or naturally occurring, they need to be produced Because of this evaluators need to know from where their data comes and what decisions have made in the course of their production The strength of the arguments and conclusions that can be drawn depend on the strengths and characteristics of the data being used
9 One important aspect in evaluating data follows from the way they have been accessed and how access has been negotiated Different partners have to be willing to share information and all stakeholders need to be convinced that they are going to get something out of an evaluation before they give access with any enthusiasm to any information they hold Investing in these kinds negotiation processes will make a difference to quality and the evaluation as a whole
10 The quantitative/qualitative divide is overstated What is needed is quantitative data to provide overviews, for example to aggregate outputs of an intervention and provide a comparative perspective, and qualitative data able to capture subtleties, people's experience and judgements.