CHAPTER 9
Introduction
Evaluations of public interventions to combat poverty and inequality generally evaluate projects and programs. More rarely, they address national development plans (NDPs). However, most developing countries
Evaluating How National Development Plans Can Contribute to Poverty and
Inequality Reduction: The Cases of Cambodia and Costa Rica
Ana Maria Fernandez, Roberto Garcia-Lopez, Thavrak Tuon, and Frederic P. Martin
Ana Maria Fernandez and Frederic P. Martin are with the Institute for Development in Economics and Administration. Roberto Garcia-Lopez is with the Latin-America and the Caribbean Community of Practices on Managing for Development Results.
Thavrak Tuon is with the Ministry of Planning, Royal Government of Cambodia.
and some intermediate and industrial countries have been organizing all of their economic and social policies within the framework of an NDP.1 This document expresses the desired stage of development for a country, which often includes poverty and inequality reduction. This vision is then converted into strategic objectives with indicators of impact and fi nal outcome as well as targets, which translate into programs and projects that are then implemented. As a road map for the period of a government mandate, an NDP is a planning instrument that helps to align the interventions of all actors (governmental, nongovernmental, and private) toward a common vision for the country’s development in the medium run.
This chapter addresses how to assess the contribution of a national devel- opment plan to a country’s economic and social development, including poverty and inequality reduction, through proper elaboration, budgeting, implementation, and monitoring and evaluation (M&E). This process has proved to be a challenge for at least four reasons.
First, by defi nition, an NDP covers all sectors of a given country, which means that it is huge in scope. These multiple sectors (economic, social, infrastructure) have quite diff erent program objectives, institutional set- ups, and technical systems to deliver public services. The plan deals with a diversity of regions, milieus (urban, rural), and populations. It addresses a variety of cross-cutting issues, which are not easily delineated and mea- sured, given the number of stakeholders and possible angles of analysis. It is supposed to integrate actions from all major stakeholders, including the national government, of course, but also subnational governments, the pri- vate sector, civil society, and development partners. The challenge becomes how to use a participatory approach involving all major stakeholders in elaborating an NDP, but to do so in a few months, to synthesize micro view- points into a macro perspective, and to make the plan a useful document for orienting all policies and programs of a new government. This macro per- spective also makes it even more complicated to deal with the attribution issue during evaluation: in the absence of a counterfactual, how do we know that the NDP made a diff erence?
Second, the time dimension complicates evaluation. A typical NDP has a medium-term perspective (for example, four years in Costa Rica and fi ve years in Cambodia). However, it needs to integrate long-term trends (20–30 years ahead), while also addressing shorter medium-term action plans, fi scal and expenditure frameworks, and yearly work plans and budgets.
Even more than is the case for a program or a project, fi nal outcomes and impacts will be felt long after the fi nal evaluation takes place, which is typi- cally at the end of the period covered by the NDP.
Third, actions and programs indicated in the NDP are not necessarily clearly identifi ed with immediate outcomes linking outputs with fi nal out- comes. Thus costing the NDP might face analytical challenges, issues related to the availability and reliability of data, as well as a huge amount of work.
Fourth, the dialogue and priorities between politicians and technocrats often make it diffi cult to harmonize the measure of objectives and goals, especially for poverty and inequality.
This chapter presents two case studies of the process of elaborating, implementing, monitoring, and evaluating an NDP: one in Cambodia2 and one in Costa Rica.3 These countries have diff erent histories and face diff erent situations. Costa Rica is an upper-middle-income country,4 while Cambodia is a low-income country.5 Despite their diff erences, they share some com- mon features, starting with their dependence on a main source of income:
ecotourism in Costa Rica, known as a “green destination,”6 and cultural tourism in Cambodia, known as a “spiritual destination.”7
In addition, both countries have a record of planning. The elaboration of an NDP in Cambodia (adoption of each document by law) and Costa Rica (Law no. 5525) is mandatory. Since 2006,8 Cambodia has named its docu- ment the National Strategic Development Plan (NSDP), which is valid for a period of fi ve years, the current one for 2014–18. The primary responsibility for preparing, monitoring, and evaluating the NSDP is assigned to the Ministry of Planning. In Costa Rica, the document is called the National Development Plan, which is valid for a period of four years; the analysis in this chapter is for the 2011–14 NDP. In 2013 Costa Rica celebrated 50 years of planning. Like in Cambodia, in Costa Rica the Ministry of National Planning and Economic Policy (MIDEPLAN)9 has the leading role in formulating, supervising, monitoring, and evaluating the NDP.
Overall, even if there are some singularities depending on each country’s context, the contents of both NDPs are organized in a similar way: (a) diag- nosis, (b) key policy priorities and actions, (c) costs and resources, (d) charts with indicators and targets, and (e) M&E orientations. Both documents also share some limits. In particular, they (a) do not take a cross-cutting approach among sectors; (b) do not take a regional approach to service delivery; (c) include various planning levels without a hierarchical logic, sometimes mixing outcomes with outputs and not identifying immediate outcomes; (d) have limited target validation processes; and (e) are unable to link programs to the budget on a multiannual basis.
The next two sections of this chapter present case studies of NDP evalu- ations. The fi rst one evaluates the process of NDP elaboration and imple- mentation in Costa Rica, while the second one evaluates the NSDP M&E system in Cambodia. The case studies illustrate complementary approaches
that can be used depending on the evaluation’s objectives, time, and budget limits. They are not models to follow necessarily, since each methodology has to be tailored to the specifi c objectives of the evaluation, time and budget constraints, as well as the characteristics of the country and its NDP.
Following these case studies, the chapter outlines lessons learned for future evaluations of an NDP and its contribution to poverty and inequality reduction.
Case Study of Costa Rica
Evaluation Objectives
The evaluation sought to accomplish the following:
1. Determine the extent to which the NDP achieves the objective of priori- tizing the agencies’ actions toward the national development targets 2. Determine the extent to which the NDP is binding for the public sector 3. Defi ne whether the document could be evaluated according to its content 4. Assess the level of validation for priorities and targets among public and
private actors.
Evaluation Process and Methods
The evaluation took place over a six-month period, applying a combination of qualitative and quantitative methods: document review, interviews and focus groups, direct observation, and a survey. The use of multiple lines of inquiry provided a level of rigor to the evaluation by off ering an opportunity for triangulation. Findings from a particular method that deviated from those obtained by other methods were analyzed, giving special attention to reasons for the deviation.
The methodology was based on a fi ve-step approach, in which the result of each step contributed to the next in order to build simultaneously on the collection of data and validation of fi ndings, trying to incorporate the per- spective of multiple actors (fi gure 9.1).
The fi rst step was to conduct a desk review to obtain background information and analyze secondary data.
The second was to conduct 33 in-depth interviews using a semistruc- tured questionnaire with civil servants from MIDEPLAN, the Ministry of Finance, priority line ministries, the presidency, the National Accounting Offi ce, Congress, and civil society representatives.
After these interviews were processed, the third step was to conduct a civil servant survey, using a semi-open questionnaire (including 25 ques- tions) administered online for seven days. The questions were graded on an ordinal scale so that the respondent could assess whether a given issue was considered to be a major challenge, a challenge, or not a signifi cant challenge.
Respondents to the anonymous online survey were 128 civil servants10 out of a sample of 198 selected in the sampling agencies in which in-depth inter- views had been conducted.
The fourth step was to conduct three focus groups in the three main the- matic areas considered as cross-cutting issues for sustainable development in the country and as domains of unsatisfactory performance in the previous NDP: environment, regions and municipalities, and civil society and private sector initiatives. The focus group participants included a mix of academics, sector experts, and private sector and nongovernmental organization repre- sentatives, along with professional civil servants. The questionnaire for each focus group was designed based on the results of the online survey, so that the focus group participants were able, among others, to prioritize the major
1
4 5
Desk review
2
In-depth interviews
Online survey Thematic
focal groups National workshop
3
Figure 9.1 Five-Step Approach Followed for the NDP Evaluation in Costa Rica
Source: Ministry of Planning 2013.
Note: NDP = national development plan.
challenges identifi ed by the civil servants surveyed and to add issues that the public sector did not consider priorities.
Finally, a workshop was organized with all participants in the evalua- tion to present, discuss, and validate preliminary fi ndings, include issues that were not initially stated, and propose solutions to improve the next round of NDP formulation and implementation. The participants were divided into six groups corresponding to key results areas, and each group provided up to fi ve solutions for each challenge. The results of this work- shop were used as input for the methodology that was recommended for the next NDP.
After the national workshop, evaluators designed a comparison matrix of prioritized challenges grouped by theme according to diff erent sources of information: online survey, focus group, national workshop, and evaluators (table 9.1).
Evaluation Effectiveness and Challenges
One of the main methodological challenges was to consult as many actors as possible in a short amount of time. According to participants, the fi ve-step methodology enabled them to consider the view of multiple actors, to com- pare information from diff erent sources, and to identify major challenges.
They appreciated being included in the discussion of the fi ndings and sharing cross-cutting ideas, saying that this ability is usually limited by insuffi cient coordination and communication between leading agencies and line agencies as well as between public sector actors and actors external to the public system.
Table 9.1 Criteria to Grade Challenges among Data Sources in Evaluation of the NDP in Costa Rica
Grade criteria Implication for the NDP elaboration process Graded as a major challenge by the four sources Requires immediate modifi cation
Graded as a major challenge by two to three sources Continue with minor modifi cations Graded as a major challenge by between zero and
one national source, with the evaluators considering this challenge not to be a major challenge according to international good practices
Continue without modifi cations
Graded as not a major challenge by any national source, but considered a major challenge by evaluators according to international good practices
Indicates a possible area of modifi cation that was not discussed thoroughly during the evaluation process
Source: Ministry of Planning 2013.
Note: NDP = national development plan.
Application of the methodology confronted the following challenges:
1. Absence of a common understanding of the underlying theory of change in public interventions
2. Tendency to operate with a micro perspective and a “silo” mentality, which made it diffi cult to conduct a systemic analysis
3. Perception that the evaluation process was a threat rather than a learning opportunity
4. Organizational inertia and resistance to change
The evaluation team addressed those challenges by (1) providing a brief introduction of concepts at the beginning of all group activities; (2) basing discussions on the responses of various actors and checking the consistency of responses among them; (3) using sensitization and group discussion management techniques and presenting fi ndings to all actors in a neutral, constructive, and action-oriented way; and (4) supporting the dissemina- tion plan for the fi ndings after the evaluation was fi nalized.
Case Study of Cambodia
Evaluation Objectives
The evaluation sought to accomplish the following:
1. Conduct a rapid and participatory assessment of the current national M&E system for the NSDP11
2. Identify and prioritize bottlenecks for improving the M&E of the NSDP implementation and results
3. Identify characteristics of the desired M&E system of the NSDP and propose orientation guidelines for the system
Evaluation Process and Methods
The methodology used for the evaluation followed the assessment for results (A4R) process, supported by a Web-based software.12 The process started with a review of relevant documentation. Then the evaluation team conducted in-depth interviews with 10 ministries and agencies and facili- tated a two-day workshop with 42 key national actors. This workshop enabled the evaluation to (a) take stock of the existing initiatives in M&E that are relevant for NSDP implementation (supply side); (b) identify strengths, weaknesses, and challenges to meeting the information needs for decision making and accountability (demand side); and (c) seek concrete
ways to improve both the demand for and supply of M&E information to monitor and assess NSDP implementation.
The A4R software supported the assessment in various ways:
• By making it easy to upgrade the indicators and subindicators of performance for diagnosing the M&E system; the subindicators were validated, and some were customized to refl ect the specifi cities of Cambodia’s situation
• By keeping a record of the assessment process to compare preassessment (done by an expert based on revised documentation and in-depth inter- views), workshop assessment, and postassessment (after more documen- tation and revision with the Ministry of Planning and the United Nations Children’s Fund, UNICEF)
• By collecting evaluation data online, which enabled the assessment to refl ect immediate results and to facilitate the plenary discussions for the participatory diagnostic, to identify and analyze bottlenecks, and to for- mulate an action plan
A4R considers six technical and institutional dimensions in the assess- ment of an M&E system (table 9.2). The exact number and label of perfor- mance subindicators were adjusted prior to the workshop after fi nalizing the in-depth interviews and in close collaboration with the Ministry of Planning and UNICEF.
During the workshop, the evaluation team introduced the basic concepts to be used for assessing the six dimensions and for guiding the discussion of the challenges encountered in each dimension. Workshop participants assessed each dimension through several performance indicators, which included several subindicators, as presented in table 9.2. The six groups of
Table 9.2 Dimensions of the M&E System and Performance Indicators and Subindicators in A4R for Cambodia
Dimension Name Indicators Subindicators
1 Preconditions for results-based M&E 4 16
2 Institutional framework for M&E 4 9
3 Plan and budget for M&E 4 10
4 Routine monitoring 5 17
5 Evaluation cycle 4 4
6 M&E information systems 3 3
Total 24 59
Source: Ministry of Planning 2015.
Note: A4R = assessment for results; M&E = monitoring and evaluation.
workshop participants gave a grade to each subindicator, using a four-level ordinal scale from 0 to 3. Each grade corresponded to specifi c assessable characteristics, which were indicated in A4R and followed international standards for M&E systems. The participants were asked to provide explicit justifi cation of the grade as well as supporting documents.
The bottleneck analysis was conducted after the diagnostic to identify the main challenges. A nominal group technique was used to identify and prioritize the bottlenecks. Every participant had a chance to present three main bottlenecks. Some of those bottlenecks were regrouped when com- mon areas were identifi ed. Afterward, each participant selected fi ve priority bottlenecks and assigned a grade on a scale from 5 to 1, where 5 is the most important bottleneck. The A4R software facilitated the scoring for ranking the bottlenecks.
Then the evaluation team proposed key characteristics of the M&E system and elements of guidelines for NSDP implementation based on the diagnos- tic and bottleneck analysis, and these were discussed by participants. Finally, participants were divided into three thematic areas based on bottleneck analysis and asked to come up with elements of a medium-term action plan, including (a) institutional arrangements, (b) capacity building, and (c) infor- mation systems.
Following the workshop, consultants revised the assessment of the participants, making sure that the grade given corresponded to the justifi ca- tion provided and, when necessary, complemented the information based on a review of relevant documentation conducted before the mission.
Evaluation Effectiveness and Challenges
The preliminary results were presented at the end of the workshop, and the full results were presented in a report a month later. This process was greatly appreciated, allowing the rapid presentation of results and creating momen- tum among national actors to move forward with the system and to coordi- nate their initiatives based on their roles and responsibilities.
Application of the methodology entailed three challenges: (1) the unequal level of knowledge of M&E concepts among participants, (2) the tendency to assess the M&E system with regard to a particular sector—for example, some ministries were conducting pilot initiatives that were not yet common to the bulk of ministries, and (3) the tendency to interpret some subindicators in the local context. The evaluation team addressed those challenges by, respectively, (1) providing basic knowledge about M&E concepts, (2) correcting some grades after the workshop to refl ect the situation in the whole of government, and (3) clarifying and disaggregating some subindicators of performance.
Lessons Learned from the Two Case Studies
The Cambodia and Costa Rica case studies provide several lessons for future evaluations of the elaboration, implementation, and M&E of an NDP and its contribution to poverty reduction.
Assessing the Global, yet Functional, Positioning of the NDP within the Results-Based Management Cycle
A common limitation of NDPs is a failure to translate a good diagnostic and a desirable vision into a set of public interventions that will most likely achieve the set targets. The strategic planning process will be of very limited value if it is not positioned as part of a results-based management cycle that articulates the planning phase with the programming and budgeting phase and with the M&E and reporting phase, as depicted in fi gure 9.2.
This results-based management cycle involves three levels of decision making: (1) strategic (deciding on priorities and services to deliver), (2) pro- grammatic (organizing service delivery), and (3) operational (delivering services). These levels correspond to the three rows in fi gure 9.2.
Talking about implementation of an NDP is misleading. What is being implemented are measures (changes in the law, new institutional arrange- ments), programs, and projects. Often, the missing link in planning between the strategic (NDP) and the operational (work plan) levels is the sector
Planning phase
Programming and budgeting phase
Monitoring, evaluation, and reporting phase
Monitoring high-level key
performance indicators
Evaluation of programs and projects Programs
and projects monitoring
AWPB monitoring Imple-
mentation of AWPB Elaboration
of annual work plan and budget (AWPB) Multi-year programming
and budgeting Fiscal framework
Program architecture
Strategic planning Diagnostic
(situational analysis) Strategic
level
Programmatic level
Operational level
Programs and projects
(ex ante analysis)
Program and project
planning
Performance measurement framework
Evaluation of strategic plan implementation
Figure 9.2 Positioning the NDP in the Results-Based Management Cycle
Source: IDEA 2014a.
Note: NDP = national development plan.