DRAFT OUTLINE FOR END OF EVALUATION REPORT SLGP EOP Evaluation STRENGTHENING LOCAL GOVERNMENT CAPACITIES FOR PLANNING, BUDGETING AND MANAGING PUBLIC RESOURCES END OF PROJECT EVALUATION FINAL REPORT 30[.]
SLGP EOP Evaluation STRENGTHENING LOCAL GOVERNMENT CAPACITIES FOR PLANNING, BUDGETING AND MANAGING PUBLIC RESOURCES END OF PROJECT EVALUATION FINAL REPORT 30 November 2010 hrg1650389134.doc SLGP EOP Evaluation DRAFT REPORT EXECUTIVE SUMMARY Acknowledgement Table of Contents Abbreviations & Acronyms Signature Page A INTRODUCTION AND METHODOLOGY A1 Background A2 Project Context A3 Objectives and Products of the EOP Evaluation A4 Evaluation Scope A5 Evaluation Approach and Methodology A6 Structure of the Report B CONTEXT, EVIDENCE AND ANALYSIS OF THE PROJECT PERFORMANCE Project Achievements 1.1 Outcome 1: Improving local capacities in pilot provinces 1.1.1 Output 1.1.2 Output 1.1.3 Output 1.1.4 Output Conclusions on Outcome 1.2 Outcome 2: Contribution to national policies 1.2.1 Output Conclusions on Outcome 2 Relevance 2.1 Project design issues, and relevance within national development context 2.2 Project design issues, and relevance vis-à-vis One UN Plan Effectiveness of management strategy and implementation arrangements Efficiency and utilization of project resources Gender issues Partnership Preliminary impact Sustainability C FINDINGS D LESSONS LEARNED E CONCLUSIONS and RECOMMENDATIONS Annexes Salient Events in the Life of SLGP Questionnaire with Pilot Units: Summary Notes Outputs: Quick Survey on Degree of Coverage Across Pilot Units Results Framework: Summary Assessment by EOPE Evaluators Impacts: Quick Survey on Importance, Depth and Sustained Effects Persons Consulted Management’s Response to MTE Legal Framework Comparison of Findings, MTE and EOPE 10 References 11 Terms of Reference hrg1650389134.doc SLGP EOP Evaluation EXECUTIVE SUMMARY Introduction This end-of-project evaluation (EOPE November 2010) addresses the issues of achievements and results; relevance; effectiveness; efficiency; impacts; sustainability; and provides lessons learned and recommendations EOPE has been facilitated by the wellorganised cooperation of staff and leaders of the 36 subnational pilot units and local organisations, the four PPMUs, the CPMU, UNDP and staff of ten Donor agencies The final progress reports of CPMU and the four pilot Provinces (September 2010) have been especially helpful, and are strongly recommended as parallel reading to the EOPE Evaluation of SLGP must be viewed in the macro context of People-State relationships, as central authorities learn how to organise the evolution of subnational political participation, financial management, resource and investment management, public services design and delivery, and more localised accountability The agenda is usually written in terms of systems and processes; but the determining dynamics are issues of control, authority, legitimacy, transparency and accountability; and by human interests in ideology, life’s values and purposes, power, status, prestige, hierarchy, dependency, liberty, duty, conformity and individuality EOPE can only be concerned with evidence of changes and constraints, and has not the resources or mandate to appreciate the internal dynamics of policy decisions in decentralisation, people’s empowerment, distribution of services, wealth and opportunities The rapid extension of market transactions, the growth of private entrepreneurs and the reduction of State enterprises and State employment bring changes to people’s lives and interests, which displace some of the assumptions and strategies of community-based state management and policy-making on which SLGP was designed and implemented The Project was formulated in 2003-5, a time of great interest in decentralisation and people’s empowerment Commonly mentioned ‘shortcomings’ at that time were: a) a rigid ‘mindset’ including (top-down direction; local dependency and subsidy; lack of communities’ participation and responsibility, a focus on quantities and norms, lack of credible data, administrative targets rather than real needs and potentials;) b) methodological issues (such as lack of strategic approach; lack of relevant legal documents, regulations, methods and tools; lack of understanding of how to fit national strategies and policies into local planning; uncertainty about the allocation of investment capital; lack of connection of planning with budget management;) and c) organisation and resources constraints (lack of local capability to improve capacities autonomously; lack of budget for doing the planning (and other devolved duties); problems of coordination and integration vertically and horizontally; and challenges in dealing with new social and cultural needs and demands emerging from private enterprise, diversification, mobility, information and individualisation of choices and actions.) SLGP was intended to directly contribute to: 1) local capacities in pilot provinces to achieve the strategic goals of pro-poor and gender sensitive plans and budgets, and improved services especially for the poor and marginalized people; and 2) national policy, by ensuring that the lessons from pilot units inform and influence improvement of national guidelines, as well as other aspects of GoVN’s overall policy on decentralization, and other donor projects Findings Overview: The opinions about SLGP among pilot partners have converged around a common ‘story’ - that SLGP came on a wave of interest in decentralisation and participation, that it has provided many relevant and good quality guidelines and learning activities, that hrg1650389134.doc SLGP EOP Evaluation the grassroots initiatives have been well adopted but with less progress on organisation-level reform, that capacity building has been partially successful but needs more extension and reinforcement, that MPI significantly shifted its interest after 2008, and that the Project is closing ‘too early’ in terms of institutionalising of practices at subnational units, and of changing laws and guidelines at national level Achievements: SLGP has contributed greatly to the needs, approaches, priorities, methods, reception or deferment of reforms in planning, budgeting, financing, managing and monitoring - not so much with new technical knowledge, but to the understanding of what is needed, of directions and of how to advance reforms in particular situations The greater part of the achievement is in the designs, preparations, trialling, training, partial adoption and some limited propagation of reforms; but little has been achieved in the qualitative changes of performance of PBS that would serve the SLGP’s strategic Outcomes (in 5.1, 5.2 above) It seems probable that some kind of grant facility would have strengthened and deepened the achievements, by ensuring the applications of SLGP-promoted methods through the whole project cycle, across more cooperating agencies and amongst more participating peoples and groups Relevance: SLGP design was highly relevant in 2005, in addressing LG capacity needs, and in a broad, unspecified strategy of “changing minds and breaking fences “, which was a response to growing local expectations and to the protections and disciplines of several enabling laws and decisions by GOVN This phase may be termed the First Tier devolution, with shifts of national powers and controls There was a general expectation of flow-on devolution, from Second Tier (Province), and so on to Third (District), Fourth (Commune), Fifth (Villages), Sixth (People’s Organisations) and Seventh (Households and individuals) The judgment on relevance in 2010 depends on the degree of support for and direction of further decentralization especially at Tiers and 3, wherein the leaders and staff believe that the proven design cannot advance further without legal directives from national level EOPE concludes that the narrow piloting to two districts and six communes means that there is not the critical mass of well-informed advocates to bring about changes at Provincial level agencies and committees Relevance now requires a broadening (more buy-in) and deepening (more learning by doing) to shift province-wide systems towards internalised reforms and more strategic outcomes Effectiveness: The demand-based capacity development, and the process approach with much attention to the feasibility of changes, proved to be an effective pilot Welldesigned learning-by-doing process, with an adequate technical backup, proved to be a good approach in capacity development interventions at local level Delays in preparing and rolling out several documents and activities resulted in less effective propagation, learning and applications Partial delegation of Project implementation to PPMUs (2008) was eventually regarded as an effective mechanism for piloting towards Outcome 1; but some partners still regard that as an indication of loss of momentum for specific reforms from the national centre, and with reduced efficacy of feedback from pilots This perceived ‘loss of momentum’ could have been offset by a more directive and demanding role of CPMU for the results-based evaluation of innovations, and for the upward feedback and use of pilot experience among national partners There is not enough formal evidence of National Steering Committee deliberations, guidance and interventions by which EOPE could make a fair assessment 10 Efficiency: National level CPMU-MPI and sub-national levels (PPMUs and localities) have demonstrated their commitment and diligence in preparing and refining the approaches, methods, documents, events and guidelines to enable good quality piloting and efficient use of resources of the Project and of partners Local leaders and staff of units consistently praise the efforts and professionalism of PPMUs and the specialist teams which they coordinated This reported competence indicates that a grant facility could have been hrg1650389134.doc SLGP EOP Evaluation well managed and used to realise and demonstrate the gains of new methods and more localised participation and responsibility 11 Sustainability: SLGP supported around fifty products, guidelines and training sets, to around 1,000 staff and leaders, with many flow-on applications and some instructions and letters confirming official adoption The LCAs and EOPE informants report the widely recognised value of certain changes, and the commitment to continue However, there is very much uncertainty about sustainability of reforms and their benefits, mainly because: a) more than half of the trials and innovations have not been adequately institutionalised nor their performance deeply reinforced (see LCA 3, Tables V.1, V.2, V.3; and the Final Progress Reports from four Provinces); b) there is some hesitancy at Second and Third Tier about pushing further on the decentralisation reforms which are permitted under existing laws; c) there are still many gaps before reaching the level of consistent performance by which political leaders would be convinced in taking the risks and time-effort costs in more reforms and more decentralised functions, such as in finance mobilisation or monitoring and oversighting 12 Changes: The most influential of SLGP changes has been in the ‘mindset’ and attitudes of leaders, staff, cadres and citizens about the need for and possibilities in reform and decentralisation However, most local partners believe that the mindset changes and people’s interest and involvement in PBS can soon be reversed, unless the legal requirements to reform PBS are promulgated and enforced across all interdependent units 13 Capacity: SLGP has noticeably enhanced local capacities in pilot units, and among a few neighbouring units, with generally good targeting of interventions, careful approaches, good quality materials and appropriate methods Individuals have benefited most; while organisational capacity is constrained by work cultures and the uncertainties in pushing reforms of subnational functions and powers in the devolving hierarchies The awareness of capacity building effects has been strengthened by the conduct and findings of three LCAs – which have also revealed the need for more reinforcement and extension across units 14 Organisations: The design and practical limits of piloting have left some units with partially reformed systems, extending over only a part of their localities and part of their bureaucracies There has been very variable depth of practice and impact, across pilot units and within pilot agencies The impact of SLGP on organisations in planning and development has been limited by having too-little horizontal cooperation and coordination, and too-limited resources with which to apply new methods and to build harmonised preparations and cooperation The finance sector parts of SLGP interventions (delegation of budgeting, mobilising resources) have lagged behind the planning parts, thus limiting the potential combined value of project initiatives Since some of the desired changes are possible under the devolved State Budget Law (SBL), it seems that some Provincial Governments now reflect the uncertainty and inertia in reform which national agencies and decision makers showed in 1995-2005 period 15 Innovation: It is possible under current laws and procedures for the Provinces to extend and equalise the spread and depth of beneficial pilot effects Some Central informants believe that subnational units could have innovated more widely using the SLGP products and opportunities Some subnational authorities have proceeded to institutionalise, within limited mandates, such as the delegations on investment project management in four pilot provinces, and propagating standard instructions and formats to non-pilot units Some subnational informants believe that the national level has been ‘too slow’, such as in working on national planning guidelines since 2005, on a planning decree since 2008, and a trust fund for planning reform hrg1650389134.doc SLGP EOP Evaluation 16 Cautiousness: Subnational opinions about apparent delay in innovation of SLGP methods and documents gather under the explanation that national-level institutionalising is required to direct the use of new methods and devolution, as local partner-initiators sense that they have reached the limits of work-culture tolerance and nationwide ideology in decentralisation and grassroots democracy EOPE strongly suggests that leaders should honestly examine the locations and reasons for cautiousness, so as to provide guidance to other projects in subnational governance 17 National Gains: Although CPMU staff believe that they have clear gains from the local levels, there is not enough clarity in the upward learning by CPMU and MPI, by which to demonstrate linkages of lessons from pilots into national deliberations and policy decisions, and into the directions for where to go in decentralisation Project communications have yet to be optimally effective in harmonising stakeholders’ understanding of objectives, progress and implementation issues, in order to enlarge project impact (see, for example, the recommendations in each of the Provinces’ Final Progress Reports) This condition is a serious anomaly in a project advocating and enabling vertical and horizontal consultation and transparency as necessary conditions for achieving its Outcomes 18 Dissemination: Although there are around twelve attractive publications (see Annex 10), and evidence of ‘crossover effects’ to six donor projects, there is not enough clarity and not enough purpose in the processing and dissemination of SLGP information and products among GOVN agencies and Donor partners CPMU-MPI is yet to exploit the potential of the capital created under SLGP in applicable knowledge and constructive professional networks, while PPMUs fear that their small ‘footholds’ in subnational units will soon be eroded 19 A giant, outside? In the overall national context, and in the localities, the rapid growth of private enterprise and diversified investment complicates the challenges and knowledge required at community-based planning, managing projects and oversighting services SLGP methods, tools and training have not yet given adequate attention to local level integration of public and private domains and functions Lessons 20 Assessment: EOPE’s assessment of effects and impacts would have benefited from recurrent and objective monitoring, throughout SLGP, of the coverage and effects of Project activities, especially the training and documents, and of the consistency of applications and follow-up Reliance on people’s memories of then, and on opinions of now, does not give enough insight into organisational changes 21 New knowledge: Most of the lessons-to-be-learned were already known at design stage (as was shown in the Risks Matrix, for example) and at MTE This finding implies that SLGP could have implemented more confidently, quickly and broadly, for example, in strategic and participatory planning The constraints were not ‘technical’, but rather more in the mechanisms and ideology of decentralisation and People-State relations 22 Making space: Participatory planning, budgeting, deciding and monitoring obviously require the sharing of responsibilities and functions which are already being exercised by specific persons in authority, most of whom are naturally reluctant to ‘give away’ their functions Project designs should contain an explicit analysis of political and administrative feasibility of their proposed interventions 23 Risk exposure: Strong ownership of the pilot provinces (and delegation) does enable and propel the initiatives However, subnational partners believe that they need the permissions and disciplines of national rules and guidelines, to be spread across all units hrg1650389134.doc SLGP EOP Evaluation Pilot partners took the risks in technical competence and in the organisations’ culture and hierarchy, but quite soon they have sensed the political limits of the decentralising reforms 24 Synergy: SLGP targeted parts of the PBS to fit its desired Outcomes, but soon stalled on financing, specialist staff and oversighting constraints Such experiments would work more successfully and effectively within a broader national framework for local governance reform to guide the process across multiple, scattered and overlapping reform interventions; and by linking the SLGP activities to social security, safety net and social protection efforts, to ensure successful poverty interventions 25 Cohesion: Translation of leaders’ strong political commitment to the PBS reform, into generous, regular and specific guidance and goal-oriented project management, is an essential factor in ensuring a cohesive vision towards project success Project owners and donors should critically and openly examine ‘leadership feasibility’ as a critical resource in project designs, management arrangements and responses to feedback over project duration 26 The new dynamic: The drive, mobility, flexibility and individuality in private sector raise big challenges to State management at all levels; but the words not appear in SLGP Can any later project concerned with local empowerment and social equality ignore private-public coordination and cooperation as a determinant of local conditions and opportunities? Recommendations 27 Focus: SLGP has achieved some important results, and partners appreciate significant benefits But the efforts have turned out to be too fragmented and delayed to make any significant gains in strategic development Future projects should make more fundamental and explicit changes at mid-term to ensure more impact, even if the range of issues, activities and products is narrowed 28 Re-think: The wave of decentralisation and democratisation, on which SLGP arrived, has abated, indicating that organisational change may be slowed at national and provincial levels It may be more effective now to pursue those strategic aims of more inclusive, transparent, pro-poor and gender-sensitive planning, investment and services, through sectoral and limited-locality interventions, including direct support to disadvantaged households This strategy does not wait on local government systems to achieve such results and effects; but it seeks to empower the local people to participate in the rapidly privatising socio-economy, and thence to have more leverage on local governance 29 Close-sensing: The coverage, depth and applications have been very variable, as can be expected in a pilot scheme with a process approach, and delegated implementation Project management of such conditions requires regular, objective, qualitative and quantitative measuring and evaluating of the practical depth and strategic effects Good M&E should be a covenant in Donor partnering 30 Recognising Inertia: Capacity building was well received and was successful on the individuals’ scale, but not so apparent amongst organisations Training events and numbers faded after 2008, especially on financial and monitoring topics The dimensions and potential constraints of work culture and hierarchy politics must be explicitly appraised in project feasibility and risk assessment 31 A dividend?: A great amount of knowledge and network capital has been created for practical applications in Outcome 1, and for propagation in Outcome 2; but the realisation is yet to be demonstrated CPMU and PPMUs should pursue the synthesis of lessons so as to ensure that there will be ‘proof’ of SLGP’s value in: hrg1650389134.doc SLGP EOP Evaluation a) the new planning decree, b) the strategy and program for the decree’s implementation, and c) any instructions by Provincial levels to District and Commune units 32 Out-selling: Technical and professional crossover from SLGP to GOVN and Donor agencies has been too opaque and non-deliberate Such projects should delegate and fund (with payments by results) an autonomous working group, having the duty and mandate to propagate the information and approved products, and giving more emphasis to small meetings and online discussions, rather than on distributed documents 33 Contingent relevance: SLGP’s design remains basically relevant (with a focus on bottom-up and participatory planning and budgeting) - but only if one assumes: a) that national and ideological factors are favouring both faster decentralisation and politico-administrative diversity; and b) that more resources will be available to actually apply new methods and responsibilities 34 Role and Methods: It seems that local governance will become increasingly important as democratisation expands and deepens, as outlined in the draft SEDS/SEDP, and will remain a crucial component in the One Plan for 2011-2015 However, the communal-based planning and managing methods will have to adapt radically if they are to remain relevant in the surging private market systems, which now engage so much of the time and concern of leaders and staff, especially at Province and District levels, and on which many ordinary people now attach their livelihoods and family opportunities 35 Vertical accountability: Some Provinces report that the ‘project has not been aggressively directed’, that ‘no units comply with certain directives’, and that ‘there has been no enforcement of implementation’ Project management would be more effective if national level project leaders demand more scientific evaluation of applications and uptake through agencies, and of the desired changes towards inclusive, pro-poor, gender-sensitive and transparent practices being rooted and sustained in pilot units; and then devote time to substantive directing and follow-ups 36 Work allocation: The new planning approach has brought new demands on staff, some of which are difficult to sustain Designs of projects must make a thorough assessment of the flexibility and capacity of human resources in host agencies 37 Outlook: The evidence (including Provinces’ final reports) inclines towards a pessimistic view of sustaining Project results, as no legal framework exists on reformed PBS, and some applications in pilot units – too brief, too narrow or too shallow - are already fading The further delay in promulgating the long expected planning decree, the lesser the Project results will be sustained 38 Securing the gains: MPI should make a detailed ‘progress report’ on its drafting and its intentions, to be delivered to all concerned units, agencies and donors If the decree is likely to be delayed until 2012-13, MPI should assemble the practices into temporary national guidelines for SEDP, to keep the waning momentum created by the Project, and to guide continued reform on the ground MPI should enable a task force / working group to continue the essential tasks at national and four pilot agencies 39 Sharing now: GOVN and donors should combine in 2011 to refine and package the products of SLGP, by means of a working group; and to produce a comprehensive and indepth analytical mapping of local governance initiatives hrg1650389134.doc SLGP EOP Evaluation 40 Competence: There is not yet enough evidence that staff and leaders can actually perform their duties to the standard required to achieve strategic aims, and there are suggestions that skills and knowledge are fading Before implementation of any new decree arrangements and functions, a scientific, objective baseline test is required of ability-to-do and knowledge of essential principles and techniques 41 In summary: designs in subnational reforms of PBS must appraise the macro-context, ensure that there are institutional permissions and disciplines, investigate the administrative feasibility, explore private sector dynamics and opportunities in the target areas / functions, set modest goals but with the flexibility of a process approach, make close theoretical and practical links from activities to strategic aims, focus resources to make robust and deep impacts in a few needy places (including households) and functions, cultivate permission at the centre but devolve responsibilities and budgets, empower the local beneficiaries to oversight (monthly) the project performance, and apply accountability equally to project owners, staff and beneficiaries hrg1650389134.doc SLGP EOP Evaluation Acknowledgement The EOPE Team members are grateful to the leaders, staff, representatives and beneficiaries who gave us the time and effort to recall, review, explain and appraise what was done under SLGP and what was/is its value in their lives, work, their constituents and their organisations The Team greatly appreciate the guidance, advice, documents, information and facilitation provided by MPI and UNDP, by the Provincial level Project staff, and by staff of cooperating Donors in Hanoi We are especially grateful to the Provinces for their Final Progress Reports, which state clearly and frankly the record of achievements, shortcomings and continuing needs In addition to the persons listed in Annex 6, we thank the hundreds of others who have contributed to plans, reports, reviews, documents and meetings referred to in Annex 10 The Team members have acted as independently as possible within the constraints of time and the EOPE’s reliance on the goodwill, documents, memories and opinions of people who have committed much time and energy and career-interest to the pilot project Please allow us the duty to comment as fairly as we can on what we have heard, read and observed, recognising that our conclusions might not might not align with what the participants and managers believe to be the true or full picture, or a correct interpretation We hope that our small effort, even with some contrarian views, will contribute a little to appraising and drawing together the vast and diverse experience in participatory and strategic planning, budgeting, financing, servicing and monitoring, towards incorporation and enactment through revised legal instruments, reformed systems and empowered-inspired beneficiaries at all levels hrg1650389134.doc 10 ... extended, and supported with on-the-job training and guidance to units preparing annual plans Repeat and extended training has been provided by in-province planners in 200 8-9 -1 0 Trainers and managers... knowledge: Most of the lessons-to-be-learned were already known at design stage (as was shown in the Risks Matrix, for example) and at MTE This finding implies that SLGP could have implemented... and 200 6-2 010 Programme Component: Achieving the MDGs and reducing human poverty Total budget allocated: US$4,000,000 • • Project Title: SLGP Project ID: 00039111 Project Duration: 200 5-2 008 Management