Luận văn thạc sĩ Quản lý công: Research for monitor and evaluation system of Program 135 in Stage II (2006-2010), propose improvement solutions for purpose of evaluating the impact of Program

Purpose: The purpose of this thesis is the evaluation of monitoring and evaluation of the Program 135 at 2" stage, analysis of strengths and weaknesses of the tools and methods of monito

UPPSALA UNIVERSITÉT &

UPPSALA UNIVERSITY OF EC! SINESS,VNUH

UNIVERSITET The Rm

MASTER THESIS OF MPPM

Research for monitor and evaluation system of Program

135 in Stage II ( 2006-2010), propose improvement

solutions for purpose of evaluating the impact of Program.

Authors Tran Kien

Lo Viet Phuong Hoang Van Tuyen

Supervisor PhD Sven-Erik Svärd Local Supervisor: Dr Quach Manh Hao Class MPPM INTAKE 4B - Group 16

During our process of research and completion for this thesis, theauthors have received encouragement and assistance from many individualsand groups

In the first place, the authors ourselves should express the deepgratitude to PhD Quach Manh Hao, Deputy Dean of Faculty of Finance andAccounting, Economics College - Vietnam National University and PhDSven-Erik Svard (Uppsala University, Swiss) who are always whole-hearted to instruct and help us complete this thesis

We do express our thank you to teachers and lecturers, ManagementBoard of Economics College (Vietnam National University), Uppsala University (Swiss); Leaders and officers of Centre for International Education and Training; Leader of Committee for Ethnic Minorities, Department of Ethnic Polocies, Project Management Unit of enhancingplanning quality and implementation of poverty reduction policies for theethnic minority and mountainous areas (Committee for Ethnic Minorities);Embassy of Ireland in Hanoi have created the most favourable conditionsfor us during out researching and learning the

During our research, we also get the help and valuable comments ofthe Committee for Ethnic Minorities of Bac Kan, Dak Lak and Soc Trangprovinces, leaders of People's Committee of districts, the departments anddivisions of the districts and communes in three provinces such as BacKan, Dak Lak; Soc Trang who have helped us complete this research;through this, allow us to give our thank you to the above units and

agencies.

Finally, we do send our deep gratitude to our family, friends and

colleagues who always give us encouragement and create the mostfavourable conditions during researching and learning

Hanoi, February 10" 2012

On behalf of authors

Head of group

Tran Kien

REPORT OF THE THESIS SUMMARY

1, Name of the topic: Research of the Monitoring and Evaluation system

in the Program 135 Phase II (2006- 2010), proposals on recovery

solutions toward impact evaluation objectives of the Project

2 Education level: Master Course on Public Administration

3 Group of authors: Tran Kien, Lo Viet Phuong, Hoang Van Tuyen

4 Instructors: Dr Quach Manh Hao and Dr Sven-Erik Svärd.

Completion time: February in 2012

6 Context: The Program 135 — Phase II (2006-2010) has applied newmonitoring and evaluation tools in comparison with traditional method applied in other similar previous Programs However, monitoring andevaluation work in the program remains shortcomings because this is thefirst time for the tools to be applied to the program of authorities in alllevels In addition, this monitoring and evaluation tool has not beendeveloped fully because it only evaluates short-term results when the program impacts local people with the policies instead of pointing out deeper and overall impact of the program.

7 Purpose: The purpose of this thesis is the evaluation of monitoring and

evaluation of the Program 135 at 2" stage, analysis of strengths and

weaknesses of the tools and methods of monitoring and evaluation of theProgram; analysis to adjust and correct weakness in order to develop aframework for monitoring and evaluation of the program 135 at the nextstage aiming at monitoring and evaluation based on the results / impact;giving policy recommendations for monitoring and evaluation program inaccordance with actual conditions in the next period.

8 Method: This is a qualitative study, using the methodology of study ofprimary and secondary documents, site observation, group discussion and

use of questionnaires interview of performance by all levels On that basis,

SWOT analysis and logical framework are used to analyze, evaluate and

ive recommendations.

9, Results: Group searches following main results

- Solution on improving effectiveness of monitor and evaluationtools of the Program 135, phase II

- Monitor and evaluation framework for the Program 135 for the next

ages

st

- Recommendation so that the system monitors and assesses theProgram based on results/ impact

- Recommendations on policies to mission on monitor and evaluation

of the Program effectively

10 Propose for future research: Due to sample research is small, just at 3provinces with limited time and scope of the thesis, group of authors does not have ambition to comprehensively change, we only want to furtherfurnish information on proces of monitor and evaluation of the Program

135, phase I] and a group of solution for the Program, next stagAdditionally, group of authors also recommend to further research on thecontents of community- based monitor and evaluation.

By the large programs and limited resources of locality, the community-useform is important in assuring of the publicity and transparency, promotepeople’s ownership and ensure the closeness of the evaluated contents

11 Thesis contributions The authors believe that the information given in this thesis is the credible ba s for the policies planners and managers at the

governmental, provincial and district levels Particularly, for the Committee

of Ethnic Minorities, Department of Ethnic Policies (where members in theteam have been coordinating) with the desire of better support in improvingand ensuring the implementation of monitor and evaluation for Program

1.1 Reasons to choose the research topic

1.2 Appropriateness of the research topic

ANALYSIS OF CURRENT SITUATION OF THE MONITORING AND

EVALUATION FRAMEWORK AND MECHANISM OF THE PROGRAM 135

PHASE II

1 Basic information on the program

1.1 Introduction about the Program 135-1.

1.2 Introduction about the M& E System of the Program 135 II

1.3 Logical framwork and route-map framework

1.4 Basic inspection tool:

1.5 AMT/PMT monitoring and reporting system

1.6 The internal M & E activities

1.7 The internal M & E activities

1.8 The independent M & E activities

2 Community-based monitoring and evaluating

CHAPTER H 27

THE SOLUTION OF OVERCOMING AND COMPLETING M & E TOOL OFTHE PROGRAM 135 AT 2Ÿ” STAGE ©5)

1 Logical Framework 27

2 Baseline Survey.

3 The AMT/ PMT monitoring and report:.

4 The M&E internal and independent to the CT:

5 Some proposal to enhance the role of community supervisor in

implementation of Program 135 in the next period:

CHAPTER II.

PROPOSE SOLUTIONS TO ENHANCE MONITOR, EVALUATION ON

R LTS FOR PURPOSE OF EVALUATING THE PROGRAM IMPACT

THE NEXT PERIOD

1 Status of M & E capacity and performance of monitoring and evalu:

based on results towards the target of impact evaluation of the Program

1.1 Human resources for monitoring and evaluation activities:

2: The budget for M & E activities:

1.3 Training for M & E based on the results.

2 Proposed monitoring and evaluation framework for the program at the

next stage

2.1 Complete the institutionalization of M & E work

2.2 Organization of supervision and evaluation

Supervisor

2.3 Development of M & E plan and design of monitoring / evaluation:

2.4 Development of M & E indicators in the logical framework

2.5 Monitoring progress

2.6 Monitor on assessing Program 135 ID)

2u Construct tool set of indicators for monitoring and ašSeSSiHg

2.8 Build a network of information (Web page)

2.9 Collect related secondary data

2.10 Perform the monitor and evaluation

3 Some measures aimed at targets of monitor and evaluation based on the results/ impacts on the Program for the next period

CHAPTER IV.

RECOMMENDATIONS TO PERFORM MONITOR AND EVALUATION

2 Documents on organizing M&E system, the program should be issued and the personnel system for M&E is promptly formed to grasp the content and the execution steps of the program 50

3 Early issue operation mechanism of M&E system with the participation of

all levels from centre to local levels 50

4 Issuance of guidance documents

5 Construction of software of data system and information link

6 M& Ework is an integral part of planning

7 Necessity of minimum regulations to publicly report contents

8 M & E skill intensification:

LIST OF ABBREVIATIONS

AMT Aligned Monitoring Tool

PI35-II Program 135 , Stage II

P135 — III Program 135 , Stage III

EMPCD Strengthen Capacity for CEMA in Ethnic Minority

Policy Formulation, Implementation and Monitoring "M&E Monitor and evaluation

JPR Joint Progress Rating

Logframe Logical frame (M&E Frame)

MIS Management Information System

PMT Provincial Monitoring Tool

SM-P135 Project of technical support for Program 135

management

CEMA Committee for Ethnic Minorities

UNDP United Nations Development Programme

WB World bank

1 Necessity of the research topic

1.1.Reasons to choose the research topic

Vietnam has recently made impression with its poverty reduction inachivements on social and economic development These achievementshave been internationally recognized and one of important reasons is thatthe Government of Vietnam has offered so many special priority forpoverty reduction policies, especially poverty reduction programs in thearea of ethnic minorities, mountainous and extremely difficult areasthroughtout the country

Program 135 Phase II (2006-2010) is a comprehensive economic andsocial development program for the ethnic minority and mountainous areasissued by the Government with positive effectiveness, which is supported

by the whole society One of the most prominent traits of the program is to apply a lot of new monitoring and evaluation tools in comparison with traditional method applied in other similar previous Programs The use of new monitoring and evaluation tools brings positive effect, contributing to the overall effectiveness of the program However, monitoring and evaluation work in the program remains shortcomings because this is the first time for the tools to be applied to the program of authorities in alllevels In addition, this monitoring and evaluation tool has not beendeveloped fully because it only evaluates short-term results when theprogram impacts local people with the policies instead of pointing outdeeper and overall impact of the program

The Program Document 135 consists of so clear General Objective

of the program The specific economic objectives are clearly stated to meetcommon objectives However, the results of the activities (tasks) hasindexes only but no specific indicator is stated as a basis to evaluatewhether to achieve the objectives or not Another important program toevaluate appropriateness to targets must reflect its logic and indexesreflecting the logic of cause and effect

The researchers thought that the research matter is practical to thecontents taught in the Master Course on Public Administration, especiallytheories that the researchers studied in Monitoring and Evaluation Subject(MPPM6).They shall make the comparison between the studied theoies andpractice from the Program 135 and their practical experience to learnlessons and to propose recommendations to improve monitoring andevaluation work of the Program in following periods

1.2 Appropriateness of the research topic

1.2.1 Appropriateness of the research to current trend

In the context of administrative reform aimed at building atransparent, sustainable, professional, effective and valid democratic administration, intensification of monitoring and evaluation work is soimportant Many programs and projects are invested with very large fund

by the Government of Vietnam but their monitoring and testing is not paidwith proper attention, causing loss and waste, impacting significantly onquality and effectiveness In addition, monitoring and evaluation areeffective tools when Vietnam is in the process of integration with the worldand toward a transparent administration

1.2.2 Appropriateness of the research to the works of the group of

The researchers are now working at construction agencies and incharge of guiding the implementation, monitoring and evaluation ofpolicies in mountainous and ethnic group areas Therefore, this studyclosely linked with the assigned missions The research results will beapplied directly to the practical work of the authors to meet the requirements on granting the Communce authority the position of theEmployer in the Program 135 Period 2012-2015

2 Research objectives

2.1.Overal objectives

The group of researchers focuses on clarifying What difficulties andchallenges are faced as applying new monitoring and evaluation methods tothe Programme 135-II? What result is it? How is it different fromtraditional monitoring and evaluation method? Then, the team wants tocombine and compare the monitoring and evaluation theory studied in thesubject of MPPM6 (Master course on public administration) with themonitoring and evaluation practice in the Program 135 - II and theirpersonal experience to propose recommendations for monitoring andevaluation work to evaluate their impacts better in poverty reduction

programs in the next stages

2.2.Specific objectives

The study focuses on three specific objectives, including:

(i) To carry out a study on the current monitoring andevaluation framework and mechanisms of the Program 35-II;

(ii) To identify options for strengthening the result

based-program monitoring and evaluation work:

(iii) Tto propose the monitoring and evaluation framework the

Progrram 135 in next stages;

3 Research Scope and Objects

The research object of the subject is a monitoring and evaluationsystem of the Program 135 Period II

The scope of research is carried out in three provinces of Bac Kan, Dak Lak and Soc Trang In each province, the researchers chose twocommunes to make surveys and collect views

4 Rearch methods

4.1.Research Methods

- Collection and analysis of secondary data

- Collection and analysis of primary information:

- Observations at the scene and interviews

1 Reading existing concerned documents of CT

2 Considering current M & E arrangement such as:

- The institutional arrangement: M & E framework, ssigning M & Einternal agencies and personnel and independent monitoring.

- The technical arrangement (M & E tools): logical framework, routemap, the basic inspection data system, AMT / PMT monitoring andevaluation software system; Policy Matrix; guidance to M & E work.

3 Evaluating AMT / PMT system, the sets of indicators to considerwhether the reflection degree is effective and sufficient to reach expected

results

4 Summarizing the data collection process to provide a set of indicators on poverty reduction and the reliability of the data, evaluatingthe operation of the database on computer of the Program

5 Assessment of internal and independent Monitoring and Evaluationwork of CT

6 Discussion of result-based M & E improvement methods.

5 Thesis structure:

Chapter 1: Analysis of current situation of the monitoring and evaluation framework and mechanism of the Programme 135 Phase II.

Chapter 2: The solutions to overcoming and completing M & E tools

of the Program 135 Phase II.

Chapter 3: Proposal of solutions to improvement of monitoring andevaluation work toward result/ impact based- evaluation of the program infollowing periods

Chapter 4: Recommendations to effective monitoring and evaluationwork in following stages

ANALYSIS OF CURRENT SITUATION OF THE MONITORINGAND EVALUATION FRAMEWORK AND MECHANISM OF THE

PROGRAM 135 - PHASE II

1 Basic information on the program

1.1.Introduction about the Program 135-II

Economic & Social Development Program in villages, particularlydisadvantaged, ethnic minorities and mountainous areas in period 2006 -

2010 (hereinafter referred to as the Program 135 — Phase II- which isCTI35IH in abbreviation) is the most important economic & socialdevelopment program of Vietnam Government with focus on sustainablepoverty reduction for ethnic minorities, remote regions and mountainousareas throughout the country The National Committee is the main standingagency coordinating with other ministries to help the Prime Minister tomonitor, evaluate the implementation of the program.

The objective of this program is to create rapid changes inproduction, promoting the change of agricultural economy restructuringtowards production associated with market; reforming and improving thematerial and spiritual life of the people of peoples in DBKK communes andvillages in a sustainable way, reducing the development gap among peoplesand among regions in the country It makes the best efforts to eliminatehungry figures, reducing poverty to below 30% and over 70% ofhouseholds achieve the income of 3,5 million/ person / year by 2010

The program points out four tasks (1) Support for development and

production and shifting economic structure, improving the production level

of the minority peoples (2) Development of essential infrastructure in theextremely difficult towns and villages (3) Training and fostering basis civilservants, improving administrative and economic skills and knowledge andpublic training to enhance to provide legal support to raise legal awareness.Four above tasks are achieved by 3 projects (infrastructure, staff training, and development of production) and one policy on improvement of living

standards

For beneficiaries of the program include DBKK communes, bordercommunes and communes in the safety zone and DBKK villages TheProgram is related to 1,848 communes and 3,274 DBKK villages of 369districts of 50 provinces The Government and some donors have set up thePartnership for Budget Support to the Programme 135-II (to improveefficiency and quality of the Programme 135-II and use Joint ProgressReview (JPR) to assess progress and implementation of policy dialogueprograms during the program implementation, since 2007 The changefrom sector toward support of donors to the Program 135-II help better collection of resources to the Program and harmonization of procedures and especially, improvement of partnerships between the Government of Vietnam and donors in the policy dialogues to improve the poverty reduction measures Therefore, the Program 135- Phase II program isdesigned comprehensively in the basis of evaluation against previousdifferent results in the first phase and combination with some advances and significant difference in some following important aspects:

+ Firstly, the Program — Phase II attracts resources of donors,

making up 30% of the total budget (about $ 1 billion) in the form of

support to budgets with conditions apart from Vietnamese Government’sbudget.

+ Secondly, scope of the Program — Phase II is broader - including

additional components on earning improvement in rural areas andcomponents of agricultural production support component in addition to thecomponent of traditional infrastructure development — apart from a clearerdirection on decentralization and distirbution of more resources forcapacity building

+ Thirdly, the Program — Phase II has been paid significantly andeffectively to monitoring and evaluation activities of the Program.Accordingly, the logical framework, policy matrix (with 13 points) with asystem of baseline survey data system, monitoring and evaluation systemshave been established with the support of donors for the program

monitoring and evaluation

1.2.Introduction about the M& E System of the Program 135 II

The Program 135II is a major program of the Government with arelatively complete monitoring and evaluation system from the central tolocal levels To monitor and evaluate scientifically the results and impact ofthe program in 2007, the technical support of the project VIE 02/001 hasassisted the UBDT to organize investigation of beginning- period data basefor phase II program as the basis for assessing the impact of the program

To monitor the output, input, disbursements and other indicators, theGovernment has issued the Decision No 04 on the application of AMT andPMT monitoring system to supervise the implementation progress of the

Program

The program is designed with an overal monitoring and evaluation

framework for overall, an implementation framework and framework ofpolicy matrix to commit to the the Government and donors on the policybased- actions The frameworks are important tools for the stakeholders tomonitor the implementation and evaluate the results.

The implementation the Program is conducted strictly from central tolocal levels At the local level: communes, districts are using AMTsoftware software to report the implementation of the Program Apart fromthe use of AMT software, the local authorities in the communes anddistricts have to report in written form or make quick reports to provinces and districts Then, the province authorities shall generalize all AMT reports of the districts with PMT software and report in written form to the Central and and provincial authorities for the Program implementation The supervision role of the community plays very important roles in supervision of implementation quality therein.

In addition to direct monitoring and evaluation, the Program isalso monitored by independent monitoring and evaluation agenciessuch as National Assembly Council for Ethnic Minorities, StateAudit of Vietnam, UBDT inspectors Monitoring activities are regularly held by the provincial People’s Council in the provincial level with representatives of Committee for Ethnic Minorities, Departments of Economics, Finance and Planning directly underHDND and concerned agencies of the province

In addition to the above monitoring activities, there are otheractivities namely preliminary conference, summarized meetings atvarious levels to assess Program implementation

M & E system of the Program 1351] is considered relative

10

sufficiently with clear legal framework, comprehensive monitoring andevaluation framework (logical framework) with AMT / PMT softwaresystem for monitoring the implementation and the results of CT TheProgram also conductes baseline survey before the intervention of CT tothe project area to obtain the input data for assessing outcomes after theProgram finishes.

The Program is monitored, supervised, and inspected by internalmanagers and inspectors of the Committee for Ethnic Minorities Inaddition, the joint assessment team also regularly engaged in monitoringand evaluating the performance of the Program, particularly with theinspection, monitoring and evaluation of independent agencies such as theNational Assembly (People's Councils at all levels ), the State Audit ofVietnam and the independent evaluation teams

1.2.1 M & E organization, guidance and implementation structure(Advantages; Some difficulties / disadvantages; The proposed solutions)

a Advantages

* The system of documents (decisions, regulations, resolutions,

information, guidance, etc, from central to commune and the village) areissued to be legal basis for implementing the Program

The Program - 135II has issued many legal documents prescribingmonitoring and evaluation of the Program, including the Circular No.01/2008/TTLT-UBDT-KHDT-TC-XD-NNPTNT, the Decision No.04/2008/QD-CEMA, the Decision No 74/2007/QD-UBDT Thesedocuments guiding the inspection, monitoring and evaluation to theprovisions of regulations and reporting forms The Program has issued the

implementation Framework for 2006-2010 and implementation manual tosupport monitoring, evaluation and reporting of the Program.

The Joint Circular No NNPTNT of the Committee for Ethnic Minorities, the Ministry ofPlanning and Investment, Ministry of Finance, Ministry ofConstruction, Ministry of Agriculture and Rural Development, guiding the implementation of the social economic development in communes, especially extremely difficult communes, mountainous and ethnicminority areas in period of 2006-2010

01/2008/TTLT-UBDT-KHDT-TC-XD-The Decision No 04/2008/QD- UBDT on on August 8, 2008 of theMinister, the Chairman of the Committee of Ethnic Minorities on issuingthe Regulations on reporting regimes and forms of the Program on social economic development in communes, especially extremely difficult communes, mountainous and ethnic minority areas in period of 2006-2010.

The Decision No 74/2007/QD-UBDT March 12, 2007 promulgating the routemap framework to the Program 135, period of 2006-2010 This isthe framework plan on implementation and monitoring in the whole 5 years

b Disadvantages

12

Although the Circular No 01/2008 and Guidelines manual includesinstructions mechanisms for inspection, monitoring and evaluation, theseguidelines have not yet comprehensive and they only focus onInfrastructure Development component (PTCSHT) without the instructionsfor the remaining three components, as well as the guidelines for M & Eplanning, budget making and implementation of the Program Theemphasis on Infrastructure Development component is true and necessary,but monitoring work is required to the entire project and the rest should not

be regardless According to reflections of provinces, other componentsshould be offered special monitoring, especially to check whether it issuitable to the beneficiary objects or not Have they gained prescribedrating of the Program? How to support? Etc to secure support to correctbeneficiary objects, contributing to meet given targets of the Program

1.3 Logical framwork and route-map framework

a Advantages of the logical framwork and route-map framework

The logical framework of the CTI35II is prepared by UNDP, theConsultants supporting the Committee for Ethnic Minorities This is thefirst time for the Government to issue a M & E framework to a CT withnational targets The Government had received flexibly the opinions ofdonors to prepare and harmonize M & E framework and route-map of CT (logframe concept is relatively new to the Government - Committee forEthnic Minorities at that time)

In our opinion, M & E framework is quite comprehensive andcomplete It presents the relationship between the overall objective of theentire program and its components (between indicators reflecting theoverall objectives of CT and other indicators of input / output and outcome

of components and indicators reflecting the results of the management of

CT) The framework also raises the information to be collected, the

frequency and the time necessary to collect data sources, the agency

assigned to collect data as well as coordinating agencies to supply data.Designed M & E framework includes three types of indicators: inputindicators, output indicators and indicators of results / impact Theframework of decentralization and spatial indicators (it means indicatorsthat will be

Along with M & E Framework, Route-map Framework of theProgramme 135 for the period 2006-2010 has been prepared and issued.This is the implementation and monitoring framework in entire 5-yearperiod of CT The framework is relatively completely designed includingthe results of CT directed to, the intermediate results to be achieved as peryears of implementation of the Program The route-map Framework ownsclear milestones to help to monitor the Program more effective, which

should be further promoted in Phase III of the Program.

b Disadvantages of Logical Framework and route- map framwork

Although they have been used in some studies (such as performanceevaluation, some of the workshops involved) but the logical framework ofCT135 II and the route map 135 is still formal and rarely used bystakeholders in the implementation process of the Program During theimplementation of CT, donors and the Government are using the PolicyMatrix as the condition to disburse and promote implementation progress.

Policy matrix includes 12 main contents of the Program 135 thatstakeholders are interested, of which each includes objectives and triggers(similar to the indexes) to measure achievement degree of objectives

14

Policy matrix is designed simply, easily to use with the primary goal is topromote disbursement and the implementation progress of the Program.

Why is the logical framework used and both the donors and theGovernment must use a tool called matrix of policies at the same time? Inthe basis of our study, the main reason is due to "logframe prepared by theGovernment and in that case there is no consultation with the sponsors",

"The goal of the Government and the donors are available at the same timebut they can not be combined in a joint framework "

1.4 Basic inspection tools

a Advantages

Baseline Survey of the Programme 135-II is performed in 266communes in the group of beneficiaries from the Program and 134communes excluded in the Program to make comparison on informationcollected in aspect of indexes of the two groups in beginning period of theProgram (2008) CT basic inspection has supplied data base with highquality and it mays be the most detailed and sufficient existing set ofdocuments on ethnic minorities BLS provides important and properbackground to the Program impact assessment as it ends

(Mainly due to high costs and much time) leading to no necessary data todemonstrate the impact of the program

1.5.AMT / PMT monitoring and reporting system

1.5.1 The general information about AMT / PMT monitoring and

reporting system

PMT is a tracking tool designed to support the Committee for EthnicMinorities (Central level) and provincial People’s Committee (local level)

to monitor the performance of districts participating in the program undertheir management Similarly, AMT helps the districts to monitor theimplementation of communes’ participation in the program to find outappropriate management decisions Also, this monitoring system providesrelevant data for report.

Communes will prepare AMT consisting of 4 Tables (Table |: Basic information; Table 2: Report on Progress and Disbursement of capital ofprogram 135; Table 3: Statistics of the contract; Table 4: The performanceindicators and output) District AMT report includes two forms MS01/BC

of quarterly performance report of the district and MS02/BC of annualperformance reports, accompanied by two reports of four AMT forms at thedistrict level Similarly, at the provincial level, reports include two forms:PMT MS03/BC of quarterly performance report of the province andMS04/BC of annual performance reports of the province, BC is attachedwith the PMT form extracted or summarized at the provincial level

a Status of the AMT / PMT monitoring system:

In addition to monitoring and evaluation framework (M & Eframework), the program has a toolkit which is an AMT / PMT monitoringsystem to monitor the implementation of the program However, theprovinces are not interested in the M & E framework of the program, theyonly know AMT / AMT PMT and the provinces see the lack of orientation

of objectives of the Project and results of the components (this section has already been in M & E Framework) AMT / PMT is mainly composed ofindicators to monitor fluctuations of input and output

Training the use of AMT / PMT for provinces has been conducted byCommittee for Ethnic Minorities since 2006-2007; provinces have prepared

16

AMT/Preports since 2008, and promoted it since the mid-term review inend-2008 During implementation, the Program also has the "Task Force" -

to support the implementation of program activities, including use of AMT

/ PMT to improve the capacity of the monitoring system AMT / PMT has

been deployed throughout 50 provinces involved in the program

According to the provincial assessment, the PMT / AMT is suitablefor national level and helps national level to collect and analyze data, butthis system seems not to be very useful to local levels In other words, thelocal level can not use this data for their purposes PMT / AMT should beuseful for not only central but all levels For such purpose, PMT / AMTshould be design in open way, which means that in addition to a number ofkey results and indicators in the general M & E framework, provinces canadd the specific information of each locality

The provinces also reflect some limitations in using the software asfollows: (i) The software does not allow adding columns, (ii) Few users canuse it simultaneously, (iii) When outputted in the form of Excel and reports

do not keep its original format

b Current status of the Indicators of AMT-PMT tool:

According to several interviewees, AMT / PMT both lacks indicatorsand has unnecessary ones While M & E framework of program hasindicators to measure the results / impact on every component, the AMT /PMT is mainly composed of indicators to monitor the changes, the outputand focuses little on measurement results, in other words lacks the results / impact indicators Therefore, at the third stage, it is necessary to have additional indicators for the results / effects of program in the AMT / PMTsystem, particularly in the context program is shifting to M & E based on

the results.

c Current status of the data collection:

The program data collection was conducted at the commune anddistrict levels to collect input data for AMT monitoring system This data ischange and fluctuation in the implementation of program in the communes

or districts in the period Data is monitored and collected on the quarterly and annual basis Communal level is the level directly implementing and monitoring the activities of the program deployed in the area and collectingdata for AMT monitoring system Data was collected at the communallevel or in the villages depending on the requirements of the indicator

content.

According to the reflection by communes, the village can collect data

of most indicators in AMT, but so many indicators and the complexity of the system has caused the burden on the communal level in data collectionand this could not be done well Therefore, in the next period, it isnecessary to simplify the AMT system, and remove some unnecessaryindicators within the framework of a program

At district level, in the process of program implementation, institutional changes affect the entire M & E work of program including the data collection for monitoring AMT system In 2008, eliminating Office for Ethnic Minorities leading to specialized agency in charge of M & E work.

Summary of data is divided into many offices! leading to overloading and

scattered data or no data for the system.

We also show some indicators not available or obtained at communallevel as below:

1

For example the training summarized by general education department, for infrastructure charged by the disggiet

project or the financial office plan for production summarized by General Agriculture Division

Table | shows some indicators without data as required in

AMT/PMT system

s ——

Table 1: Some indicators without data of the AMT-PMT system

Table 1: The basic information:

- The percentage of female students at the school age (indicator 20)

Table 4: The performance indicators or outputs

Manufacturing support component:

- Indicator 12

Infrastructure Works Component

- For indicators from 4 to7, data can not be collected from contractors This

indicator should be turned into: "The number of day work contributed by thepeople"

- Little data collect for the output indicators on Education and Health.

Component of living standard improvement

- It is difficult and takes much time for communes to get data and district can not get it for indicators 10, 11.

Table 1: Some indicators without data of the AMT-PMT system

d AMT and PMT report

13511 program has been implemented widely in 50 provinces, butonly about 15 provinces are able to prepare report of AMT at communal

and district levels and PMT report the provincial level’ In provinces, most

communes can not prepare the AMT report’ Some communes and districtsprinted out and gave it to communes but they can not prepare reports’.About 30 % of the provinces can prepare PMT report causing manydifficulties for the national level.

For example Bac Kan has 62 communes of 8 districts participating in the Program 135)

Committee for Ethnic Minorities have never s

(i) Central level lacks data for preparation of report, leading to nobasis for the levels of leadership to make immediate and accuratedecision on management;

(ii) If Central level can prepare the report, such report will have lowquality due to unreliable data’ That reduces the accuracy of management decisions given by the levels of leadership.

Some reasons for inability of communal levels to prepare the report:

(iii) Social level does not have enough time to prepare the report

(iv) The low qualification and capacity at the commune level, no staff

in charge of program 135 at communal level.

(v) Staffs working the program 135 at the provincial, district andcommunal levels had no allowance, so it is impossible to encouragestaffs to work for program

(vi) Staff transfer also caused difficulties for the report (The trained staffs were transferred or assigned to other tasks, and alternativeothers do not know how to do)

(vii) At the district level: In 2008, eliminating Office for EthnicMinorities leading to specialized agency in charge of M & E work.

however Provincial

ATM reports of the commune in hard copy as well as soft copies Only one district (Ngan Son) out of 8 districts may submit reports to the provincial level but such report still lacks data

* For example the training epartment for infrastructure charged by the district

Agriculture Division

20

Summary of data is divided into many offices leading to overloadingand scattered data or no data for the system The AMT reports wereprepared by district administrative staffs but they are very busy with

so many other works leading to limited quality of report

(viii) No mechanism is strong enough (salary reduction, no capital allocation ) to "force" the provinces to prepare reports and submittimely and strictly implement this mechanism

1.6 The internal M & E activities

Monitoring and evaluation are critical for management agencies topromptly monitor the progress of activities, inputs, outputs and results ofactivities as per originally designed monitoring and evaluation framework

of a project or a program Moreover, monitoring and evaluation activitieshelp managers to know whether the implementation activities can achievethe overall goal or not? Expected effectiveness can be achieved with therelevance of programs, policies for beneficiaries, the discovery ofinadequacies and solutions as well as other adjustment to theseshortcomings

1.7.The internal M & E activities

At the central level, Committee for Ethnic Minorities and ministriesalso assigned inspection groups to implement the program to timelydiscover problems and shortcomings in implementing and finding out solution for better local implementation and better performance as overall aimed by the program At the local level, the provincial and district directing board has frequently directed and inspected The province has established the Assistant Group for the Directing Board with members

from leaders at Department level such as DOF, DPI, and InvestmentCommittee , and Office-level leaders The district also has the program

13511 with the same components as at the provincial level At communelevel, there is no Supervision and Administration Committee Provinceshave also frequently supervised and monitored the implementation of theprogram in the framework and action plan of provinces in programimplementation At the communal and district levels, the monitoring andevaluation, besides working as investor by the implementing agency,supervisory boards of commune and community also participated in monitoring and supervising the implementation process, in particular the work of community supervision has brought greater efficiency in ensuringquality of implementation of activities

At the local levels, implementation of the S, M & E (Supervision,monitoring and evaluation) has also been done differently It has been donecomprehensively for all program’s components in some provinces, but foronly infrastructure under Circular 01/2008 in others In our opinion, focus

on infastructure developmnet is right and necessary, but we should notunderestimate the supervision of the rest, especially considering whetherthe support comes to right place, within the level as required by theprogram or not and how to support? to support the right objectscontributing to achieving the objectives of the program

However, in the process of M & E, the local level faceddifficulties: (i) The communes of program are mainly difficult communeslocated in areas where travelling is difficult or remote areas; (ii) The leveland capacity of the communes generally could not kept pace with othervillages (limited), (iii) Officers are so busy, does not have time to do or do

22

deeply, mostly on part-time basis, (iv ) There is no treatment regime (noallowance, no work fee) and (v) Funding for M & E is too limited.

The above difficulties have reduced effectiveness of supervision, so at thenext stage, the program should have more sufficient arrangements inhuman resources, enhance training to improve capacity for local officials,give better treatment regimes and financial mechanisms for M & E.

1.8 The independent M & E activities6

In addition to monitoring, inspection and supervision by theimplementing agencies, the program also includes the supervision andevaluation by independent agencies such as the National Assembly /People’s Council at all levels, Sponsors, the State Auditor / Inspector andthe independent evaluation teams Every year, the National AssemblyCouncil of Ethnic Minorities also regularly organizes the monitoring ofimplementation of national policies, including monitoring the program 135II and reporting to the Standing Committee of the National Assembly

on the implementation of national policy, and Congress on monitoringobjectives, effectiveness and impact of the program and monitoring tocriticize the policy with the role of the Legislative body

At the provincial, district and communal level, People's Councilplays a very important role in investment and in particular monitoring theimplementation of programs, the provincial People's Council holds annualmonitoring twice a year (Committee for Ethnic Minorities of the provincialPeople's Council and its relevant departments such as Department ofeconomy, finance and plan of the provincial People's Council) inconjunction with relevant departments at provincial levels to be on a fact-finding tour to monitor the implementation of the program.

In addition to monitoring and evaluation by the National Assembly /People’s Council at all levels, the Sponsors and the government have alsoevaluated the overall program during the implementation of program Thepurpose of General Program Evaluation is to monitor the performance ofprogram and timely detection of problems and inadequacies in the process

of implementation for timely management so that the program can achieveexpected objectives As committed to the Sponsors on the implementation

of General Program Evaluation between Sponsors and government, before

2009, General Program Evaluation has been conducted twice a year, butsince 2009, parties agreed to conduct General Program Evaluation once a

year.

State Auditor and the inspection teams of Committee for Ethnic Minorities also conducted financial audits and inspection of the implementation of program Because an area-wide support of the program,the audit was randomly sampled in the provinces, and mainly audit offinancial data and compliance auditing.

2 Community-based monitoring and evaluating

Community-based M & E is considered the successful model of the program I35II in general and other development support programs of Government and international organizations According to evaluation at central and provincial level, the community supervision of the program hasincreased transparency, prevent losses, ensure project quality and enhancecommunity capacity through direct participation in monitoring localactivities of the program Community can be involved in the planning,meeting, and discussion for options of investment and oversight

* Models of communal and community supervisory boards of the

24

program I35II

Communal supervisory board includes: Head of supervisory

boards is Vice President of communal People's Council or chairman of thecommunal People's Council, members of supervisory boards include frontChairman, Head of People’s Inspection Committee, representatives ofsectors such as Youth Association, Women's Association, Farmers!Association, and Veterans Association

When conducting meetings of people and giving priority toinvestment in a certain work, a community supervision board will beestablished for such works For activities in support of productiondevelopment, and livelihood improvement policy, the supervision was verydifferent among provinces; some provinces conduct the monitoring andsupervising activities of only infrastructure, but others conducted it ratherwell and established groups for self management and supervision as well assupervision of Village chief, Village Secretary and members in the villagedevelopment committee

For social works invested by district, the community supervisionboard was established One drawback is bad coordination between the supervisory board in such area and that of investor; the supervisory board

of Investor (A) signed the contract directly with the district and it follow only directions of the district and does not comply with directions of thecommunity supervision board; another limitation is limited qualification ofmembers in community supervision board causing difficulties forsupervision; sometime conflict between community supervision board andthe supervisory board of Investor leads to suspension of works whichwould continued to built only after dispute settlement

Beside the success and positive characteristics of the community

supervision board, it is necessary to improve some points for moreeffective monitoring by community:

Capacity: Current capacity of the commune level is limited Manycommunes are incapable to complete bidding documents, so they askedcontractors to prepare bidding documents and the entire process to beapproved by the district; quality monitoring was difficult because of limitedcapacity, communal and community supervisors can not the technicaldrawings, estimates and inspect quality of activities such as purchasingseedlings, machinery, cattle in the component of production support.Community supervision board was established, but many village andcommunity officials are not very keen on the job leading scattered andineffective supervision, particularly in communes with weak capacity due

to several causes including failure of Community supervision board insupervising, bad qualification of supervisors of Investor (A supervisor) and

supervisors of Investor colluding with contractors, which can not be

controlled by Community supervision board

M & E based on community has promoted strengths of M & E unavailable in other forms of M & E, worked with and added to other forms of M & E of program management levels, National Assembly / Council at all levels, the State Audit, Inspectors and Sponsors; it has contributed to enhancing the effectiveness of program.

26

CHAPTER II

THE SOLUTION OF OVERCOMING AND COMPLETING M & E

TOOL OF THE PROGRAM 135 AT 2Ÿ” STAGE

Monitoring and evaluation are critical for management agencies topromptly monitor the progress of activities, inputs, outputs and results of activities as per originally designed monitoring and evaluation framework

of a project or a program for timely and accurate decisions Moreover, monitoring and evaluation activities help managers to know whether theimplementation activities can achieve the overall goal or not? Expectedeffectiveness can be achieved with the relevance of programs, policies forbeneficiaries, the discovery of inadequacies and solutions as well as otheradjustment to these shortcomings for expected effectiveness of the program M & E system enhances accountability of all levels because of the proof for the activities, changes in the area with interference of the program and the obtained program results during implementation As per detailed research and analysis of the current M & E layout of the programfrom design to the actual deployment, based on existing evidence, researchsuggests that 135-II program monitoring and evaluation system arearranged and designed fully and completely, but the actualimplementation is still limited and inadequate, existing M & E systemcan achieve about 60-70% of goal of the original design

1 Logical Framework

M & E framework of the program has been developed during design

of the program This is the first time Government has provided a M & E framework for a national target program with technical assistance from UNDP This framework is rather complete but the monitoring and

evaluation under this framework have not implemented in a systematicway, Sponsors and Government used a policy matrix to facilitatedisbursement and promote the progress of program implementation.Solutions to a useful and proper M & E framework at next stages include:

(1) When preparing M & E framework, we should take comments ofthe parties, harmonize the desire of all parties When the parties agreedwith the logic framework, we must be based on this framework to monitor and evaluate performance to avoid further development other parallelframeworks / tools

(lii) M & E framework should be prepared to logically reflectbetween the overall objective of the program and the main results of thecomponents in a coherent, simple, understandable and easier to use way

2 Baseline Survey

Result before and after the intervention of program has an importantmeaning in considering the impact of the program, improving theAMT/PMT monitoring and report system Basic survey should beconducted before commencement of the Progam to identify initialindicators of the Program and set foundation for other activities andevaluation activities In adition to end-period basic survey, mid-term basicsurvey is needed to measure impact and adjust the implementation plan orobjectives in accordance with actual implementation.

3 The AMT / PMT monitoring and report:

PMT is a tracking tool designed to support the Committee for Ethnic Minorities (Central level) and provincial People’s Committee (local level)

to monitor the performance of districts participating in the program under

28

their management and provide relevant data for report The system hasbeen deployed widely in 50 provinces, but in the actual deploymentprocess, they faced some difficulties and shortcomings:

(i)

(ii)

According to several interviewees, AMT / PMT both lackssome indicators and also has unnccesary ones It lacksindicators to measure the results / impact on every component,and includes so many indicators to monitor the changes, theinput and output; there is no close logic relations amongoutput indicators of AMT/PMT system and result indicators in the M & E framework In the context of promotion of M & E based on results of the 3rd stage, it is required to review and add lacked result indicators and simplize Indicator set and retain indicators needed for measurement by the program for results and expected goals Indicators must meet the standard

of SMART or CREAM

Data collection of CT was conducted at the commune anddistrict levels to collect input data for AMT monitoringsystem Too many indicators, however, have caused theburden for implementation level The institutional changes atthe district level also affects the data collection and quality.Most indicators can be collected at the commune and districtlevels, however, there are some indicators impossibly collected, and some may be collected but not exact The decentralization also affects the investor in data collection, it is required to further decentralize the investor for communelevel

(iii) AMT/PMT report Qa report quantity: CT135II is spreadly

implemnted in 50 provinces in the whole country, however,there only have about 15 provinces possibly make AMTreports at commune and district levels, and PMT reports at

provincial level’ In the consulted provincial groups, most of commune levels impossibly be reported in AMT* When only

about 30% of the provinces possibly be reported in PMT hascaused great difficulty for the central level due to lack ofreported data, and in case of making reports, the quality is nothigh and this leads to the reduction of timeliness and accuracy

in management decisions of the leadership; On report quality:

At all levels, the quality is still not high due to lacking ofaccurate data The qualily of commune report is the weakest It

is recommended to increase human resources for M&E at thecommune level, commit to work long-term and not transferredduring CT, strengthen capacity of data collection and makereports at the commune and district levels, and need areasonable sanction

4 The M&E internal and independent to the CT:

CT is periodically conducted by the managerial level based on CTimplementation schedule and provincial action plan However, there aresome difficulties reduced the effectiveness of internal monitor, thus in theStage III, the program should have a more comprehensive arrangement of human resources, enhance training to increase capacity for local officials,

7

It means that the communes and districts possibly make AMT reports and the provinces could summarize and make

PMT reports.

Ê For example: Bac Kan has 62 communes in 8 districts participating CT13511, however the Committee for Ethnic

Minorities of provinces has not realized the AMF reports of commune level in soft and hard copies cl There has

only a disrit (Nean Son Distriet) in 8 districts which possibly submits reports to the provineal level oweveg dhe

report is still lack of data