Case Study in using the DCED Standard Bamboo processing in Vietnam with the Prosperity Initiative1 8th February 2011 This case shows a programme using several results chains, with different levels of detail, to capture its various interventions Some of these interventions are expected to deliver impact in the short run, while others will take longer; all, however, are linked through the results chains to the programme’s ultimate goal of reducing poverty Part 1: Summary of Prosperity Initiative’s Experience with the Standard Description of Programme: The Prosperity Initiative (PI) aims to stimulate investment and encourage policy reforms that are pro-poor This case focuses on PI’s interventions in Vietnam and Laos’ industrial bamboo sectors; the work is funded by a consortium of donors, and has an initial budget of USD1.1M The main interventions are: - Tailored services to businesses and investors to adopt the most profitable and highest impact bamboo processing technologies Policy and advisory services to provincial and national authorities in support of the bamboo industry and wider socio-economic development Facilitating the growth of a market for business advice to bamboo producer communities, particularly around how best to sell high-value aged bamboo PI works with lead catalytic firms in the bamboo sector by supporting them with business advisory services to invest and generate new business, to contribute towards growth of the sector The project aims to increase the demand for bamboo, which would increase the price that target groups (small bamboo producers) receive.2 How and Why Prosperity Initiative Became Involved with the Standard: When PI was founded in 2007, the organisation needed to develop a Monitoring & Evaluation framework that suited its market-based approach This was not easy In the words of Executive Director John Marsh, most Monitoring and Evaluation (M&E) guidelines ‘are much better suited to development interventions of a more predictable and direct intervention nature.’ The DCED Standard appeared to offer a viable alternative In 2008, John Marsh and Bob Baulch, PI’s poverty specialist, joined the discussions around the content of the Standard In 2009, PI sent project manager Viet Kim Cuong and a consultant, Philipp Essl, to the DCED’s introductory course Shortly afterwards, Jim Tomecko, an expert on the DCED Standard, We thank John Marsh, Philipp Essl and Bridget Lee Dougherty for all of their assistance in preparing this case study For more information about Prosperity Initiative, visit www.ProsperityInitiative.org visited Hanoi to carry out a mock audit Philipp Essl oversaw the design of PI’s new M&E procedures and conducted in-house training on the Standard The bamboo project was initially designed in 2007, prior to PI’s engagement with the Standard In 2009, at the end of its first phase, the bamboo project was re-designed in line with a one-year extension by PI’s donors The project has been refined and simplified as a result of various challenges and lessons learned in the initial phase In particular, developing results chains made a significant contribution to helping the PI team to develop a tighter focus for the next phase Cost and Benefits of Working towards the Standard John Marsh and his team have prepared a results and project management framework that captures the various components of the DCED Standard, whilst also meeting donor requirements such as log-frames Participation has included attending DCED events (approx $7,000), drafting initial results chains and plans with consultancy support ($20,000) and ongoing costs, including a pre-audit ($20,000) The Standard brings greater clarity and effectiveness to implementation By using results chains to examine the logic of their interventions, the PI team has found it easier to link each activity to development goals, and to decide which ones should be prioritised Results chains have also been useful when developing a new log-frame for the bamboo project Furthermore, as a dynamic tool, staff can update their results chains when either market conditions or the project itself changes According to Philipp Essl, a former PI consultant, ‘The Standard is more a project management tool than a results measurement tool.’ For John Marsh, the Standard has also allowed PI to better plan for and communicate results, both internally and externally In other words the cost and time spent working towards the DCED Standard in not only spent on results measurement but only about gaining clarity about one’s work Challenges, Responses and General Lessons Learned: Capturing the bamboo project’s overall impact is not easy, especially as the scale and scope of PI’s activities have evolved and expanded over time There have so far been three major challenges In adopting the Standard one key concern was how to capture overall impact for the project, given that the programme was implementing different actions in its first years Initially, the team was unclear about the likely intermediate impacts of its interventions In the redesign, PI had to spend a lot of time thinking about how to capture its results at different stages, as different activities would have different timelines for impact: some longer term and some shorter term The process of drawing results chains has actually made staff more aware of these differences In the end, after experimenting with a few variations of results chain, it was decided that PI would consider adopting a hierarchy of results chains to capture all actions that contribute to PI’s sector level goals Quoting from PI’s results measurement framework, ‘For each sector or sub-sector that PI engages in, a sector level results chain is drafted During the project development phase (business case) PI drafts an overall project results chain that summarizes the overall project strategy, as well as the various project components that the project contains Based on the project results chain, project managers develop results chains for each project component, which provide more details to the project strategy and underlying impact logic Depending on the complexity and scope of project components – as well as allocation of management responsibilities – more than one project component can be captured in one single component results chain.’ The second major challenge was to capture how PI’s separate interventions and separate strategies all contribute towards reaching the same desired goal of poverty reduction Here again using the hierarchy of results chains helped staff to capture all the actions that contribute to the programmes’ sector-level goals This hierarchy of results chains is explained in detail in the ‘Articulating the Results Chain’ section Finally, PI conducted industry level baseline surveys at the outset, to measure household level impact data, in an attempt to establish benchmarks in line with the longer-term impact objectives These surveys were not linked to specific, shorter-term project interventions When programme staff drafted results chains for individual shorter-term interventions, it was not clear how each of these interventions translated into impact It became clear that the general household surveys, while interesting, did not allow PI to attribute impacts to interventions Part 2: Prosperity Initiative’s Work towards the DCED Standard: Articulating the Results Chain: PI has devised the structure for all its results chains and is finalising the results chains for all its interventions The main reason for the delay in finalising results chains is that the project has been restructured recently under a one-year extension, which started in July 2010 The focus for the project documentation with donors was the logframe and associated indicators, which has been completed and agreed Draft results chains were developed during this process, and fine-tuning is taking place to finalise the results chains and incorporate other elements of the DCED Standard In fact, drafting the results chain has been helpful in this re-strategising process and in developing the logframe itself, because senior project staff have used it in their key meetings to think through how their activities can be used to generate highest impact, and how to select the most appropriate indicators PI intends to create results chains at three or four different levels The first is a sector-level results chain, incorporating systemic change Secondly, project-level results chains are drafted These show how the different components of the project combine for overall impact The third set of results chains are at the component-level Component-level results chains show how different groups of activities within one programme component together lead to higher-level change As component results chains are drafted to some extent at a conceptual level, more detailed intervention results chains are sometimes drafted in addition Project and intervention managers often find intervention results chains the most useful The diagram below shows PI’s hierarchy of results chains for the industrial bamboo sector in Vietnam and Laos Annex gives examples of project-level and component-level results chains Already in the project design phase, each staff member is involved in drafting their respective results chain This ensures that staff can use the results chains as a practical instrument to guide their work Results chains have been updated and improved over time For example, PI originally projected the amount that target firms would invest, without linking this explicitly to its outputs (e.g deals made, services provided) Closing this gap has helped to clarify the project’s logic Defining Indicators of Change: PI aims to integrate the indicators used in its logframe into its results chain This will deepen the explanation of PI’s intervention strategy by providing additional information on indicators, means of verification, assumptions (risks) and required resources This will also help them to measure what is important and to avoid duplication of work PI aims to measure the long-term impact of its various activities on poverty reduction This is difficult to with a high degree of scientific rigour, however, because many factors contribute to poverty As suggested in the Standard, PI also measures how many people benefit from its interventions, plus the additional jobs and additional income going to poor people as a result of its interventions The PI team use these indicators to establish whether their work is achieving its goals PI develops projections for the following standardised impact indicators: Producers’ income from sales Target groups’ income from salaries Total number of new waged jobs created (Full-time Equivalent) % of new jobs going to female workers Total Income: the sum of all household income from sales and wages Measuring Changes in Indicators: Although PI is not yet at a stage where it measures changes in indicators, PI has designed a method for measuring changes in indicators in its Results and Project Management Framework This methodology aims to conform with good research practices, as outlined in the Standard It includes quick household surveys, semi3 structured interviews and focus groups, as well as available secondary data on poverty PI has a poverty scorecard methodology, which allows for the classification of households as poor, near-poor, or non-poor, based on a number of non-monetary poverty indicators that are covered in a brief interview with a household member PI piloted this method in its first firm-based impact assessment, carried out in early 2009 It will next be applied to partner firms which make investments, both to capture baselines and to measure the impact of the investments that these firms make Estimating Attributable Changes: PI has prepared a draft attribution measurement plan, but has not yet tested it The plan is illustrated in the diagram As more programmes apply the Standard, John Marsh hopes that there will be more examples of how programmes have dealt with the issue of attribution Results Chain Level of change Attribution Methods • Case studies Poverty impact • Quasi-experimental (household income) surveys • Case studies Outcome (Enterprise/government & sector level) • Sector stakeholders’ opinions • Quasi-experimental surveys • Trend analysis Capturing Wider Changes in the System or Market: Systemic change is included in PI’s sector-level results chains The project •Case studies Output • Sector stakeholders’ opinions considers, for example, how increased demand for bamboo generated by PI’s interventions would also increase the price that target groups receive for their bamboo In practice, it has proven difficult to establish precise causal relationships between PI’s interventions and systemic impact The size of PI’s budget and the likelihood that the project will end in mid-2011 mean that PI is unlikely to measure systemic change in detail Tracking Programme Cost: PI’s accounting system, managed by the operations manager, measures all the programme’s costs annually Reporting Results: PI has developed a corporate reporting system to measure its effectiveness This system also measures compliance with the DCED Standard and with donor requirements The system includes time reporting for staff, and regular strategic review meetings, once or twice per year Annex 1: Sector-level and Component-level Results Chain for PI’s work in industrial bamboo Sector-level: Component-level, relating to Business and Investment ... chain, it was decided that PI would consider adopting a hierarchy of results chains to capture all actions that contribute to PI? ??s sector level goals Quoting from PI? ??s results measurement framework,... precise causal relationships between PI? ??s interventions and systemic impact The size of PI? ??s budget and the likelihood that the project will end in mid-2011 mean that PI is unlikely to measure systemic... initial phase In particular, developing results chains made a significant contribution to helping the PI team to develop a tighter focus for the next phase Cost and Benefits of Working towards