Case Study in using the DCED Standard Bamboo processing in Vietnam with the Prosperity Initiative1 8th February 2011 This case shows a programme using several results chains, with different levels of detail, to capture its various interventions Some of these interventions are expected to deliver impact in the short run, while others will take longer; all, however, are linked through the results chains ƚŽƚŚĞƉƌŽŐƌĂŵŵĞ͛Ɛultimate goal of reducing poverty WĂƌƚϭ͗^ƵŵŵĂƌLJŽĨWƌŽƐƉĞƌŝƚLJ/ŶŝƚŝĂƚŝǀĞ͛Ɛ džƉĞƌŝĞŶĐĞǁŝƚŚƚŚĞ^ƚĂŶĚĂƌĚ Description of Programme: The Prosperity Initiative (PI) aims to stimulate investment and encourage policy reforms that are pro-‐poor This case ĨŽĐƵƐĞƐ ŽŶ W/͛Ɛ ŝŶƚĞƌǀĞŶƚŝŽŶƐ ŝŶ sŝĞƚŶĂŵĂŶĚ>ĂŽƐ͛ŝŶĚƵƐƚƌŝĂůďĂŵƐĞĐtors; the work is funded by a consortium of donors, and has an initial budget of USD1.1M The main interventions are: - Tailored services to businesses and investors to adopt the most profitable and highest impact bamboo processing technologies - Policy and advisory services to provincial and national authorities in support of the bamboo industry and wider socio-‐economic development - Facilitating the growth of a market for business advice to bamboo producer communities, particularly around how best to sell high-‐value aged bamboo PI works with lead catalytic firms in the bamboo sector by supporting them with business advisory services to invest and generate new business, to contribute towards growth of the sector The project aims to increase the demand for bamboo, which would increase the price that target groups (small bamboo producers) receive.2 How and Why Prosperity Initiative Became Involved with the Standard: When PI was founded in 2007, the organisation needed to develop a Monitoring & Evaluation framework that suited its market-‐based approach This was not easy In the words of Executive Director :ŽŚŶ DĂƌƐŚ͕ ŵŽƐƚ DŽŶŝƚŽƌŝŶŐ ĂŶĚ ǀĂůƵĂƚŝŽŶ ;DΘ Ϳ ŐƵŝĚĞůŝŶĞƐ ͚ĂƌĞ ŵƵĐŚ ďĞƚƚĞƌ ƐƵŝƚĞĚ ƚŽ ĚĞǀĞůŽƉŵĞŶƚŝŶƚĞƌǀĞŶƚŝŽŶƐŽĨĂŵŽƌĞƉƌĞĚŝĐƚĂďůĞĂŶĚĚŝƌĞĐƚŝŶƚĞƌǀĞŶƚŝŽŶŶĂƚƵƌĞ͛͘ The DCED Standard appeared to offer a viable alternative In 2008, John Marsh and Bob ĂƵůĐŚ͕W/͛ƐƉŽǀĞƌƚLJƐƉĞĐŝĂůŝƐƚ͕ũŽŝŶĞĚƚŚĞdiscussions around the content of the Standard In 2009, PI sent project manager Viet Kim Cuong and a consultant, Philipp Essl, to the ͛Ɛ introductory course Shortly afterwards, Jim Tomecko, an expert on the DCED Standard, We thank John Marsh, Philipp Essl and Bridget Lee Dougherty for all of their assistance in preparing this case study For more information about Prosperity Initiative, visit www.ProsperityInitiative.org visited Hanoi to carry out a mock audit Philipp Essl ŽǀĞƌƐĂǁ ƚŚĞ ĚĞƐŝŐŶ ŽĨ W/͛Ɛ ŶĞǁ DΘ procedures and conducted in-‐house training on the Standard dŚĞ ďĂŵ ƉƌŽũĞĐƚ ǁĂƐ ŝŶŝƚŝĂůůLJ ĚĞƐŝŐŶĞĚ ŝŶ ϮϬϬϳ͕ ƉƌŝŽƌ ƚŽ W/͛Ɛ ĞŶŐĂŐĞŵĞŶƚ ǁŝƚŚ ƚŚĞ Standard In 2009, at the end of its first phase, the bamboo project was re-‐designed in line with a one-‐LJĞĂƌĞdžƚĞŶƐŝŽŶďLJW/͛ƐĚŽŶŽƌƐ͘dŚĞƉƌŽũĞĐƚŚĂƐďĞĞŶƌĞĨŝŶĞĚĂŶĚƐŝŵplified as a result of various challenges and lessons learned in the initial phase In particular, developing results chains made a significant contribution to helping the PI team to develop a tighter focus for the next phase Cost and Benefits of Working towards the Standard John Marsh and his team have prepared a results and project management framework that captures the various components of the DCED Standard, whilst also meeting donor requirements such as log-‐frames Participation has included attending DCED events (approx $7,000), drafting initial results chains and plans with consultancy support ($20,000) and on-‐ going costs, including a pre-‐audit ($20,000) The Standard brings greater clarity and effectiveness to implementation By using results chains to examine the logic of their interventions, the PI team has found it easier to link each activity to development goals, and to decide which ones should be prioritised Results chains have also been useful when developing a new log-‐frame for the bamboo project Furthermore, as a dynamic tool, staff can update their results chains when either market conditions or the project itself changes ĐĐŽƌĚŝŶŐ ƚŽ WŚŝůŝƉƉ ƐƐů͕ Ă ĨŽƌŵĞƌ W/ ĐŽŶƐƵůƚĂŶƚ͕ ͚dŚĞ ^ƚĂŶĚĂƌĚ ŝƐ ŵŽƌĞ Ă ƉƌŽũĞĐƚ ŵĂŶĂŐĞŵĞŶƚƚŽŽůƚŚĂŶĂƌĞƐƵůƚƐŵĞĂƐƵƌĞŵĞŶƚƚŽŽů͛͘&Žƌ:ŽŚŶDĂƌƐŚ͕ the Standard has also allowed PI to better plan for and communicate results, both internally and externally In other words the cost and time spent working towards the DCED Standard in not only spent on results measurement but only about gaining clarity about ŽŶĞ͛ƐǁŽƌŬ Challenges, Responses and General Lessons Learned: ĂƉƚƵƌŝŶŐ ƚŚĞ ďĂŵ ƉƌŽũĞĐƚ͛Ɛ ŽǀĞƌĂůů ŝŵƉĂĐƚ ŝƐ ŶŽƚ ĞĂƐLJ͕ ĞƐƉĞĐŝĂůůLJ ĂƐ ƚŚĞ ƐĐĂůĞ ĂŶĚ ƐĐŽƉĞ ŽĨ W/͛Ɛ ĂĐƚŝǀŝƚŝĞƐ ŚĂǀĞ ĞǀŽůǀĞĚ and expanded over time There have so far been three major challenges In adopting the Standard one key concern was how to capture overall impact for the project, given that the programme was implementing different actions in its first years Initially, the team was unclear about the likely intermediate impacts of its interventions In the redesign, PI had to spend a lot of time thinking about how to capture its results at different stages, as different activities would have different timelines for impact: some longer term and some shorter term The process of drawing results chains has actually made staff more aware of these differences In the end, after experimenting with a few variations of results chain, it was decided that PI would consider adopting a hierarchy of results chains to capture all actions that contribute ƚŽ W/͛Ɛ ƐĞĐƚŽƌ ůĞǀĞů ŐŽĂůƐ͘ YƵŽƚŝŶŐ ĨƌŽŵ W/͛Ɛ ƌĞƐƵůƚƐ ŵĞĂƐƵƌĞŵĞŶƚ ĨƌĂŵĞǁŽƌŬ͕ ͚&Žƌ ĞĂĐŚ sector or sub-‐sector that PI engages in, a sector level results chain is drafted During the project development phase (business case) PI drafts an overall project results chain that summarizes the overall project strategy, as well as the various project components that the project contains Based on the project results chain, project managers develop results chains for each project component, which provide more details to the project strategy and underlying impact logic Depending on the complexity and scope of project components ʹ as well as allocation of management responsibilities ʹ more than one project component can be captured in one ƐŝŶŐůĞĐŽŵƉŽŶĞŶƚƌĞƐƵůƚƐĐŚĂŝŶ͛͘ dŚĞ ƐĞĐŽŶĚ ŵĂũŽƌ ĐŚĂůůĞŶŐĞ ǁĂƐ ƚŽ ĐĂƉƚƵƌĞ ŚŽǁ W/͛Ɛ ƐĞƉĂƌĂƚĞ ŝŶƚĞƌǀĞŶƚŝŽŶƐ ĂŶĚ ƐĞƉĂƌĂƚĞ strategies all contribute towards reaching the same desired goal of poverty reduction Here again using the hierarchy of results chains helped staff to capture all the actions that ĐŽŶƚƌŝďƵƚĞ ƚŽ ƚŚĞ ƉƌŽŐƌĂŵŵĞƐ͛ ƐĞĐƚŽƌ-‐level goals This hierarchy of results chains is ĞdžƉůĂŝŶĞĚŝŶĚĞƚĂŝůŝŶƚŚĞ͚ƌƚŝĐƵůĂƚŝŶŐƚŚĞZĞƐƵůƚƐŚĂŝŶ͛ƐĞĐƚŝŽŶ͘ Finally, PI conducted industry level baseline surveys at the outset, to measure household level impact data, in an attempt to establish benchmarks in line with the longer-‐term impact objectives These surveys were not linked to specific, shorter-‐term project interventions When programme staff drafted results chains for individual shorter-‐term interventions, it was not clear how each of these interventions translated into impact It became clear that the general household surveys, while interesting, did not allow PI to attribute impacts to interventions WĂƌƚϮ͗WƌŽƐƉĞƌŝƚLJ/ŶŝƚŝĂƚŝǀĞ͛ƐtŽƌŬƚŽǁĂƌĚƐƚŚĞ ^ƚĂŶĚĂƌĚ͗ Articulating the Results Chain: PI has devised the structure for all its results chains and is finalising the results chains for all its interventions The main reason for the delay in finalising results chains is that the project has been restructured recently under a one-‐year extension, which started in July 2010 The focus for the project documentation with donors was the logframe and associated indicators, which has been completed and agreed Draft results chains were developed during this process, and fine-‐tuning is taking place to finalise the results chains and incorporate other elements of the DCED Standard In fact, drafting the results chain has been helpful in this re-‐strategising process and in developing the logframe itself, because senior project staff have used it in their key meetings to think through how their activities can be used to generate highest impact, and how to select the most appropriate indicators PI intends to create results chains at three or four different levels The first is a sector-‐level results chain, incorporating systemic change Secondly, project-‐level results chains are drafted These show how the different components of the project combine for overall impact The third set of results chains are at the component-‐level Component-‐level results chains show how different groups of activities within one programme component together lead to higher-‐level change As component results chains are drafted to some extent at a conceptual level, more detailed intervention results chains are sometimes drafted in addition Project and intervention managers often find intervention results chains the most ƵƐĞĨƵů͘dŚĞĚŝĂŐƌĂŵďĞůŽǁƐŚŽǁƐW/͛ƐŚŝĞƌĂƌĐŚLJŽĨƌĞƐƵůƚƐĐŚĂŝŶƐĨŽƌƚŚĞŝŶĚƵƐƚƌŝĂůďĂŵ sector in Vietnam and Laos Annex gives examples of project-‐level and component-‐level results chains Already in the project design phase, each staff member is involved in drafting their respective results chain This ensures that staff can use the results chains as a practical instrument to guide their work Results chains have been updated and improved over time For example, PI originally projected the amount that target firms would invest, without linking this explicitly to its outputs (e.g deals made, services provided) Closing this gap has helped to clarify the ƉƌŽũĞĐƚ͛ƐůŽŐŝĐ͘ Defining Indicators of Change: PI aims to integrate the indicators used in its log-‐ ĨƌĂŵĞŝŶƚŽŝƚƐƌĞƐƵůƚƐĐŚĂŝŶ͘dŚŝƐǁŝůůĚĞĞƉĞŶƚŚĞĞdžƉůĂŶĂƚŝŽŶŽĨW/͛ƐŝŶƚĞƌǀĞŶƚŝŽŶƐƚƌĂƚĞŐLJďLJ providing additional information on indicators, means of verification, assumptions (risks) and required resources This will also help them to measure what is important and to avoid duplication of work PI aims to measure the long-‐term impact of its various activities on poverty reduction This is difficult to with a high degree of scientific rigour, however, because many factors contribute to poverty As suggested in the Standard, PI also measures how many people benefit from its interventions, plus the additional jobs and additional income going to poor people as a result of its interventions The PI team use these indicators to establish whether their work is achieving its goals PI develops projections for the following standardised impact indicators: WƌŽĚƵĐĞƌƐ͛ŝŶĐŽŵĞĨƌŽŵƐĂůĞƐ dĂƌŐĞƚŐƌŽƵƉƐ͛ŝŶĐŽŵĞĨƌŽŵƐĂůĂƌŝĞƐ Total number of new waged jobs created (Full-‐time Equivalent) % of new jobs going to female workers Total Income: the sum of all household income from sales and wages Measuring Changes in Indicators: Although PI is not yet at a stage where it measures changes in indicators, PI has designed a method for measuring changes in indicators in its Results and Project Management Framework This methodology aims to conform with good research practices, as outlined in the Standard It includes quick household surveys, semi-‐ structured interviews and focus groups, as well as available secondary data on poverty PI has a poverty scorecard methodology, which allows for the classification of households as poor, near-‐poor, or non-‐poor, based on a number of non-‐monetary poverty indicators that are covered in a brief interview with a household member PI piloted this method in its first firm-‐based impact assessment, carried out in early 2009 It will next be applied to partner firms which make investments, both to capture baselines and to measure the impact of the investments that these firms make Results Chain Estimating Attributable Changes: PI has Attribution Methods Level of change prepared a draft attribution measurement plan, but has not yet tested it The plan is illustrated Case studies Poverty impact Quasi-‐experimental (household in the diagram As more programmes apply the income) surveys Standard, John Marsh hopes that there will be more examples of how programmes have dealt Case studies with the issue of attribution Outcome ^ĞĐƚŽƌƐƚĂŬĞŚŽůĚĞƌƐ͛ ŽƉŝŶŝŽŶƐ (Enterprise/government Quasi-‐experimental surveys & sector level) Trend analysis Capturing Wider Changes in the System or Market: ^LJƐƚĞŵŝĐ ĐŚĂŶŐĞ ŝƐ ŝŶĐůƵĚĞĚ ŝŶ W/͛Ɛ sector-‐level results chains The project Case studies Output ^ĞĐƚŽƌƐƚĂŬĞŚŽůĚĞƌƐ͛ ŽƉŝŶŝŽŶƐ considers, for example, how increased demand ĨŽƌ ďĂŵ ŐĞŶĞƌĂƚĞĚ ďLJ W/͛Ɛ ŝŶƚĞƌǀĞŶƚŝŽŶƐ would also increase the price that target groups receive for their bamboo In practice, it has proven difficult to establish precise causal ƌĞůĂƚŝŽŶƐŚŝƉƐďĞƚǁĞĞŶW/͛ƐŝŶƚĞƌǀĞŶƚŝŽŶƐĂŶĚƐLJƐƚĞŵŝĐŝŵƉĂĐƚ͘dŚĞƐŝnjĞŽĨW/͛ƐďƵĚŐĞƚĂŶĚƚŚĞ likelihood that the project will end in mid-‐2011 mean that PI is unlikely to measure systemic change in detail Tracking Programme Cost: W/͛Ɛ ĂĐĐŽƵŶƚŝŶŐ ƐLJƐƚĞŵ͕ ŵĂŶĂŐĞĚ ďLJ ƚŚĞ ŽƉĞƌĂƚŝŽŶƐ ŵĂŶĂŐĞƌ͕ŵĞĂƐƵƌĞƐĂůůƚŚĞƉƌŽŐƌĂŵŵĞ͛ƐĐŽƐƚƐĂŶŶƵĂůůLJ͘ Reporting Results: PI has developed a corporate reporting system to measure its effectiveness This system also measures compliance with the DCED Standard and with donor requirements The system includes time reporting for staff, and regular strategic review meetings, once or twice per year Annex 1: Sector-‐level and Component-‐level Results Chain for PI͛s work in industrial bamboo Sector-‐level: Component-‐level, relating to Business and Investment ... ƚŽ W/͛Ɛ ĞŶŐĂŐĞŵĞŶƚ ǁŝƚŚ ƚŚĞ Standard In 2009, at the end of its first phase, the ? ?bamboo project was re-‐designed in line with a one-‐LJĞĂƌĞdžƚĞŶƐŝŽŶďLJW/͛ƐĚŽŶŽƌƐ͘dŚĞƉƌŽũĞĐƚŚĂƐďĞĞŶƌĞĨŝŶĞĚĂŶĚƐŝŵplified... chains have also been useful when developing a new log-‐frame for the bamboo project Furthermore, as a dynamic tool, staff can update their results... ŝŶƚĞƌǀĞŶƚŝŽŶƐ would also increase the price that target groups receive for their bamboo In practice, it has proven difficult to establish precise causal ƌĞůĂƚŝŽŶƐŚŝƉƐďĞƚǁĞĞŶW/͛ƐŝŶƚĞƌǀĞŶƚŝŽŶƐĂŶĚƐLJƐƚĞŵŝĐŝŵƉĂĐƚ͘dŚĞƐŝnjĞŽĨW/͛ƐďƵĚŐĞƚĂŶĚƚŚĞ