Monitoring and Evaluating Programs and Projects Using GIS

Monitoring and Evaluating Programs and Projects Using GIS How to use GIS to support M&E systems for sustainable programs and projects Robert Catherman Director of Safe Water Development MEDRIX™ June 2013 Edition E.2.12 Monitoring and Evaluating Programs and Projects Using GIS How to use GIS to support M&E systems for sustainable programs and projects Robert Catherman Director of Safe Water Development - MEDRIX™ June 2013 Edition E.2.12 Preface The purpose of this handbook is to provide a standardized source of information for developing and operating a Monitoring and Evaluation System using Geographic Information System outputs to display program and project outcomes This handbook includes: • Part One: General understanding of the principles of monitoring and evaluation (M&E) systems • Part Two: Instructions for creating GIS maps supporting an M&E system • Part Three: Instructions for preparing M&E reports • Annexes: Information and forms referenced in the main sections of the handbook MEDRIX declares the contents of this handbook to be open-source, available without charge Copies of this handbook are available for downloading from the MEDRIX website Users assume full responsibility for the outcomes of constructing and operating GISM&E systems using these instructions A handbook edition number identifies the most up-to-date information The letter before the edition number signifies the language of the document The first number identifies the version; the number following the period indicates the level of minor revision including corrections and updates For example, the edition number “E.2.3” indicates this is the second English version released for publication with three levels of minor revisions The edition number “V.2.2” indicates this is the Vietnamese translation of the English version E.2.2 This handbook is designed to permit you, the user, to update pictures and information to reflect the realities in your part of the world Feel free to substitute your own digital pictures to make the information more relevant to readers in your region Your feedback about this handbook is welcome Improving the quality of open source documents is the responsibility of all who use the information Submit suggestions and recommended changes for this handbook to: Director of Safe Water Development MEDRIX PO Box 178 Redmond, WA 98073 USA office@medrix.org Include the edition number of this handbook in any communications This work is licensed under the Creative Commons Attribution-Share Alike 2.5 License To view a copy of this license, visit http://creativecommons.org/licenses/by-sa/2.5/ or send a letter to Creative Commons, 543 Howard Street, 5th Floor, San Francisco, California, 94105, USA You are free: • • to Share to copy, distribute, display, and perform the work to Remix to make derivative works Under the following conditions: • • • Attribution You must attribute the work to MEDRIX as the original creator and refer users to the MEDRIX website at www.medrix.org Share Alike If you alter, transform, or build upon this work, you may distribute the resulting work only under a license identical to this one For any reuse or distribution, you must make clear to others the license terms of this work Any of these conditions can be waived if you get permission from the copyright holder Disclaimer This publication is based on the collective experience of MEDRIX in monitoring and evaluating the software, methods and procedures described herein While every effort has been made to ensure the accuracy of this work, any judgments as to the suitability of information for the reader’s purposes are the reader’s responsibility MEDRIX does not extend any warranties, and assumes no responsibility, for the suitability of this information or the consequences of its use Acknowledgments Dr Joe Hannah, of faculty of Geography Department at University of Washington, my instructor in GEOG 360 who taught me principles of map making and how to effectively use GIS technology as well as shared frequent consultations over coffee during the development of this project Student Team A from University of Washington class GEOG 335, Service Learning Project, helped with the idea formulation and rough drafts of the first maps: students Geoffrey Morgan, Rahima Niyazova and Robert Sepler Student Team B from University of Washington class GEOG 469, Senior project, converted the ArcGIS instructions into instructions for using QGIS open source software: students Alyssa Knight, Duc Chi Du, Mike Ridgely, Pao Thao, and Thanh Nguyen And thanks to Dr Sarah Elwood and graduate student Joe Eckert for overseeing the 2011 Senior Project work Staff of Preventative Medicine department of Health Services, Thua Thien – Hue province, Vietnam, reviewed and critiqued the initial drafts and provided valuable input regarding data collection methods Special thanks to Dr Tu, Vice Director, for his enthusiastic support of the idea Mrs Phạm Thị Thu Hà of MEDRIX staff in Hanoi, Vietnam, who prepared and refined the data collection forms, translating the English into Vietnamese Ms Nguyễn Thị Tằm of IREB, Hue, Vietnam, provided translation help early in the project and coauthored an article for a joint Vietnamese-Korean conference held in Hue, Vietnam when these ideas were first aired to the general public Ms Hoàng Ngọc Tường Vy, software engineer in Hue, Vietnam, who had no prior knowledge of GIS, carefully tested the accuracy and readability of the English instructions for creating the QGIS maps, corrected errors and made helpful suggestions for revisions Dr Nguyễn Thụy Thị Hồng, while a PhD candidate at Washington State University, gave the instructions a thorough test, corrected some errors, and made numerous insightful suggestions for improving readability She also began translation of the instructions from English into Vietnamese Mr Dewey Calfee, reviewer extraordinaire, painstakingly followed these instructions to the letter and uncovered omitted steps, clarified imprecise wording and rewrote the introduction, all after starting from ground zero in learning to use GIS Student Team C from University of Washington class of 2012 GEOG 469, capstone project, investigated the capabilities and limitations of converting from desktop-based GIS software to on-line cloud-based GIS software - students Thong Nguyen and Thảo Vũ Thanks to Dr Timothy Nyerges for overseeing the Senior Project work Ms Thảo Vũ, BA from UW in 2012, who worked as a summer intern at MEDRIX and completed the mapping case study included in Chapter 17 Student Team D from University of Washington class of 2013 GEOG 469, capstone project, revised formatting to improve readability and added new material - students Anthony Caratao, Dan Kim and Matt Peterson Thanks to Dr Timothy Nyerges for overseeing their work Contents Chapter 1: Introduction Part 1: General Principles of M&E Chapter 2: About Monitoring and Evaluation Systems .3 Chapter Integrating M&E and Geographic Information Systems .5 Chapter 4: Deciding if M&E-GIS is Appropriate for Your Situation Chapter 5: Developing an M&E-GIS System for your Situation Part 2: Creating Maps Supporting M&E Systems 13 13 Chapter 6: Selecting GIS Software 14 Chapter 7: GIS Terminology 16 Chapter 8: Locating GIS Maps 19 Chapter 9: Creating a Reference Map .21 Chapter 10: Defining the Data Requirements and Key Indicators 36 Chapter 11: Defining the Data Collection Process 38 Chapter 12: Designing and Creating the Data Input Table 43 Chapter 13: Training the Data Input Staff and Updating Field Reports 47 Chapter 14: Determining Latitude and Longitude Coordinates 49 Chapter 15: Creating the Thematic Map 54 Part 3: Preparing M&E Reports .84 Chapter 16: Preparing the Working Project Reporting Maps 85 Chapter 17: Case Study of M&E for a Medical Education Project .92 Chapter 18: Continuation of GIS Use .110 Part 4: ANNEX 111 Annex A: Installing Quantum GIS on Mac 112 Annex B: References 114 Annex C: Intermediate Outcome Measurement .115 Chapter 1: Introduction Chapter Objectives • Understanding the approach of this Handbook • Introducing Monitoring and Evaluation using GIS Maps are familiar to most of us – maps are like good friends that help us find our way and orient us to our surroundings On maps, we can easily see that “this is here and that is there” and we can gain a sense of the distance and direction between “here and there” In this handbook, we will challenge you to take a slightly different view of maps We will be asking maps to help us think – think about items that have a spatial component as one of their many attributes Specifically, we want to think about how we can monitor and evaluate the activities of a specific project so we can document positive outcomes, increase sustainability and develop effective solutions to problems we discover For example, we want to think about finding some effective methods of keeping the water treatment systems in some rural schools in good working order, repairing the systems that are broken and planning preventative maintenance to complete in the future In this handbook, we will examine some ways we can use maps to define, measure and display project outcomes To reach our destination, we’ll explore the technical topic of Monitoring and Evaluation (M&E) methods that are used worldwide in many development projects And we’ll use the technical tools of Geographic Information Systems (GIS) to help us reach our goals This handbook is not intended to be a comprehensive text on either M&E or GIS Excellent sources of information and instruction on both topics are referenced in each section of this handbook The intent of this handbook is to provide a step-by-step guide to developing a combined M&E-GIS system and to emphasis some key points that apply to reaching this goal Some of the key points come from lessons learned from experience with this methodology and some points come from taking advantage of specific features of M&E and GIS that support the goal we are working toward Let’s begin our journey Edition E.2.12 Monitoring and Evaluating Programs and Projects Using GIS Part 1: General Principles of M&E Edition E.2.12 Chapter 2: About Monitoring and Evaluation Systems Chapter Objectives • Learning about Monitoring and Evaluation and its benefits • Exploring the applications of M&E What is Monitoring and Evaluation? Monitoring and Evaluation (M&E) is a powerful management tool that can be used to improve the way governments and organizations achieve results Just as governments and organizations need financial, human resource, and audit systems, they also need good performance feedback systems Monitoring is the process of data collection and measurement of progress toward program objectives Evaluation is the periodic assessment of changes in desired outcomes that can be attributable to a program’s interventions Recent history of M&E Recent years have seen an evolution in the field of monitoring and evaluation These changes have involved a movement away from traditional implementation-based approaches and toward new results-based approaches Governments and organizations may successfully implement programs or policies but have they actually produced the intended results? Have government and organizations delivered on promises made to their stakeholders? For example, it is not enough to simply implement health programs and assume that successful implementation is equivalent to actual improvements in public health It is important to examine if outcomes and impacts were achieved The introduction of a results-based M&E system takes decision makers onestep further in assessing if and how goals are being achieved over time These systems help to answer the all-important “so what” questions and respond to stakeholder’s growing demands for results How does M&E work? M&E is important tool targeting two key elements to any development project – effectiveness and sustainability In the ideal situation, M&E should span the life cycle of the project providing a continuing stream of data and feedback In the initial stages of the project, M&E can aid in developing and clarifying the goals and objectives Once the project reaches operational status, M&E can promote greater transparency and Edition E.2.12 Choose Size = 1.0 mm Select Advanced tab Choose Placement over centroid Choose Label Distance = mm OK Remove the “X” from the layer Populations Step 7: Save your project file Select menu option File > Save Project Step 8: Display population data as a category map using district population data Right click Districts layer > Properties Select Style tab Choose Graduated in drop-down box at upper left Choose Column box = Pop% Choose Classes = Choose Mode = Natural Breaks (Jenks) Select Classify button Choose Color Ramp = new color ramp Choose Color Ramp Type = ColorBrewer > OK Choose Scheme Name = choose left light color and right dark color > OK Type name for your color ramp > OK Double click on each entry in the Range and Label columns and change to the numbers in the following figure Apply > OK NOTE: Most map-readers intuitively understand that dark colors represent higher values and light colors represent lower values When choosing your colors, remember that most map-readers will generally understand a dark-to-light color scheme Step 9: Add staff training statistics as a text layer 103 Edition E.2.12 In Excel, enter the above data to create an Excel spreadsheet Select File > Save As > Staff in folder Project-Category In the Save as type box, choose Unicode Text Save > OK > yes File > Exit > no Return to your open QGIS program Select menu option Layer On the drop down sub-menu select Add Delimited Text Layer Browse for the file Staff.txt > Open Check Selected delimiters circle Check Comma and Tab boxes Choose X Field = Lon Choose Y Field = Lat OK Select Coordinate Reference System = WGS 84 > OK Right click Staff layer > Properties Select Style tab Choose Graduated in drop-down box at upper left Choose Column box = (whichever staff position you desire to map) Choose Classes = Choose Mode = Natural Breaks (Jenks) Select Classify button Choose Color Ramp = yellow Double click on first entry in Range column and change first range to 0-0 Change second range to 0.0001-10, etc 104 Edition E.2.12 Double click on each Symbol and increase its font size by over the previous symbol If the range value of zero has special meaning, change the Symbol and its color to attract attention Step 10: Save your project file Select menu option File > Save Project Step 11: Add legends, titles, and sources of data Since two sets of data are display, two legends are needed If you need a refresher on how to create a legend, refer to Step 4.6 of Exercise of Chapter 15 A useful feature for formatting the legend titles is the “wrap” feature found in Print Composer by left clicking in the Legend box and scrolling to the bottom of the General tab Once there you can define a “wrap character” and insert that character in the legend title to force the text following the wrap character to move onto the next line In your map title, a common terminology to use begins with “Distribution of ….” Your map should resemble the following map although color choices may vary: 105 Edition E.2.12 Select menu option File > Save Project REMEMBER: Take a screenshot of your progress and/or report any issues and email it to your instructor We encourage you to communicate any problems, issues, comments, or concerns you have encountered; all feedback we receive is utilized in further improvements of the handbook 106 Edition E.2.12 Exercise 11 (optional): Moving an Existing M&E Mapping System to a Different Computer Time to complete: 30 minutes Exercise Objectives • Learning how to transfer GIS software and data to an alternate computer If you want to install an existing mapping system on a different computer, follow these steps: Step 1: Copy files to be moved to new computer to a thumb drive or CD • On the original computer  Find the folder and sub-folders where QGIS data is stored  Insert a thumb drive or a CD to be written to  Copy folder and subfolders to thumb drive or CD  Remove thumb drive or CD Step 2: Install QGIS program on the new computer Follow the instructions in Step of Exercise Step 3: Copy folders and subfolders from thumb drive or CD to new computer Copy the main folder of QGIS data files from your thumb drive or CD Paste copied folder into a folder on your new computer Step 4: Reset folder pointers Start QGIS Select menu option File > Open project Browse to folder containing your project map on the new computer Find a QGIS map file created on the old computer Open You will need to “repoint” the layer files to the different folder on this computer You will work with the files in the column titled “New file”; the file names should be in red meaning they could not be found; you must “repoint” to the current locations on the new computer but only this one time For example, double click on the “UV_Sites” file and browse for the file with the 107 Edition E.2.12 same name in the new folder you created in Step After you find and select the moved file, the text should turn from red to green Remember the number of the “adm?” file > double click on the first “adm?” file and browse for the file with the same name and matching number in the new folder you created in Step After you find and select the moved file, the text should turn from red to green Repeat the same process for the next “adm?” files Apply > OK Select menu option File > Save Project As browse to the folder where you want to save the QGIS project file and name file so you can find it again Select menu option File > Exit Step 5: Finding styles and templates for the first time Start QGIS Select menu option File > Open project > your saved file name from Step Open Refer to Exercise to prepare a map to print You will need to browse for styles and templates the first time you try to load them Templates have the file type of “qpt” and Styles have file type of “qml” When you are finished Select menu option File > Save Project Select menu option File > Exit In the future QGIS will remember the folder where the Template and Style files are saved and you will not need to repeat this step Step 6: Test your new configuration Start QGIS Select menu option File > Open project 108 Edition E.2.12 Browse to your saved file name from Step Open The map should appear without any further actions Select menu option File > Exit REMEMBER: Take a screenshot of your progress and/or report any issues and email it to your instructor We encourage you to communicate any problems, issues, comments, or concerns you have encountered; all feedback we receive is utilized in further improvements of the handbook 109 Edition E.2.12 Chapter 18: Continuation of GIS Use Chapter Objectives • Harnessing the potential of GIS The Diversity of GIS Congratulations on designing your Monitoring and Evaluation system and creating your first maps using GIS technology! Now that you’ve made it this far, you can utilize the valuable skills you have learned from this handbook and apply them to real world situations The contents of this handbook will assist you in your experimental design and application of Monitoring and Evaluation using GIS to identify and map problems selected by you GIS software has incredible capabilities and can assist in the modeling of a wide array of real world issues found in professional settings and industries such as, business, law enforcement, government, healthcare, education, transportation, etc An example of using a Monitoring and Evaluation system in conjunction with GIS could be to track the spread of disease just like John Snow's 1855 map of the cholera outbreak in London Ideally, one could record the coordinates of known cases of a disease and then map them out to see if there is any spatial pattern to that data Perhaps all the cases are close to a certain river or spatially concentrated around a certain area The data could also be overlaid with other types of data, such as population demographics, pollution sources, or locations of forests By doing this it is possible to discern relationships between variables, and therefore come a step closer to monitoring and evaluating the spread and cause of the disease, ultimately leading to prevention or cures Another example of using M&E combined with GIS could be to study the effects and management of water resources As China’s construction of dams and navigation channels along the Mekong River continues, the inevitable changes to the river’s natural ecosystem and will drastically effect the livelihood of millions of people living in Burma, Thailand, Laos, Cambodia and Vietnam In this situation, M&E combined with GIS can be used to track and analyze the repercussions of dam construction such as, reduced fish population, loss of biodiversity, shoreline erosion, family relocation, and more Before GIS, the technology to perform this type of analysis wasn’t accessible to the general public, and what was available was limited by difficult user interface as well as a very high price tag Now that you have completed MEDRIX’s Monitoring and Evaluating Programs and Projects Using GIS handbook, you are equipped with the skills necessary to be able to perform your own GIS analysis, according to your specific research needs Good luck on your quest for geographic knowledge! 110 Edition E.2.12 Monitoring and Evaluating Programs and Projects Using GIS Part 4: ANNEX 111 Edition E.2.12 Annex A: Installing Quantum GIS on Mac Mac - *requires the installation of the GDAL and GSL frameworks before installing the QGIS package* Go to http://hub.qgis.org/projects/quantum-gis/wiki/Download Scroll down to the section labeled “4 MacOS X” Click on “KyngChaos QGIS download page” under the section named 4.1 Release First download and install the GDAL Complete 1.9 framework package α Click on on “GDAL Complete 1.9 framework package” under “Requirements” to be directed to a new page β Click on “GDAL 1.9 Complete [33.6 MiB] 2013-5-1” to begin downloading the dmg file to your mac χ In the folder where you downloaded it, select and open the GDAL_Complete-1.9.dmg file next δ Double click on the “GDAL Complete.pkg” icon to begin installation ε Follow the instructions on the installer wizard to complete installation Next download and install the GSL framework package also found on the “KyngChaos Qgis download page”: α Click on on “GSL framework” under “Requirements” to be directed to a new page β Click on “GSL framework v1.15-2 [1.5 MiB]” to begin downloading the dmg file to your mac χ In the folder where you downloaded it, select and open “GSL_framework1.15-2.dmg” file next δ Double click on the “GSL framework.pkg” icon to begin installation ε Follow the instructions on the installer wizard to complete installation Once both the GDAL and GSL frameworks are installed, download and install the QGIS software also found on “KyngChaos Qgis download page”: α Click on the “QGIS 1.8.0-2 [75.9 MiB]” link under “Download” to begin downloading the dmg file to your mac β In the folder where you downloaded it, select and open “QGIS-1.8.02.dmg” next χ Double click on the “QuantumGIS.pkg” to begin installation δ Follow the instructions on the installer wizard to complete installation 112 Edition E.2.12 You are now ready to begin using Quantum GIS 113 Edition E.2.12 Annex B: References Ten Steps to a Results-based Monitoring and Evaluation System, Kusek and Rist, The World Bank, 2004, (available in numerous languages online for free download) GIS for Dummies, Michael N DeMers, Wiley Publishing Inc, Indianapolis, Indiana, USA, 2009 114 Edition E.2.12 Annex C: Intermediate Outcome Measurement QUANTITATIVE PERFORMANCE MEASUREMENTS FOR WATER SUPPLY DEVELOPMENT PROJECTS: APPLICATION IN RURAL ERITREA Andrew Fuys May 12, 2003 Used by permission of Andrew Fuys granted May 17, 2012 Executive Summary Long recognized as a basic human necessity, safe drinking water was declared a human right last year by the United Nations Committee on Economic, Cultural and Social Rights (UNECSR) In Eritrea, approximately 60 percent of rural households not enjoy this right, and expanding access to safe drinking water in rural areas remains a significant development challenge Expanding the use of quantitative performance measurements could assist the Eritrean government to maximize its limited resources for water supply investments Appropriate project indicators could help the government to identify which investments best meet its water policy goals Selecting performance measurements that are both appropriate to Eritrea’s water policy objectives and relevant to potential international donors could also help to generate investments in the water supply sector To address this question of which performance measurements would be most appropriate in Eritrea, this analysis examines water supply data from the country’s Debub region, case studies from other East African countries, and international agency water supply project evaluations The analysis will consider measurements of four aspects of project performance: • Outputs, or the tangible products that projects generate In the case of water supply projects, outputs are usually physical in nature, e.g., wells, transmission pipes, or covering over freshwater springs • Intermediate outcomes, or the observable and measurable changes that result from the existence of outputs, e.g., lower bacterial levels, increased consumption of protected spring water, or a shorter distance from a village to a water source • Final outcomes, or the way people experience changes in their material lives because of intermediate outcomes, e.g., villagers are sick less frequently because of lower levels of bacterial contamination, or increase consumption because wells are located closer to villages • Process, or the way that a project is designed and implemented, e.g., the involvement of women in water supply management, or attention to the 115 Edition E.2.12 affordability of services for poor households The analysis concludes that intermediate outcome measurements – the observable and quantifiable changes that result from a project, such as increased water consumption – are the most likely to be both meaningful and measurable These measurements address many key water supply development challenges that exist in rural Eritrea, such as increasing source protection and lowering bacteriological counts, decreasing the distance from villages to sources, and improving maintenance and reliability Final outcome measurements – indicators of a project’s effects on people’s material lives, such as health or household income – are central to water sector policy goals It is difficult, however, to attribute causality of final outcomes to individual water projects through impact analysis This is for two reasons: (a) necessary baseline data is often unavailable and (b) multiple explanatory factors make it difficult to model causality such that a significant impact can actually be demonstrated Efforts to analyze whether changes in performance indicators can be attributed to individual projects should thus focus on the causal relationship between water projects and intermediate outcomes, for which modeling causality is less complex In contrast to outcomes, output measurements – those that measure a project’s physical outputs, such as number of wells – will not necessarily tell an evaluator whether a project has achieved its policy goals As such, performance evaluation should not rely solely on these measurements Finally, process measurements related to gender and equity should also be incorporated Women’s involvement in projects is often correlated with success on other outcome indicators, and can be shown through indicators such as representation of women on water committees Equity and affordability measurements can indicate the degrees to which a project benefits poor households, an important consideration for projects that have a specific objective of improving water services to the poor This analysis offers three principal recommendations concerning the use of quantitative performance measurements for water projects in rural Eritrea: Focus on intermediate outcomes, which are measurable, relevant to Eritrea’s water supply needs and commonly used by international agencies These include: • Distance to water source • Time spent collecting water • Protected source coverage • Percentage of facilities that are functional • Bacteriological count • Frequency of water testing • Total household consumption (for all water uses) • Use of improved sources for consumption of drinking water 116 Edition E.2.12 Provide health outcome measurements, such as the number of households that suffered from water-related disease in a given time period, but focus efforts to analyze impact on the intermediate outcomes listed above In order to perform impact analysis on any outcome, project design must allow for the comparison of treatment and control groups Include process measurements that reflect how and whether projects address gender and equity These include the percentage of women on village water committees, and water costs as a percentage of total household consumption 117 Edition E.2.12 .. .Monitoring and Evaluating Programs and Projects Using GIS How to use GIS to support M&E systems for sustainable programs and projects Robert Catherman Director... Programs and Projects Using GIS Part 1: General Principles of M&E Edition E.2.12 Chapter 2: About Monitoring and Evaluation Systems Chapter Objectives • Learning about Monitoring and Evaluation and. .. Project Form that will start the M&E GIS design process that will work for your project 12 Edition E.2.12 Monitoring and Evaluating Programs and Projects Using GIS Part 2: Creating Maps Supporting