This Framework provides policy makers with a concrete, explicit, practical and accessible guide to best practice evaluation methods for SME and entrepreneurship policies and programmes, drawing upon examples from a wide range of OECD countries It examines the benefits of evaluation and how to address common issues that arise when commissioning and undertaking SME and entrepreneurship evaluations Key evaluation principles are set out, including the “Six Steps to Heaven” approach, and illustrated with examples of evaluations of national, regional and local programmes that can be explored further by the reader The publication focuses not only on the evaluation of individual policies and programmes but also on bigger picture peer review evaluations and assessment of the impact on SMEs and entrepreneurship of mainstream programmes that not have business development as their principal aim The full text of this book is available on line via this link: www.sourceoecd.org/industrytrade/9789264040083 Those with access to all OECD books on line should use this link: www.sourceoecd.org/9789264040083 SourceOECD is the OECD’s online library of books, periodicals and statistical databases For more information about this award-winning service and free trials, ask your librarian, or write to us at SourceOECD@oecd.org ISBN 978-92-64-04008-3 85 2007 04 P ����������������������� 852007041cov.indd OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes FIRST IF FIF TH IRST F H T R OND F U ID SEC IRD FO H H THR T D T R N U D O O S EC IF TH F EC O N IRST F FIRST IRST S OND F FIF TH IF TH F D S EC H F TH IR T H H IF R T F T U T FIRS OURTH D FO FO U R CO ND IF TH F D THIR THIRD N IRD SE D IRST F H F O T N C D O H N E S EC S EC O FO U R T ST S THIRD FIF TH FIRST TH FIR OURTH OURTH FIF TH TH FIF H R IF TH F IF TH F T F F U R T T N S ME O S U S F O D FIR N FIR HIRD LUATIO A LU F T A EC O N V V S D LUATIO E E A D N S V Y IR Y E ICY H TH RSHIP S EC O OLIC POLIC IP POL FO U R T R E N EU FIRST SHIP P R EU R S H FIF TH IP NTR EP N U T E H E S H E R N S IR S P N F R IO E UR RE E NTR E ALUAT HIP D IP SM R E N EU PR E N E URSH NTR EP ICY E V S ME N EU R S E POL NTR EP PR E N E TI R E PR E S ME E E NTR E ATION U N E L E NTR E VALUA N ENT HIP SM M E IO S A S ME E N IO S T R Y V T IO U A A E IC T N E OL VALU REN ALUA E VALU UATIO OLICY SME P O L Y LICY E P NTR EP ICY E V P IP A E O L H IC P V O IP S N L IP E P E R H SM SME LUATIO URS LICY R E N EU IP PO URSH Y E VA RSHIP NTR EP PR E N E HIP PO EU R S H PR E N E POLIC E S R E N EU E N N EN ION E E R P R T E R IO E M T A T U T R R S U A N T E L N P E A IP N EN EE E VALU TR E REN URSH ICY E V LUATIO OLICY PR E N E E POL ME EN ION SM NTR EP S ME Y E VA SME P IP SM E NTR E POLIC ALUAT S ME E TION S V RSHIP A EU R S H ATION N ATION U E U N U E U E L L IO Y N L R A E A T P V A R IC V A V L RE E EP U Y E E T O R L N Y IC T Y P L E A N O IC V E ION E P YE OLIC SHIP POL ATION ALUAT IP SM POLIC E VALU SHIP P URSH E N EU R ICY E V OLICY PR E N E E POL E N EU R URSHIP T R E PR R E SME P N IP SM P E NTR E N E H E IP E S H E R R S R R ENT N SM R E N EU R E N EU NTR EP NTR EP N S ME NTR EP LUATIO S ME E TION E ION E Y E VA LUATIO VALUA A ALUAT IC E V V L E Y E O Y IC P OL OLIC LICY ION SME P URSHIP EURSHIP SME P ALUAT HIP PO ICY E V N PR E N E N EU R S E POL E M R R E PR E S T P N E NTR E IP E E H N NTR N EU R S LUATIO R E PR E S ME E Y E VA N ENT POLIC ATION U UATIO L L A A V V E YE ICY E POL POLIC HIP SM E NTR E PR E N E URS HIP SM N EU R S R E PR E T I N E ATION EU R S H E VALU E PR E N R OLICY P T I T E N A M E IP S ME E VALU URSH OLICY TION S PR E N E SME P E NTR E VALUA E RSHIP HIP PO U Y E S N IC R E R E PR E N EU IP POL R H P N ENT S E IO R T R ION A U T ALUAT E VALU E EN RENE ICY E V OLICY ION SM NTR EP E POL T SME P E M A S E IP N U H IP L M S H R E VA ION S LUATIO N EU R S R E N EU S OLICY R E PR E NTR EP Y E VA ALUAT N ENT ION E RSHIP SHIP P POLIC ICY E V R L ALUAT LUATIO U IP R E N EU V A O E E H V E P E N Y S R E Y R NT P IP LIC R IC U E O L H P E P N O S E P N P E IO R E NTR TR E PR E ALUAT HIP SM IP SM R E N EU URSH ICY E V S ME E N EU R S ME EN E NTR E E POL NTR EP PR E N E R E PR E NT ATION TION S N S ME U A L E NTR E HIP SM N ENT ION E U IO S A T S ME E N L IO T R A V T U A IO A U A E E T L V LUA E VA REN VALU LICY ICY E E VALU O L Y E VA Y OLICY NTR EP LICY E P EV O IC E P O L IC P P Y E N O IP L P E M IO SM SME HIP S POLIC URSH IP PO ALUAT RSHIP RSHIP N EU R S ICY E V RSHIP PR E N E EU R S H R E N EU E U N P R E N EU UR R E PR E E POL E R P E E T E M E T N N N R R S R E E E N T T P R IP E PR EN N EN TR E ATION URSH NTR EP LUATIO E VALU E NTR E PR E N E N S ME TION E ME EN Y E VA PO OLICY VALUA E NTR E N S ME LUATIO POLIC TION S SME P A IO A LICY E ATION V T U IP O U A E L URSHIP L P H A S U E A E Y V R L V N IC A E SM E EU Y E L V N R Y IC IP E E O L P N H R O IC P S P ME P HIP NTR E N EU R POL LUATIO NTR E OLICY Y E VA SHIP S R E PR E TION E S ME E N EU R S SHIP P E POLIC N R N ENT VALUA E N EU R R E E R U IO IO P P M T P Y E T E E S A E IC R A N U L IP OL LU ENT NTR NTR PR E URSH Y E VA SME P Y E VA S ME E POLIC PR E N E S ME E E NTR E POLIC E NTR E -:HSTCQE=UYUU]X: 11-Dec-2007 10:43:50 AM OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT The OECD is a unique forum where the governments of 30 democracies work together to address the economic, social and environmental challenges of globalisation The OECD is also at the forefront of efforts to understand and to help governments respond to new developments and concerns, such as corporate governance, the information economy and the challenges of an ageing population The Organisation provides a setting where governments can compare policy experiences, seek answers to common problems, identify good practice and work to co-ordinate domestic and international policies The OECD member countries are: Australia, Austria, Belgium, Canada, the Czech Republic, Denmark, Finland, France, Germany, Greece, Hungary, Iceland, Ireland, Italy, Japan, Korea, Luxembourg, Mexico, the Netherlands, New Zealand, Norway, Poland, Portugal, the Slovak Republic, Spain, Sweden, Switzerland, Turkey, the United Kingdom and the United States The Commission of the European Communities takes part in the work of the OECD OECD Publishing disseminates widely the results of the Organisation’s statistics gathering and research on economic, social and environmental issues, as well as the conventions, guidelines and standards agreed by its members This work is published on the responsibility of the Secretary-General of the OECD The opinions expressed and arguments employed herein not necessarily reflect the official views of the Organisation or of the governments of its member countries Also available in French under the title: Cadre de l’OCDE sur l’évaluation des politiques et des programmes l’égard des PME et de l’entrepreneuriat Corrigenda to OECD publications may be found on line at: www.oecd.org/publishing/corrigenda © OECD 2007 No reproduction, copy, transmission or translation of this publication may be made without written permission Applications should be sent to OECD Publishing rights@oecd.org or by fax 33 45 24 99 30 Permission to photocopy a portion of this work should be addressed to the Centre français d’exploitation du droit de copie (CFC), 20, rue des Grands-Augustins, 75006 Paris, France, fax 33 46 34 67 19, contact@cfcopies.com or (for US only) to Copyright Clearance Center (CCC), 222 Rosewood Drive, Danvers, MA 01923, USA, fax 978 646 8600, info@copyright.com FOREWORD Foreword T he OECD Working Party on Small and Medium-sized Enterprises and Entrepreneurship (WPSMEE), in line with a recommendation of the 2004 Istanbul Ministerial Declaration on Fostering the Growth of Innovation and Internationally Competitive SMEs, has prepared this report aimed at strengthening the conceptual framework for SME policy evaluation This report seeks to be of direct practical assistance to public administrators and politicians concerned with evidence on the effectiveness of SME and entrepreneurship policies and programmes at a national and local level The Framework was written by Dr Jonathan Potter, Principal Administrator, OECD Centre for Entrepreneurship, SMEs and Local Development, and Prof David Storey, Warwick Business School, UK, and prepared under the supervision of Mme Marie-Florence Estimé, Deputy Director of the OECD Centre on Entrepreneurship, SMEs and Local Development (CFE) A Steering Group, co-chaired by Dr Roger Wigglesworth, New Zealand and Mr George Bramley, United Kingdom, guided the preparation of the Framework The Co-Chairs along with the members of the Steering Group offered many valuable comments during the drafting, revisions and review of the Framework: Mrs Sue Weston and Ms Vicki Brown, Australia; Mrs Laura Morin, and Ms Kaili Levesque, Canada; Ms Annukka Lehtonen and Mr Pertti Valtonen, Finland; Mr Serge Boscher and Mr Jean-Hugues Pierson, France; Mr Tamas Lesko and Dr Ágnes Jánszky, Hungary; Mr Young-Tae Kim and Dr Sung Cheon Kang, Korea; and Ms Ana María Lagares Pérez, Spain Sincere appreciation is extended to the Delegates of the OECD WPSMEE for their numerous comments and inputs during the compilation of the Framework Thanks also go to Mr Kevin Williams, Principal Administrator, OECD Council and Executive Committee Secretariat, Mr Hans Lundgren, Head of Section, Evaluation, Development Co-operation Directorate, and Mrs Mariarosa Lunati, Administrator, CFE/ SME and Entrepreneurship Division for their drafting suggestions and Ms Brynn Deprey, Mr Jorge Gálvez Mérdez, Mr Damian Garnys, and Ms Elsie Lotthe for their operational support OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 TABLE OF CONTENTS Table of Contents Summary and Route Map Section Evaluation Issues Defining evaluation Why an evaluation? Typical objections to evaluation and responses Key evaluation debates Doing evaluations Some key principles for evaluation practice 15 16 17 19 22 27 32 Notes 34 Section Evaluation of Individual National Programmes Introduction Evaluations of financial assistance Enterprise culture Advice and assistance Technology Conclusion 37 38 39 42 43 47 48 Section Evaluation of Regional and Local Programmes Introduction Advice, consultancy and financial assistance Clusters and local innovation systems Support to areas of geographical disadvantage Conclusion 53 54 55 57 59 64 Section The Role of Peer Review in Evaluation Introduction The peer review methodology OECD national SME reviews OECD regional and local entrepreneurship reviews OECD evaluation guidance 67 68 68 70 71 72 Section Reviewing the Aggregate Impact of Public Policies Introduction Impact of mainstream policies on SMEs Capturing the total policy package 75 76 77 91 OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 TABLE OF CONTENTS Conclusion 94 Notes References 95 97 Appendix A Six Steps to Heaven: Methods for Assessing the Impact of SME Policy Appendix B The OECD Istanbul Position Appendix C Examples of Evaluation Guidance Appendix D Assessing the Quality of an Evaluation Appendix E Framework Condition Indicators: Entrepreneurship Conditions in Denmark in 2005 Appendix F Summary of the Evaluation of State Aid to SMEs in the Member States, European Economic Area and the Candidate Countries 106 103 109 111 113 120 List of tables 1.1 1.2 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 2.22 2.23 2.24 Qualitative compared with quantitative evaluation The choice of internal and external evaluators SME and entrepreneurship policy areas covered Loan guarantee scheme, Japan Loan guarantee scheme, Canada Assistance to new enterprises started by young people, Italy Grant assistance and small firm performance, Ireland Public subsidies to business angels: EIS and VCT, UK Public subsidies to business angels: EIS, UK Assisting young disadvantaged people to start up businesses, UK Graduates into business, UK Investment readiness, New Zealand Impact of marketing advice, UK Impact of business advice, Belgium Impact of advisory support, Bangladesh Bank customers receiving business advice, UK Assistance and advice for mature SMEs, UK Use and impact of business advice, UK Evaluating entrepreneurial assistance programs, US Encouraging partnerships amongst SMEs, Sweden Technology assistance to small firms, US The SBIR program, US The UK SMART programme Impact of science parks, Greece Impact of science parks, Sweden University/SME links, New Zealand 23 25 38 39 40 40 40 41 41 42 43 43 44 44 44 45 45 46 46 47 48 48 49 49 50 50 OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 TABLE OF CONTENTS 2.25 2.26 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8 3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 5.1 5.2 5.3 5.4 5.5 B.1 D.1 Impact of management training on SMEs, UK 51 Small firms training loans, UK 51 Regional/local policy areas covered 55 Subsidised consulting, Belgium, Wallonia 56 Business advisory services, UK, South West England 56 Enhancing the capability of the SME owner through use of consultants, UK, Scotland 57 Export information and advice, Canada, Quebec 57 Enterprise partnerships for exporting, Sweden, Örebo 58 Small business grants, UK, North East England 58 Regional development agency grants, Ireland, Shannon 59 Local innovation system policy, EU regions 60 Business networking, UK, North East England 61 Enterprise Zone evaluation, US, Indiana 61 Enterprise Zone evaluation, US, Five States 62 Enterprise Zone evaluation, UK 62 Evaluation of enterprise support in disadvantaged areas, UK 63 Regional policy evaluation, UK 63 Regional policy evaluation, Italy 64 Rural policy evaluation, Canada, Quebec 64 Rural enterprise support, United Kingdom, Northumberland 65 The indicators 83 Ease of Doing Business ranking 84 Starting a business in 1999, 2004 and 2006 85 Average conversion rates young businesses/nascent entrepreneurs, 2000-2004 92 Selecting policy areas 92 Six Steps to Heaven: Methods for assessing the impact of SME policy 106 Grid for a synthetic assessment of the quality of evaluation work 112 Figure 1.1 New Zealand Trade and Enterprise (NZTE) Growth Range Programme Logic Model 31 OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 APPENDIX C OECD: Development Assistance Committee, Evaluation Resource Centre (www.oecd.org/document/63/0,2340,en_35038640_35039563_35067327_ 1_1_1_1,00.html) OECD: Development Assistance Committee Glossary of Key Terms in Evaluation and Results Based Management, OECD, 2002 (www.oecd.org/ dataoecd/29/21/2754804.pdf) and Joint Evaluations (www.oecd.org/dataoecd/ 28/9/35353699.pdf) Organization for Security and Co-operation in Europe (OSCE): Forthcoming publications on “Systems of Indicators for Evaluation of State-supported SME Development Programs” and “Monitoring and Evaluation Handbook” UNDP/Global Environment Facility (GEF): Measuring and Demonstrating Impact Resource Kit, 2005 (www.undp.org/gef/05/documents/me/UNDP_GEF_Measuring_ and_Demonstrating_Impact_Mar05.doc) UNICEF: Standards (www.unicef.org/evaluation/files/Evaluation_standards.pdf) and (www.unicef.org/evaluation/index_13486.html) World Bank: Framework for Evaluating Framework the Impact of SME Programs Kris Hallberg Operations Evaluation Department, World Bank (http:// info.worldbank.org/etools/docs/library/86503/Session%201%20Hallberg.pdf) World Bank: Ten Steps to a Results-Based Monitoring and Evaluation System: a Handbook for Development Practitioners, 2004 (www-wds.worldbank.org/ servlet/WDSContentServer/WDSP/IB/2004/08/27/000160016_20040827154900/ Rendered/PDF/296720PAPER0100steps.pdf) OTHERS Evaluate Europe: (www.evaluate-europe.net/handbook/) and (www.evaluateeurope.net/tools/eval_of_elearning/sme_enviro/) Nexus Associates: Framework for Evaluating the Impact of Small Enterprise Initiatives (Eric Oldsman, Kris Hallberg), 2003 (www.enterprise-impact.org.uk/ pdf/FrameworkforEvaluatingImpact-Oldsman2003.pdf) Performance Assessment Resource Centre (PARC): provides support services on performance management combined with the development of an information resource centre on performance assessment (www.parcinfo.org) Enhancing Organizational Performance: a Toolbox for Self-assessment, 1999 (www.idrc.ca/en/ev-9370-201-1-DO_TOPIC.html) 110 OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 APPENDIX D APPENDIX D Assessing the Quality of an Evaluation Although the quality of an evaluation is often judged on the sole basis of the final report, it largely depends on the way in which the entire evaluation process was managed, from the commissioning of the evaluation to the conduct of the work, the drawing of conclusions, the formulation of recommendations and the communication of results It is necessary to assess the quality of an evaluation in order to verify whether the evaluation team satisfied the commissioners’ requirements and to ensure that the evaluation can stand up to criticism that may arise when judgements on success and failure are made In particular, it has to be decided whether the collection and analysis of data is sufficient or must be improved, whether the conclusions drawn from the evaluation are acceptable in view of the proof gathered, and whether the report answers all the questions mentioned in the terms of reference or whether further work needs to be under taken A grid for assessing the quality of an evaluation is provided in the following table OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 111 APPENDIX D Table D.1 Grid for a synthetic assessment of the quality of evaluation work With regard to this criterion, the evaluation report is: Unacceptable Acceptable Good Excellent Meeting needs: Does the evaluation adequately address the requests for information formulated by the commissioners and does it correspond to the terms of reference? Relevant scope: Have the rationale of the programme, its outputs, results, impacts, interactions with other policies and unexpected effects been carefully studied? Defensible design: Is the design of the evaluation appropriate and adequate for obtaining the results (with their limits of validity) needed to answer the main evaluative questions? Reliable data: Are the primary and secondary data collected or selected suitable? Are they sufficiently reliable compared to the expected use? Sound analysis: Are the quantitative and qualitative data analysed in accordance with established rules and are they complete and appropriate for answering the evaluative questions correctly? Credible results: Are the results logical and justified by the analysis of data and by interpretations based on carefully presented explanatory hypotheses? Impartial conclusions: Are the conclusions just and non-biased by personal or partisan considerations, and are they detailed enough to be implemented concretely? Clear report: Does the report describe the context and goals, as well as the organisation and results of the evaluated programme in such a way that the information provided is easily understood? In view of the contextual constraints bearing on the evaluation, the evaluation report is considered to be Source: European Commission (1999) 112 OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 APPENDIX E APPENDIX E Framework Condition Indicators: Entrepreneurship Conditions in Denmark in 2005 The National Agency for Enterprise and Construction in Denmark identified indicators of entrepreneurship conditions which were selected on the following four grounds: ● relevance; ● accuracy; ● availability; and ● coherence The indicators are shown below in groups a) Technology transfer University/industry research collaboration measures business executives’ perceptions of the scope of R&D collaboration between their firm and local universities Source: World Economic Forum (WEC), Global Competitiveness Report.1 Technological co-operation measures business executives’ assessment of the level of technological co-operation between companies Source: International Institute for Management Development (IMD), World Competitiveness Yearbook.2 b) Entry barriers Barriers to competition consists of two indices that measure national, state or provincial laws or other regulations that restrict the number of competitors allowed to operate, and scope of exemptions to competition law for public enterprises or government mandated behaviour Source: OECD OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 113 APPENDIX E Public ownership measures the scope of publicly owned enterprises across multiple product and service sectors, the size of state-owned enterprises, and the existence of government special voting rights in privately-owned enterprises Source: OECD Public involvement in business operation reflects the extent to which government use coercive or incentive-based regulation in specific sectors Source: OECD c) Access to foreign markets Share of new enterprises with exports measures the share of exporting companies among new enterprises Source: FORA, Denmark Access to capital markets measures to what extent business executives assess domestic access to capital markets Source: IMD Export credit and insurance measures top managements’ perceptions of the availability of export credits and insurance at reasonable prices Source: IMD d) Loan capital The scope of loan guarantees measures the maximum extent of loan guarantees as a per cent of a given loan Sources: EU Commission, OECD, SBA, SMBA Private credit (average 1997-2001) measure the ratio of credit going to the private sector from deposit-taking financial institutions relative to GDP Sources: World Bank, Doing Business Interest rate spread measures the lending rate minus the deposit rate based on the average annual interest rates for each country Source: World Bank, Doing Business Cost to create collateral measures creditor costs incurred when creating and registering collateral (as a percentage of income per capita) Source: World Bank, Doing Business Legal Rights Index measures the extent to which legislation pertaining to collateral and bankruptcies are beneficial to the entrepreneur Source: World Bank, Doing Business Country Credit Rating measures country credit ratings The rating is based on an assessment by the Institutional Investor Magazine Source: IMD, World Competitiveness Yearbook e) Venture capital Venture capital (early stage) measures the average total annual venture capital investment (early stage) as a share of GDP Source: OECD Venture Capital Database 114 OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 APPENDIX E Venture capital (expansion stage) measures the average total annual venture capital investment (expansion stage) as a share of GDP Source: OECD, OECD Venture Capital Database f) Exit opportunities Capitalisation of secondary stock market (average 1999-2002) measures the capitalisation (the total value of issued shares) in the secondary stock market in percentage of GDP Source: OECD New companies quoted in secondary stock market (average 1999-2002) measures the number of new companies quoted in the secondary stock market per million inhabitants Source: OECD Capitalisation of primary stock market (average 1997-2001) measures the capitalisation (the total value of issued shares) in the primary stock market in percentage of GDP Source: World Bank Turnover in primary stock market (average 1997-2001) measures the value of total shares traded on the stock market exchange in percentage of GDP Source: World Bank Buy outs (average 1999-2002) measures the total annual value of buy-outs as a share of GDP Source: OECD Venture Capital Database g) Wealth and bequest tax Revenue from bequest tax measures the revenue from bequest tax as per cent of GDP – 3-year moving average Source: OECD Revenue from wealth tax measures the revenue from wealth tax as per cent of GDP – 3-year moving average Source: OECD Top marginal bequest tax rate measures the top marginal bequest tax rate for spouses or children to the deceased Source: Chen, Lee and Mintz, OECD h) Capital tax Taxation of dividends – top marginal tax rate measures the top marginal tax rate of dividend income Source: Chen, Lee and Mintz, OECD Taxation of dividends – top marginal tax rate for the self-employed measures the top marginal tax rate of dividend income for the self-employed Source: Chen, Lee and Mintz, OECD Taxation of stock options measures the effective tax rate of stock options for a hypothetical tax payer with certain assumptions regarding income, family situation and portfolio development Source: EU Commission OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 115 APPENDIX E Taxation of capital gains on shares – short term measures the level of the tax rate when shares are issued and sold in the short term The indicator is comprised of two sub-indicators; one tax rate for companies and one tax rate for individuals Source: Chen, Lee and Mintz, OECD Taxation of capital gains on shares – long term measures the level of the tax rate when shares are issued and sold in the long term The indicators are comprised of two lower-level indicators; one tax rate for companies and one tax rate for individuals Source: Chen, Lee and Mintz, OECD i) Restart possibilities Time that a creditor has claims on a bankrupt’s assets measures the time it takes to settle old debt after bankruptcy The indicator is measured by the number of years Source: OECD j) Entrepreneurship education Entrepreneurship education at primary and secondary levels measures the perception of entrepreneurship experts of the quantity and quality of entrepreneurship education in primary and secondary education Source: GEM and Schøtt (2005c) Entrepreneurship education at higher levels measures the perception of entrepreneurship experts of the quantity and quality of entrepreneurship education in higher education Source: GEM and Schøtt (2005c) k) Traditional business education Quality of management schools measures business executives’ assessment of the quality of management or business schools Source: WEF, Global Competitiveness Report l) Government programmes Government programmes measures entrepreneurial experts’ assessment of government counselling programmes implemented to promote the creation and subsequent growth of new enterprises Source: GEM m) Personal income tax Highest marginal income tax rate plus contributions measures the highest rate of taxation for single, high income person without children in percentage of the gross wage Source: OECD Average income tax plus social contributions measures the average tax rate for a single person without children in percentage of the gross wage Source: OECD 116 OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 APPENDIX E n) Corporate tax (including fiscal incentives) SME tax rates measure the corporate tax rate for small and medium-sized businesses Source: Chen, Lee and Mintz, OECD Revenue from corporate taxes measures the revenue from corporate income tax as a percentage of GDP (3-year moving average) Source: OECD o) Bankruptcy legislation Actual cost to close a business measures the cost of closing a business in per cent of the estate, based on a standard business closure Source: World Bank, Doing Business Actual time to close a business measures the actual time to close a business recorded in calendar years The indicator is based on a standard business closure Source: World Bank, Doing Business Recovery rate measure the effectiveness of bankruptcy legislation by estimating the pay back rate, i.e how many cents per dollar that creditors are paid from an insolvent estate Source: World Bank p) Administrative burdens Number of procedures for starting a business records the generic procedures that are officially required for an entrepreneur to start an industrial or commercial business Source: World Bank, Doing Business Average number of days for starting a business measures the average time spent during each enterprise start-up procedure in calendar days per start-up procedure Source: World Bank, Doing Business Costs incurred to start a business measures the official costs when starting a business Source: World Bank, Doing Business Regulatory and administrative opacity measures the simplicity in obtaining permits and licenses from the government as well as the regulatory and administrative opacity in business-related rules and administrative practices The measure is an index of questions covering the process of obtaining permits, license and authorisations, as well as the extent to which authorities make concerted efforts in making business-related regulation intelligible and simple Source: OECD Enforcing contracts – number of procedures measures the number of procedures mandated by law or court regulation that require interaction between the concerned parties or between them and the judge, administrator or court officer Source: World Bank, Doing Business Enforcing contracts – time measures the average number of days from the moment the plaintiff files the lawsuit until the settlement or payment Source: World Bank, Doing Business OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 117 APPENDIX E Enforcing contracts – cost in percentage of debt measures the official costs of going through court procedures in percentage of the debt value Source: World Bank, Doing Business Required capital for starting a business with limited liability measures the required capital in a company with limited liability Source: World Bank, Doing Business q) Labour market regulation Flexibility of hiring index measures whether legislation or other regulations have implications for the difficulties of hiring a standard worker in a standard company Source: World Bank, Doing Business Flexibility of firing index measures whether legislation or other regulation have implications for the difficulties of firing a standard worker in a standard company Source: World Bank, Doing Business Flexibility of overtime work is an index with five components that measure the rigidity of rules pertaining to overtime work Source: World Bank, Doing Business Number of administrative procedures when recruiting the first employee measures all mandatory contracts that employers have to make with administrative units in the recruitment process for the first employee Source: European Commission Number of administrative procedures when recruiting additional employees measures all mandatory contracts that employers have to make with administrative units in the recruitment process of additional employees Source: European Commission Cost of firing measures the costs related to firing employees; advance notice requirements, severance payments and penalties due in terms of weekly wages in a standard company Source: World Bank, Doing Business r) Culture/motivation Cultural and social norms measures entrepreneurial experts’ perception of cultural and social norms based n five questions Source: GEM and Schøtt (2005b) Entrepreneurial motivation The indicator measures entrepreneurial experts’ assessment of entrepreneurial motivation The indicator is based on four questions Source: GEM and Schøtt (2005b) Self-employment preferences measures individuals’ preferences towards being self-employed or being a regular employee Source: Eurobarometer Desirability of becoming self-employed – short term measures people’s desire to become self-employed in the short term Source: Eurobarometer 118 OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 APPENDIX E Desirability of becoming self-employed – long term measures people’s desire to become self-employed within the next five years Source: Eurobarometer Risk of business failure measures people’s perception of being willing to start a business if a risk exists that it might fail Source: Eurobarometer Notes The World Economic Forum’s Global Competitiveness Report is based on an “Executive Opinion Survey” as well as data collected from other sources In the Executive Opinion Survey managers from across the business world assess the quality of national framework conditions conducive to competitiveness The applied indicators IMD’s Competitiveness Yearbook is based on an “Executive Opinion Survey”, as well as data collected from other sources In the Executive Opinion Survey managers from across the business world assess the quality of national framework conditions conducive to competitiveness OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 119 APPENDIX F APPENDIX F Summary of the Evaluation of State Aid to SMEs in the Member States, European Economic Area and the Candidate Countries Country Q A E Short description Austria y y s Some schemes are evaluated, partly internally, partly outsourced Evaluations are mainly quantitative (using questionnaire surveys) with attention for qualitative aspects (e.g “improvement of market position” of the supported firms) Outputs include sets of ratios and multipliers that measure the effects in terms of money value granted An example of this would be the ratio additional sales prompted per € assistance, or the subsidy amount per job created Austria applies both ex-nunc monitoring and ex post evaluations Belguim y y s Aid and evaluation are the responsibility of the regions Some regions evaluate For example, the Walloon region recently commissioned an evaluation of consultancy aid This extensive study describes the (entry) problems SMEs experience, the support provided, and the effects of the support The study employs various tools and techniques for data collection (among users and non-users of the support) such as literature research, file research, telephone survey and face-to-face interviews Also, they use various methods for data analysis, such as graphic representations, statistical analyses and control groups There is a separate chapter on synthesis and recommendations Bulgaria n n ? Cyprus n y ? Czech Republic y n s The Ministry of Trade and Industry evaluates some schemes A major evaluation of the CzechMoravian Guarantee and Development Bank (whose objective it is to stimulate the SME sector) is currently underway Denmark y y s Denmark does evaluate state aid, but not specific SME schemes Some of the evaluations are outsourced Reports are in Danish Estonia n y ? Finland y y s France n y ? 120 Ex ante, ex-nunc and ex post evaluations re mandatory by law, but in practice they not always take place As a result, there is an impressive record of Finnish evaluation studies However, a government working group judged the valuation efforts unsystematic Therefore, the working group has made 10 recommendations so that evaluation results will be integrated in the policy process in a systematic manner OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 APPENDIX F Country Q A E Short description Germany y y s Germany evaluates some of its state aid Part of these evaluation studies are performed internally, other studies are outsourced Evaluation has helped Germany to improve state aid objectives considerably during the last years In general, German evaluations of state aid to SMEs use quantitative approaches, where the results of assisted firms are compared with the results of non-assisted firms German evaluators also apply time series analysis Greece y y n In Greece, state aid measures are not evaluated yet A method for the appraisal of aid granted is being developed Hungary y y s Some Hungarian state aid schemes are evaluated, partly by external consultants Evaluations are often based on statistical analyses, but an interesting good practice in Hungary is the analysis of 80 randomly chosen micro-credit documents (including the micro-credit contracts and the recipients’ business plans) Ireland y y a Italy y y s Italy evaluates some of its aid schemes Examples include a quantitative analysis of the net impacts of SME subsidies on employment This evaluation uses literature data, data from the tax and social security offices and a questionnaire Based on this, indicators of the support scheme’s effects are calculated The indicators are applied to measure the extent to which the support objectives have been reached Latvia y y n Latvia does not evaluate its state aid to SMEs Lithuania y y ? Luxembourg n n ? Malta n n ? The Netherlands y y a It is stated in the law that: For each new or adjusted policy objective or instrument it should be considered whether exante evaluation is useful Each existing support measure should be evaluated, at least once per five years For ex post evaluations, the Dutch Ministry of Economic Affairs uses one standard format The main purpose of evaluation is to check whether the policy instrument contributes to the main goal for which it was created Norway y y s In Norway state aid is targeted at horizontal objectives, most of which concern SMEs Besides evaluation results, political and administrative priorities have driven changes in the Norwegian state aid policy Poland y y n Poland does not evaluate state aid to SMEs Portugal y y s A large proportion of Portuguese state aid to SMEs is included in the Community Support Framework An ex ante evaluation of this framework was carried out and in 2003 the support measures (including state aid) under this framework were subject to mid-term evaluations using a combination of qualitative (case studies) and quantitative (financial data, physical indicators, beneficiary questionnaires) tools, based on the MEANS quality criteria Only state aid schemes that are not included in the Community Support Framework are not evaluated Romania n n ? Slovakia n y ? Slovenia n y ? The State Aid Act obliges the Slovak government to evaluate all state aid This is done using one standard evaluation technique for all state aid This technique consists of statistical (input-output) analyses of economic characteristics of aid recipients and a (total population) control group, (turnover, value added, profits, taxes, exports and number of employees) The evaluation focuses on four topics: technology transfer, innovation, exports and competitiveness The outputs of the analyses are data, such as: number of jobs created through the scheme, number of companies established due to the measure, etc The analyses are performed both on micro and macro levels Aided SMEs are followed for three consecutive years to see how they develop Data about the assisted companies are obtained through questionnaire surveys Data about the control group are obtained from the Tax Office and the Statistical Office OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 121 APPENDIX F Country Q A E Spain Sweden n y ? y y s Turkey n n ? United Kingdom y y s Short description The Swedes use quantitative matched sample control group approaches, total population control group approaches, cost-benefit analyses, supported with more qualitative interviews They use these methods to measure the effect of state aid on the survival and sales of firms, employment, regional development, etc They also assess the influence of other variables, such as sector, firm size and region Evaluations have yielded important insights into the effects of support The Swedes not use the outcomes of evaluation studies for re-focussing state aid objectives Recently, evaluation of all DTI services acted as inputs for a new strategy to reduce the number of services to 10-14 All evaluations are out-sourced and follow DTI evaluation guidelines Where possible they use a set of standard indicators Notes: Q = questionnaire received from the country y: yes; n: no A = attended the member State meeting on 25 May 2004 y: yes; n: no E = perform state aid evaluation a: all state aid schemes are evaluated s: some state aid schemes are evaluated n: no state aid scheme is evaluated ? no information available Source: EIM (2004) 122 OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN 978-92-64-04008-3 – © OECD 2007 OECD PUBLICATIONS, 2, rue André-Pascal, 75775 PARIS CEDEX 16 PRINTED IN FRANCE (85 2007 04 P) ISBN 978-92-64-04008-3 – No 55919 2006 This Framework provides policy makers with a concrete, explicit, practical and accessible guide to best practice evaluation methods for SME and entrepreneurship policies and programmes, drawing upon examples from a wide range of OECD countries It examines the benefits of evaluation and how to address common issues that arise when commissioning and undertaking SME and entrepreneurship evaluations Key evaluation principles are set out, including the “Six Steps to Heaven” approach, and illustrated with examples of evaluations of national, regional and local programmes that can be explored further by the reader The publication focuses not only on the evaluation of individual policies and programmes but also on bigger picture peer review evaluations and assessment of the impact on SMEs and entrepreneurship of mainstream programmes that not have business development as their principal aim The full text of this book is available on line via this link: www.sourceoecd.org/industrytrade/9789264040083 Those with access to all OECD books on line should use this link: www.sourceoecd.org/9789264040083 SourceOECD is the OECD’s online library of books, periodicals and statistical databases For more information about this award-winning service and free trials, ask your librarian, or write to us at SourceOECD@oecd.org ISBN 978-92-64-04008-3 85 2007 04 P ����������������������� 852007041cov.indd OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes FIRST IF FIF TH IRST F H T R OND F U ID SEC IRD FO H H THR T D T R N U D O O S EC IF TH F EC O N IRST F FIRST IRST S OND F FIF TH IF TH F D S EC H F TH IR T H H IF R T F T U T FIRS OURTH D FO FO U R CO ND IF TH F D THIR THIRD N IRD SE D IRST F H F O T N C D O H N E S EC S EC O FO U R T ST S THIRD FIF TH FIRST TH FIR OURTH OURTH FIF TH TH FIF H R IF TH F IF TH F T F F U R T T N S ME O S U S F O D FIR N FIR HIRD LUATIO A LU F T A EC O N V V S D LUATIO E E A D N S V Y IR Y E ICY H TH RSHIP S EC O OLIC POLIC IP POL FO U R T R E N EU FIRST SHIP P R EU R S H FIF TH IP NTR EP N U T E H E S H E R N S IR S P N F R IO E UR RE E NTR E ALUAT HIP D IP SM R E N EU PR E N E URSH NTR EP ICY E V S ME N EU R S E POL NTR EP PR E N E TI R E PR E S ME E E NTR E ATION U N E L E NTR E VALUA N ENT HIP SM M E IO S A S ME E N IO S T R Y V T IO U A A E IC T N E OL VALU REN ALUA E VALU UATIO OLICY SME P O L Y LICY E P NTR EP ICY E V P IP A E O L H IC P V O IP S N L IP E P E R H SM SME LUATIO URS LICY R E N EU IP PO URSH Y E VA RSHIP NTR EP PR E N E HIP PO EU R S H PR E N E POLIC E S R E N EU E N N EN ION E E R P R T E R IO E M T A T U T R R S U A N T E L N P E A IP N EN EE E VALU TR E REN URSH ICY E V LUATIO OLICY PR E N E E POL ME EN ION SM NTR EP S ME Y E VA SME P IP SM E NTR E POLIC ALUAT S ME E TION S V RSHIP A EU R S H ATION N ATION U E U N U E U E L L IO Y N L R A E A T P V A R IC V A V L RE E EP U Y E E T O R L N Y IC T Y P L E A N O IC V E ION E P YE OLIC SHIP POL ATION ALUAT IP SM POLIC E VALU SHIP P URSH E N EU R ICY E V OLICY PR E N E E POL E N EU R URSHIP T R E PR R E SME P N IP SM P E NTR E N E H E IP E S H E R R S R R ENT N SM R E N EU R E N EU NTR EP NTR EP N S ME NTR EP LUATIO S ME E TION E ION E Y E VA LUATIO VALUA A ALUAT IC E V V L E Y E O Y IC P OL OLIC LICY ION SME P URSHIP EURSHIP SME P ALUAT HIP PO ICY E V N PR E N E N EU R S E POL E M R R E PR E S T P N E NTR E IP E E H N NTR N EU R S LUATIO R E PR E S ME E Y E VA N ENT POLIC ATION U UATIO L L A A V V E YE ICY E POL POLIC HIP SM E NTR E PR E N E URS HIP SM N EU R S R E PR E T I N E ATION EU R S H E VALU E PR E N R OLICY P T I T E N A M E IP S ME E VALU URSH OLICY TION S PR E N E SME P E NTR E VALUA E RSHIP HIP PO U Y E S N IC R E R E PR E N EU IP POL R H P N ENT S E IO R T R ION A U T ALUAT E VALU E EN RENE ICY E V OLICY ION SM NTR EP E POL T SME P E M A S E IP N U H IP L M S H R E VA ION S LUATIO N EU R S R E N EU S OLICY R E PR E NTR EP Y E VA ALUAT N ENT ION E RSHIP SHIP P POLIC ICY E V R L ALUAT LUATIO U IP R E N EU V A O E E H V E P E N Y S R E Y R NT P IP LIC R IC U E O L H P E P N O S E P N P E IO R E NTR TR E PR E ALUAT HIP SM IP SM R E N EU URSH ICY E V S ME E N EU R S ME EN E NTR E E POL NTR EP PR E N E R E PR E NT ATION TION S N S ME U A L E NTR E HIP SM N ENT ION E U IO S A T S ME E N L IO T R A V T U A IO A U A E E T L V LUA E VA REN VALU LICY ICY E E VALU O L Y E VA Y OLICY NTR EP LICY E P EV O IC E P O L IC P P Y E N O IP L P E M IO SM SME HIP S POLIC URSH IP PO ALUAT RSHIP RSHIP N EU R S ICY E V RSHIP PR E N E EU R S H R E N EU E U N P R E N EU UR R E PR E E POL E R P E E T E M E T N N N R R S R E E E N T T P R IP E PR EN N EN TR E ATION URSH NTR EP LUATIO E VALU E NTR E PR E N E N S ME TION E ME EN Y E VA PO OLICY VALUA E NTR E N S ME LUATIO POLIC TION S SME P A IO A LICY E ATION V T U IP O U A E L URSHIP L P H A S U E A E Y V R L V N IC A E SM E EU Y E L V N R Y IC IP E E O L P N H R O IC P S P ME P HIP NTR E N EU R POL LUATIO NTR E OLICY Y E VA SHIP S R E PR E TION E S ME E N EU R S SHIP P E POLIC N R N ENT VALUA E N EU R R E E R U IO IO P P M T P Y E T E E S A E IC R A N U L IP OL LU ENT NTR NTR PR E URSH Y E VA SME P Y E VA S ME E POLIC PR E N E S ME E E NTR E POLIC E NTR E -:HSTCQE=UYUU]X: 11-Dec-2007 10:43:50 AM ... OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT The OECD is a unique forum where the governments of. .. to evaluation and some examples of evaluations undertaken, some of which are better than others in terms of their technical merit OECD Framework for the Evaluation of SME and Entrepreneurship Policies. .. more efficiently, and for the design of new programmes, these seem to be very modest costs indeed OECD Framework for the Evaluation of SME and Entrepreneurship Policies and Programmes – ISBN