Improving Learning in South African Schools: The Quality Learning Project (QLP) Summative Evaluation (2000 to 2004) A Kanjee & CH Prinsloo August 2005 HUMAN SCIENCES RESEARCH COUNCIL Free download from www.hsrcpress.ac.za Compiled by the Assessment Technology and Education Evaluation Research Programme, Human Sciences Research Council Funded by the Business Trust Intervention Programme Managed by JET Education Services Evaluation by the Human Sciences Research Council Published by HSRC Press Private Bag X9182, Cape Town, 8000, South Africa www.hsrcpress.ac.za © 2005 Human Sciences Research Council First published 2005 All rights reserved. No part of this book may be reprinted or reproduced or utilised in any form or by any electronic, mechanical, or other means, including photocopying and recording, or in any information storage or retrieval system, without permission in writing from the publishers. ISBN 0-7969-2145-8 Cover Design and Layout: Vinesh Naidoo Production: Shereno Printers Distributed in Africa by Blue Weaver PO Box 30370, Tokai, Cape Town, 7966, South Africa Tel: +27 (0) 21 701 4477 Fax: +27 (0) 21 701 7302 email: orders@blueweaver.co.za www.oneworldbooks.com Distributed in Europe and the United Kingdom by Eurospan Distribution Services (EDS) 3 Henrietta Street, Covent Garden, London, WC2E 8LU, United Kingdom Tel: +44 (0) 20 7240 0856 Fax: +44 (0) 20 7379 0609 email: orders@edspubs.co.uk www.eurospanonline.com Distributed in North America by Independent Publishers Group (IPG) Order Department, 814 North Franklin Street, Chicago, IL 60610, USA Call toll-free: (800) 888 4741 All other enquiries: +1 (312) 337 0747 Fax: +1 (312) 337 5985 email: frontdesk@ipgbook.com www.ipgbook.com Free download from www.hsrcpress.ac.za CONTENTS LIST OF FIGURES AND TABLES vi LIST OF ANNEXURES vii ACKNOWLEDGEMENTS viii PREFACE x ABBREVIATIONS USED FOR THE QLP DISTRICTS xi Map showing location of Quality Learning Project (QLP) districts xi EXECUTIVE SUMMARY xii Background 1 Framework for the Evaluation Study 1 Evaluation Methodology and Research Design 3 Evaluation Criteria 6 Success of the Quality Learning Project 7 Results of the QLP Evaluation 10 District-level Functioning 10 School-level Functioning 11 Classroom-level Functioning 13 Learner Context and Performance 18 Intervention Coverage and Quality 20 Effects of Interventions on Functioning 21 Effects of Improved Functioning on Learner Performance 22 Effects of Interventions on Learner Performance 22 Additional Observations Emanating from the Path Analysis 23 Concluding Statements and Recommendations 24 Recommendations 24 Recommendations on Methodology and Design 24 District-level Recommendations 26 School-level Recommendations 27 Classroom-level Recommendations 28 Causal Modelling Recommendations 28 Conclusion 29 NOTES 37 HUMAN SCIENCES RESEARCH COUNCIL v CONTENTS QLP SUMMATIVE EVALUATION Free download from www.hsrcpress.ac.za LIST OF FIGURES AND TABLES Figures Map showing location of Quality Learning Project (QLP) districts xi Figure A: The QLP model at district, school and teacher level 2 Figure B: Number of schools in the evaluation survey and case-study samples 3 Figure C: District-functionality levels in 2004 and change from 2002 to 2004 10 Figure D: School-functionality levels in 2004 and change from 2002 to 2004 12 Figure E: Grade 9 mathematics classroom-functionality levels in 2004 and change from 2002 to 2004 14 Figure F: Grade 11 mathematics classroom-functionality levels in 2004 and change from 2002 to 2004 14 Figure G: Grade 9 language classroom-functionality levels in 2004 and change from 2002 to 2004 15 Figure H: Grade 11 language classroom-functionality levels in 2004 and change from 2002 to 2004 15 Figure I: National mean mathematics scores for QLP and control schools by year and grade 18 Figure J: National mean language scores for QLP and control schools by year and grade 19 Figure K: Index scores for district-intervention coverage and quality by year 20 Figure L: Index scores for school-intervention coverage and quality by year 21 Figure M: Index scores for teacher-intervention coverage and quality by year, subject and grade 21 Tables Table I: Indicators at Grade 12 level of the success of the QLP (from 2000 to 2004) xiii Table A: Total sample obtained for mid-term and summative evaluations 4 Table B: Number of schools sampled per district 4 Table C: Change in Grade 12 learner performance between 2000 and 2004 across QLP and control schools by province 8 Table D: Indicators at Grade 12 level of the success of the QLP (from 2000 to 2004) 9 HUMAN SCIENCES RESEARCH COUNCIL vi FIGURES AND TABLES QLP SUMMATIVE EVALUATION Free download from www.hsrcpress.ac.za LIST OF ANNEXURES (Graphs comprising findings from the path analyses) Annexure 1: Standardised path weights for modelling Grade 9 mathematics causal patterns 30 Annexure 2: Standardised path weights for modelling Grade 9 language causal patterns 31 Annexure 3: Standardised path weights for modelling Grade 11 mathematics causal patterns 32 Annexure 4: Standardised path weights for modelling Grade 11 language causal patterns 33 Annexure 5: Standardised path weights for modelling matriculation pass rate causal patterns 34 Annexure 6: Standardised path weights for modelling matriculation English pass rate causal patterns 35 Annexure 7: Standardised path weights for modelling Grade 12 (SG) mathematics pass rates 36 HUMAN SCIENCES RESEARCH COUNCIL vii ANNEXURES QLP SUMMATIVE EVALUATION Free download from www.hsrcpress.ac.za ACKNOWLEDGEMENTS The hard work and commitment of many dedicated individuals and organisations have over five years made it possible to bring the formal QLP evaluation to a close by publishing the final, summative report. All contributions are appreciated and are hereby acknowledged. The Business Trust is acknowledged for providing the funding for this project. Acknowledgement is made of JET Education Services and the Business Trust for managing the QLP and for providing valuable support and assistance through the project Steering Committee and other mechanisms. Nick Taylor, Anele Davids, Hemant Waghmarae, Jackie Moyana, and Leigh-May Moses, all from the QLP programme management team of JET Education Services and Charles Barnard, Brian Whittaker, Mdu Ndhlovu and Nomfundo Mqadi, with Theuns Eloff at the outset, from Business Trust, all deserve special mention. They contributed continued guidance throughout the project, with considerable effort and time spent on many rounds of comment to draft versions of instruments and documents of many kinds, and especially the baseline, mid-term and current summative reports. The Department of Education is acknowledged for its continually increasing role in direct communications and meetings between the HSRC and participants at national and provincial level (at the Director General’s office and through strategic planning sessions, respectively). Consortium members and service providers are thanked for valuable inputs at various stages and through feedback sessions, Partners Forum meetings, and otherwise. Acknowledgement is made of the fieldworkers and observers, especially through contracts with AC Nielsen, assisted by Mictert in 2002, and the many qualified teachers, as well as the data-capturing team of Datanet under the guidance of Pio Combrink. Professor Johann Mouton is thanked for his comments and ongoing advice, mainly on methodology, during the first half of the study. Special mention has to be made of the contributions by district managers, regional or circuit managers (institutional development officials), learning area specialists, school principals, teachers, learners and their caregivers, for allowing researchers into their institutions, offices, classrooms and lives, and for making themselves available for observations, interviews, the completion of questionnaires, and for making learners available for sitting for performance tests. The role of and contributions by the official QLP co-ordinators in the office of JET Education Services are also acknowledged. The QLP co-ordinators became an institutionalised channel through which certain business was conducted just so much more efficiently. Access to the districts and schools, and district-level and intervention information collection are two cases in point. [The co-ordinators were: Alfred Mabina (Gauteng); Kedibone Boka (Mpumalanga); Darwin Solomon (Northern Cape); Noel Daniels (Western Cape); Samuel Nkosi (KwaZulu-Natal, Inanda and Ixopo districts); Thulani Dlamini (KwaZulu-Natal, Ubombo district); Rose Machobane (North West Province); Nosipho Nxiweni (Free State); Vuyani Mrwetyana (Eastern Cape); and Maxwell Malatji (Limpopo).] Marcel Croon, professor at Tilburg University in the Netherlands, is thanked for providing invaluable assistance and training in 2002/3, and again in 2004/5, with the data modelling and analysis, and with related software. HUMAN SCIENCES RESEARCH COUNCIL viii ACKNOWLEDGEMENTS QLP SUMMATIVE EVALUATION Free download from www.hsrcpress.ac.za Shereno Printers must be mentioned for printing and packing research instruments within very tight schedules during the three evaluation years, and also for producing the mid-term technical report, this summative report, and its technical companion report. Wordsmiths English Consultancy is acknowledged for language editing, formatting and laying out the manuscript of the mid-term technical report in 2003, and for language editing this summative report and its technical companion report in 2005. Professor H.S. Bhola, Professor Brahm Fleisch, and Hersheela Narsee are thanked for reviewing the final manuscript and for their helpful comments and assistance in improving it. The following HSRC team members (listed alphabetically) are also noted with gratitude for their respective roles as part of the evaluation team at different points in time during the final evaluation phase following 2003: 1 Brutus Malada, Carla Pheiffer, Elsie Venter, Gerda Diedericks, Godwin Khosa, Heidi Paterson, Hendrik de Kock, Lerato Mashego, Lolita Winnaar, Makola Phurutse, Matthews Makgamatha, Natalie le Roux, Nicolaas Claassen, Sannie Reyneke, Sophie Strydom, Vijay Reddy, Xola Mati and Zinhle Kgobe. HUMAN SCIENCES RESEARCH COUNCIL ix ACKNOWLEDGEMENTS QLP SUMMATIVE EVALUATION 1 Participation and contributions during the mid-term and baseline evaluations are acknowledged in the appropriate reports following those periods. Free download from www.hsrcpress.ac.za PREFACE This report marks the end of a unique, long-term and extensive teaching and learning intervention programme, the Quality Learning Project (QLP). In concluding the evaluation activities of the QLP, a reflection on the evaluation processes and findings is desirable. For this reason, some of the most important functions and roles of the QLP and its evaluation are placed in perspective. In doing so, the report emphasises the crucial nature and function of evaluation for teaching and learning, more so in view of the transformation context of the South African education system. At the most apparent and immediate level, this summative report provides a conclusive account to the sponsors of the QLP of how successfully the funds of the project have been spent. Additional value also lies in reflecting more deeply on the complexities inherent in large-scale and lengthy endeavours such as the QLP. These reflections carry a positive verdict about the methods and models selected for the QLP intervention programme and its evaluation. Finally, the reflections allow affirmation of the policy decision implicit in undertaking the QLP work at the outset. In this regard, professional and policy experts can find justification in the soundness, replicability and sustainability of the road travelled by the QLP. This report also provides background information on the interventions as well as on the evaluation design and methodology of the QLP, its findings, and the conclusions and recommendations derived from the findings. HUMAN SCIENCES RESEARCH COUNCIL x PREFACE QLP SUMMATIVE EVALUATION Free download from www.hsrcpress.ac.za ABBREVIATIONS USED FOR THE QLP DISTRICTS Lu = Lusikisiki Fl = Flagstaff Li = Libode ThM = Thabo Mofutsanyana JSM = Johannesburg South Mega SeW = Sedibeng West In = Inanda Ix = Ixopo Ub = Ubombo Mo = Moretele Ma = Mafikeng Ze = Zeerust Ka = Karoo Bo = Bolobedu Ko = Konekwena Zb = Zebediela WCME = Western Cape Metropole East Map showing location of Quality Learning Project (QLP) districts* * District labels appear next to markers indicating the centroid of the particular district. It has to be noted that for certain districts, such as Zeerust and Karoo, actual district areas may be quite large. HUMAN SCIENCES RESEARCH COUNCIL xi ABBREVIATIONS QLP SUMMATIVE EVALUATION Free download from www.hsrcpress.ac.za HUMAN SCIENCES RESEARCH COUNCIL xii EXECUTIVE SUMMARY EXECUTIVE SUMMARY The summative report of the Quality Learning Project (QLP) encapsulates the evaluation work of the project spanning the past five years. In addition, it focuses on the successes and findings of the QLP, and the implications and recommendations flowing from the evaluation study. The details underpinning the summative report can be found in the technical companion report. The success of the QLP The QLP adopted a specific theoretical model for interventions designed to improve learning and teaching in schools, and for evaluating the success of these interventions. As such, the hierarchical levels of the system (districts, schools and classrooms) were taken into account. Observations were made at three points in time to study trends and causal patterns. Comparisons were also made between project and control schools. Performance targets for the QLP were set at the outset. QLP schools were to show an improvement, measured by overall learner performance, against a comparable sample, by the end of 2004. What was required was: • A 10% improvement in mean overall matriculation pass rate; • A 10% improvement in mean mathematics pass rate; and • A 10% improvement in mean English Second Language pass rate. However, pass rates, when used as sole indicators, have certain weaknesses. For example, small increases from low baselines (previous poor matriculation results) appear as large improvements. Moreover, schools are able to artificially inflate Grade 12 pass rates by holding back potentially unsuccessful Grade 11 learners or by requiring learners to take subjects at the Standard Grade (SG). These targets were therefore refined after the 2002 mid-term evaluation, using categories that more reliably reflected school-performance outcomes. These categories were: (a) The increase in the absolute number of learners passing, as an indication of the quantitative improvement of learner results; (b) The increase in the number of learners passing with university exemption, and with mathematics at Higher Grade (HG), rather than Standard Grade (SG), as an indication of an improvement in quality of the learner results; and (c) The increase in matriculation pass rate, as an indication of improved efficiency in learner results. Table I compares the performance of QLP evaluation schools with that of control schools in terms of the final evaluation criteria adopted. It shows that the matriculation results of QLP schools consistently improved more than those of control schools with respect to quantity, quality and efficiency. QLP SUMMATIVE EVALUATION Free download from www.hsrcpress.ac.za [...]... between: (1) the QLP schools in all nine provinces and the control schools in the four provinces; and (2) the control schools in the four provinces and the corresponding QLP schools in these provinces Table C provides information on the change in Grade 12 learner performance, aggregated by province, from 2000 to 2004 in terms of: • The number of learners passing their matriculation examinations (e.g., 53... dramatic in QLP schools, at 332 percentage points (for QLP schools from all nine provinces against control schools in four provinces) and at 924 percentage points (direct comparison across four provinces) higher than control schools However, very low baselines contributed to some inconsistency and to the seemingly large changes The QLP and control schools were rather similar in terms of the increases in. .. points (94% improvement in QLP schools above the 87% in control schools) in direct comparisons made in the four provinces The figure changed to three percentage points in favour of the control schools when the full QLP group in all nine provinces was considered (the 84% improvement in QLP schools is below the 87% in the control schools) Results of the QLP Evaluation Analysis was conducted to determine... to schools is required in the Eastern Cape, and the North West and Limpopo provinces On the index for curriculum leadership, schools in nine of the 17 districts from five provinces fell within the low category, indicating that a great deal of improvement is required in this area In addition, the index for professional development indicated a widespread need for further intervention across schools in. .. QLP schools in all nine provinces and control schools located in the four provinces Such analyses would also be in the spirit of initial intentions to keep a close watch on the improvement in the matriculation results of the schools participating in the QLP as one of the criteria set for evaluating the project’s success Another basis for such a comparison is the initial criteria set for evaluating... in QLP schools above the 15% in control schools) when direct comparisons were made in the four provinces The figure dropped to 11 percentage points when the full QLP group in all nine provinces was considered (26% improvement in QLP schools above the 15% in control schools) In the case of Grade 12 mathematics (SG), improvements in QLP schools exceeded those in control schools by HUMAN SCIENCES RESEARCH... are many indications that service providers targeted interventions dynamically and interactively in areas that needed them most • Interventions improved functioning in areas targeted by the QLP This is evidenced in improved school functioning driven by good classroom and teacher interventions District interventions also played a role in improving school functioning • Improved functioning within the QLP... materials in teaching and learning The control schools seemed to have deteriorated in terms of classroom adequacy and the effect of shortages of classroom materials on teaching and learning, suggesting that the QLP schools have experienced improved physical conditions as a result of the interventions employed at these schools As revealed in the technical report, an increase in the monitoring of curriculum... approximately 17 percentage points higher than those in control schools For learners passing English (HG, Second Language), the difference, when compared across the four provinces that comprised both QLP and control schools, was 36 percentage points in favour of the QLP schools However, when comparing QLP schools in all nine provinces to the control schools from the four provinces, the increase was slightly... of the experimental schools undergoing interventions against the control schools, which did not receive any interventions As a result, the initial disadvantages of the loss of continuity in monitoring programme impact at the mid-point of the programme were turned into advantages for the final summative evaluation phase in 2004 Final 2002 and 2004 evaluation sample figures are shown in Figure B and Tables . entailed improving: • Learning outcomes in the languages of instruction and mathematics in Grades 8 to 12 in 524 schools; • The teaching of mathematics, and reading and writing skills in 524 schools; •. language in learning, given that reading and writing skills, abstract thinking, and producing meaning are central learning objectives. In addition, resource allocation should be prioritised according. Improving Learning in South African Schools: The Quality Learning Project (QLP) Summative Evaluation (2000 to 2004) A Kanjee & CH Prinsloo August 2005 HUMAN SCIENCES