PISA 2009 Assessment Framework Key competencies in reading, mathematics and science Programme for International Student Assessment 3 PISA 2009 ASSESSMENT FRAMEWORK – KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 Foreword The OECD Programme for International Student Assessment (PISA), created in 1997, represents a commitment by the governments of OECD member countries to monitor the outcomes of education systems in terms of student achievement, within a common internationally agreed framework. PISA is a collaborative effort, bringing together scientic expertise from the participating countries and steered jointly by their governments on the basis of shared, policy-driven interests. Participating countries take responsibility for the project at the policy level. Experts from participating countries also serve on working groups that are charged with linking the PISA policy objectives with the best available substantive and technical expertise in the eld of internationally comparative assessment. Through involvement in these expert groups, countries ensure that the PISA assessment instruments are internationally valid and take into account the cultural and curricular context of OECD member countries. They also have strong measurement properties, and place an emphasis on authenticity and educational validity. PISA 2009 represents a continuation of the data strategy adopted in 1997 by OECD countries. As in 2000, reading literacy is the focus of the PISA 2009 survey, but the reading framework has been updated and now also includes the assessment of reading of electronic texts. The framework for assessing mathematics was fully developed for the PISA 2003 assessment and remained unchanged in 2009. Similarly, the framework for assessing science was fully developed for the PISA 2006 assessment and remained unchanged in 2009. This publication presents the guiding principles of the PISA 2009 assessment, which are described in terms of the skills students need to acquire, the processes that need to be performed and the contexts in which knowledge and skills are applied. Further, it illustrates the assessment domains with a range of sample tasks. These have been developed by expert panels under the direction of Raymond Adams, Juliette Mendelovits, Ross Turner and Barry McCrae from the Australian Council for Educational Research (ACER) and Henk Moelands (CITO). The reading expert group was chaired by Irwin Kirsch of Educational Testing Service in the United States. The mathematics expert group was chaired by Jan de Lange of the University of Utrecht in the Netherlands and the science expert group was chaired by Rodger Bybee of the Biological Science Curriculum Study in the United States. The questionnaire expert group was chaired by Jaap Scheerens of University of Twente in the Netherlands. The members of the expert groups are listed in Annex C of this publication. The frameworks have also been reviewed by expert panels in each of the participating countries. The chapters on reading, mathematics and science were drafted by the respective expert groups under the direction of their chairs, Irwin Kirsch (reading), Jan de Lange (mathematics) and Rodger Bybee (science). The chapter on the questionnaire framework was drafted by Henry Levin of Teachers College, Columbia University, New York, and is based on a review of central issues, addressed in conceptual papers for the PISA Governing Board, prepared by Jaap Scheerens in collaboration with the questionnaire expert group. The publication was prepared by the OECD Secretariat, principally by Andreas Schleicher, Karin Zimmer, Juliet Evans and Niccolina Clements. The report is published on the responsibility of the Secretary-General of the OECD. Table of conTenTs 5 PISA 2009 ASSESSMENT FRAMEWORK – KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 Table of Contents Foreword 3 Executive Summary 9 Basic features of PISA 2009 11 What makes PISA unique 13 An overview of what is being assessed in each domain 13 Assessing and reporting PISA 2009 15 The context questionnaires and their use 16 Collaborative development of PISA and its assessment framework 17 CHAPTER 1 PISA 2009 Reading Framework 19 Introduction 20 Continuity and change in the reading literacy framework 20 The structure of the reading literacy framework 20 Reading literacy as a foundational skill 21 The importance of electronic texts 22 Motivational and behavioural elements of reading literacy 22 Defining reading literacy 23 Organising the domain 25 Situation 25 Text 27 Aspect 34 Summary of the relationship between printed and electronic reading texts and tasks 43 Assessing reading literacy 45 Building tasks in the print medium 45 Building tasks in the electronic medium 60 Motivational and behavioural constituents of reading literacy 69 Reading engagement 69 Metacognition in reading 72 Reporting proficiency in reading 75 Interpreting and using the data 75 Reporting PISA 2009 reading literacy 76 Conclusion 78 References 80 Table of conTenTs 6 PISA 2009 ASSESSMENT FRAMEWORK – KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 CHAPTER 2 PISA 2009 Mathematics Framework 83 Introduction 84 Definition of the domain 84 Theoretical basis for the PISA mathematics framework 85 Organisation of the domain 90 Situations and context 91 Mathematical content – the four overarching ideas 93 Mathematical processes 105 Assessing mathematics in PISA 116 Task characteristics 116 Assessment structure 119 Aids and tools 120 Reporting proficiency in mathematics 120 Conclusion 122 References 123 CHAPTER 3 PISA 2009 Science Framework 125 Introduction 126 Defining the domain 127 Scientific literacy 128 Organising the domain 129 Situations and context 130 Scientific competencies 137 Scientific knowledge 138 Attitudes towards science 141 Assessing Science in PISA 141 Test characteristics 141 Science assessment structure 142 Reporting proficiency in science 145 Conclusion 146 References 148 Table of conTenTs 7 PISA 2009 ASSESSMENT FRAMEWORK – KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 CHAPTER 4 PISA 2009 Questionnaire Framework 149 Introduction 150 Types of background information and their purposes 151 Educational system as a whole 153 School level 155 Instructional settings 158 Student level 161 Contents of the questionnaires 162 School questionnaire 163 Student questionnaire 163 Parent questionnaire (international option) 163 Questionnaire on educational career (international option) 164 Questionnaire on student familiarity with ICT (international option) 164 Information for in-depth investigations 164 System level indicators 165 Effective learning environments in reading 166 School effectiveness and school management 167 Educational equity 168 References 170 ANNEX A1: Print reading sample tasks 173 ANNEX A2: Electronic reading sample tasks 233 ANNEX B: Background questionnaires 249 ANNEX C: PISA expert groups 289 InTroducTIon 9 PISA 2009 ASSESSMENT FRAMEWORK – KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 Executive Summary Parents, students, teachers, governments and the general public – all stakeholders - need to know how well their education systems prepare students for real-life situations. Many countries monitor students’ learning to evaluate this. Comparative international assessments can extend and enrich the national picture by providing a larger context within which to interpret national performance. They can show what is possible in education, in terms of the quality of educational outcomes as well as in terms of equity in the distribution of learning opportunities. They can support setting policy targets by establishing measurable goals achieved by other systems and help to build trajectories for reform. They can also help countries work out their relative strengths and weaknesses and monitor progress. In response to the need for cross-nationally comparable evidence on student performance, the Organisation for Economic Co-operation and Development (OECD) launched the OECD Programme for International Student Assessment (PISA) in 1997. PISA represents a commitment by governments to monitor the outcomes of education systems through measuring student achievement on a regular basis and within an internationally agreed common framework. It aims to provide a new basis for policy dialogue and for collaboration in dening and implementing educational goals, in innovative ways that reect judgements about the skills that are relevant to adult life. PISA is a collaborative effort undertaken by its participants – the OECD member countries as well as over 30 non-member partner economies – to measure how well students, at age 15, are prepared to meet the challenges they may encounter in future life. Age 15 is chosen because at this age students are approaching the end of compulsory education in most OECD countries. PISA, jointly guided by the participating governments, brings together the policy interests of countries with scientic expertise at both national and international levels. PISA has been measuring the knowledge, skills and attitudes of 15-year-olds over the last ten years and is therefore able to give some insight into how countries are faring over time. The PISA assessment takes a broad approach to measuring knowledge, skills and attitudes that reect current changes in curricula, moving beyond the school-based approach towards the use of knowledge in everyday tasks and challenges. It is based on a dynamic model of lifelong learning in which new knowledge and skills necessary for successful adaptation to a changing world are continuously acquired throughout life. PISA focuses on things that 15-year-old students will need in the future and seeks to assess what they can do with what they have learned – reecting the ability of students to continue learning throughout their lives by applying what they learn in school to non-school environments, evaluating their choices and making decisions. The assessment is informed, but not constrained, by the common denominator of national curricula. Thus, while it does assess students’ knowledge, PISA also examines their ability to reect, and to apply their knowledge and experience to real-life issues. For example, in order to understand and evaluate scientic advice on food safety an adult would need not only to know some basic facts about the composition of nutrients, but also to be able to apply that information. The term Ò literacyÓ is used to encapsulate this broader concept of knowledge and skills, and the PISA assessment aims to determine the extent to which 15-year-old students can activate various cognitive processes that would enable them to make effective use of the reading, mathematical and scientic knowledge and skills they have acquired throughout their schooling and related learning experiences up to that point. PISA is designed to collect information through three-yearly assessments and presents data on domain-specic knowledge and skills in reading, mathematics and science of students, schools and countries. It combines the assessment of science, mathematics and reading with information on students’ home background, their approaches to learning, their learning environments and their familiarity with computers. Student outcomes are then associated with these background factors. Thereby, PISA provides insights into the factors that inuence the development of skills and attitudes at home and at school, and examines how these factors interact and what the implications are for policy development. InTroducTIon 10 PISA 2009 ASSESSMENT FRAMEWORK – KEY COMPETENCIES IN READING, MATHEMATICS AND SCIENCE © OECD 2009 PISA uses: 1) strong quality assurance mechanisms for translation, sampling and test administration; 2) measures to achieve cultural and linguistic breadth in the assessment materials, particularly through countries’ participation in the development and revision processes for the production of the items; and 3) state of the art technology and methodology for data handling. The combination of these measures produces high quality instruments and outcomes with superior levels of validity and reliability to improve the understanding of education systems as well as students’ knowledge, skills and attitudes. This publication presents the theory underlying the PISA 2009 assessment, including a re-developed and expanded framework for reading literacy, which incorporates an innovative component on the capacity to read and understand electronic texts, thus reecting the importance of information and computer technologies in modern societies. It also provides the basis for the assessment of mathematics and science. Within each domain, the knowledge content that students need to acquire is dened, as well as the processes that need to be performed and the contexts in which knowledge and skills are applied. It also illustrates the domains and their aspects with sample tasks. Finally, the theory underlying the context questionnaires is presented. These are used to gather information from students, schools and parents on the students’ home background and attitudes, their learning histories and their learning environments at school. Box A What is PISA? Basics An internationally standardised assessment that was jointly developed by participating countries and administered to 15-year-olds in educational programmes. A survey implemented in 43 countries and economies in the first cycle (32 in 2000 and 11 in 2002), 41 in the second cycle (2003), 57 in the third cycle (2006) and 67 in the fourth cycle (2009). The test is typically administered to between 4 500 and 10 000 students in each country/economy. Content PISA 2009 covers the domains of reading, mathematics and science not merely in terms of whether students can reproduce specific subject matter knowledge, but also whether they can extrapolate from what they have learned and apply their knowledge in novel situations. Emphasis is on the mastery of processes, the understanding of concepts and the ability to function in various situations within each domain. Methods Paper-and-pencil tests are used, with assessments lasting a total of two hours for each student. In a range of countries and economies, an additional 40 minutes are devoted to the assessment of reading and understanding electronic texts. Test items are a mixture of multiple-choice items and questions requiring students to construct their own responses. The items are organised in groups based on a passage setting out a real-life situation. A total of about 390 minutes of test items is covered, with different students taking different combinations of test items.