Problem solving for tomorrows world

Programme for International Student Assessment Problem Solving for Tomorrow’s World First Measures of Cross-Curricular Competencies from PISA 2003 OECD ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT ORGANISATION FOR ECONOMIC CO-OPERATION AND DEVELOPMENT Pursuant to Article of the Convention signed in Paris on 14th December 1960, and which came into force on 30th September 1961, the Organisation for Economic Co-operation and Development (OECD) shall promote policies designed: – to achieve the highest sustainable economic growth and employment and a rising standard of living in member countries, while maintaining financial stability, and thus to contribute to the development of the world economy; – to contribute to sound economic expansion in member as well as non-member countries in the process of economic development; and – to contribute to the expansion of world trade on a multilateral, non-discriminatory basis in accordance with international obligations The original member countries of the OECD are Austria, Belgium, Canada, Denmark, France, Germany, Greece, Iceland, Ireland, Italy, Luxembourg, the Netherlands, Norway, Portugal, Spain, Sweden, Switzerland, Turkey, the United Kingdom and the United States The following countries became members subsequently through accession at the dates indicated hereafter: Japan (28th April 1964), Finland (28th January 1969), Australia (7th June 1971), New Zealand (29th May 1973), Mexico (18th May 1994), the Czech Republic (21st December 1995), Hungary (7th May 1996), Poland (22nd November 1996), Korea (12th December 1996) and the Slovak Republic (14th December 2000) The Commission of the European Communities takes part in the work of the OECD (Article 13 of the OECD Convention) Publiộ en franỗais sous le titre : Résoudre des problèmes, un atout pour réussir – Premières évaluations des compétences transdisciplinaires issues de PISA 2003 PISATM, OECD/PISATM and the PISA logo are trademarks of the Organisation for Economic Co-operation and Development (OECD) All use of OECD trademarks is prohibited without written permission from the OECD © OECD 2004 Permission to reproduce a portion of this work for non-commercial purposes or classroom use should be obtained through the Centre franỗais dexploitation du droit de copie (CFC), 20, rue des Grands-Augustins, 75006 Paris, France, tel (33-1) 44 07 47 70, fax (33-1) 46 34 67 19, for every country except the United States In the United States permission should be obtained through the Copyright Clearance Center, Customer Service, (508)750-8400, 222 Rosewood Drive, Danvers, MA 01923 USA, or CCC Online: www.copyright.com All other applications for permission to reproduce or translate all or part of this book should be made to OECD Publications, 2, rue André-Pascal, 75775 Paris Cedex 16, France Foreword Foreword All stakeholders – parents, students, those who teach and run education systems as well as the general public – need to be informed on how well their education systems prepare students for life Knowledge and skills in school subjects such as languages, mathematics and science are an essential foundation for this but a much wider range of competencies is needed for students to be well prepared for the future Problem-solving skills, i.e the capacity of students to understand problems situated in novel and cross-curricular settings, to identify relevant information or constraints, to represent possible alternatives or solution paths, to develop solution strategies, and to solve problems and communicate the solutions, are an example of this wider range of competencies The 2003 assessment of the Organisation for Economic Co-operation and Development’s (OECD) Programme for International Student Assessment (PISA) included an assessment of students’ problem-solving skills, providing for the first time a direct assessment of life competencies that apply across different areas of the school curriculum About one in five 15-year-olds in OECD countries can be considered a reflective, communicative problem solver.These students are able not only to analyse a situation and make decisions, they are also capable of managing multiple conditions simultaneously They can think about the underlying relationships in a problem, solve it systematically, check their work and communicate the results In some countries, more than a third of students reach this high level of problem-solving competencies In other countries, however, the majority of students cannot even be classified as basic problem solvers, a level at which they are required to deal with only a single data source containing discrete, well-defined information How can countries raise their performance in this increasingly important competency area and what can countries with lower performance levels learn from those where students well? This report seeks to answer such questions It complements Learning for Tomorrow’sWorld – First Results from PISA 2003, which focuses on knowledge and skills in mathematics, science and reading, and it goes beyond an examination of the relative standing of countries in students’ problemsolving performance by considering how problem-solving performance relates to learning outcomes in other areas and how it varies between the genders and between socio-economic groups It also provides insights into some of the factors that are associated with the development of problem-solving skills and into how these factors interact and what the implications are for policy development Most importantly, the report sheds light on countries that succeed © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Foreword in achieving high performance standards while at the same time providing an equitable distribution of learning opportunities Results in these countries pose challenges for other countries by showing what it is possible to achieve The report is the product of a collaborative effort between the countries participating in PISA, the experts and institutions working within the framework of the PISA Consortium, and the OECD The report was drafted by John Dossey, Johannes Hartig, Eckhard Klieme and Margaret Wu, under the direction of the OECD Directorate for Education, principally by Claire Shewbridge and Andreas Schleicher, with advice and analytic support from Raymond Adams, Barry McCrae and Ross Turner The PISA problem-solving framework and assessment instruments were prepared by the PISA Consortium and PISA Problem Solving Expert Group under the direction of Raymond Adams at the Australian Council for Educational Research Data analytic support was provided by Alla Berezener, Johannes Hartig and Margaret Wu The development of the report was steered by the PISA Governing Board, which is chaired by Ryo Watanabe (Japan) Annex C of the report lists the members of the various PISA bodies as well as the individual experts and consultants who have contributed to this report and to PISA in general The report is published on the responsibility of the Secretary-General of the OECD Ryo Watanabe Chair of the PISA Governing Board © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Barry McGaw Director for Education, OECD CHAPTER PISA 2003 AND PROBLEM SOLVING 11 Table of Contents Table of Contents Introduction 12 Problem solving in PISA 2003 16 Organisation of this report 20 READERS’ GUIDE 22 CHAPTER PROBLEM SOLVING IN PISA 2003 – HOW IT WAS MEASURED AND HOW STUDENTS PERFORMED 25 Introduction 26 Problem solving in PISA 26 Organisation of the assessment area 27 Problems chosen for the PISA problem-solving assessment 28 The PISA problem-solving scale 28 • Level 3: Reflective, communicative problem solvers 29 • Level 2: Reasoning, decision-making problem solvers 30 • Level 1: Basic problem solvers 30 • Below Level 1: Weak or emergent problem solvers 30 • Decision making – the Cinema Outing problem 32 • System analysis and design – the Children’s Camp problem 34 • Trouble shooting – the Irrigation problem 36 The percentage of students at each proficiency level of problem solving 39 • Mean performance of countries 41 The distribution of problem-solving capabilities within countries 44 Implications for policy 46 CHAPTER STUDENT PERFORMANCE IN PROBLEM SOLVING COMPARED WITH PERFORMANCE IN MATHEMATICS, READING AND SCIENCE 49 Introduction 50 Problem-solving framework and test development 50 • Emphasis on problem-solving processes 50 • Low content requirements 51 • The key skills tested in problem solving 51 • Correlations between performance in reading, mathematics, science and problem solving 54 © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Table of Contents Comparison between performances in mathematics and problem solving at the country level 55 Implications for policy 57 CHAPTER STUDENT PERFORMANCE ON THE PROBLEM-SOLVING ITEMS 59 Introduction 60 Decision-making units 62 • Energy Needs 62 • Cinema Outing 67 • Holiday 70 • Transit System 73 System analysis and design units 76 • Library System 76 • Design by Numbers© 82 • Course Design 88 • Children’s Camp 91 Trouble-shooting units 94 • Irrigation 94 • Freezer 98 Summary 101 CHAPTER THE ROLE THAT GENDER AND STUDENT BACKGROUND CHARACTERISTICS PLAY IN STUDENT PERFORMANCE IN PROBLEM SOLVING .103 Introduction 104 Gender differences in problem solving 104 Comparison with gender differences in other assessment areas 107 Parental occupational status 110 Parental education .112 Possessions related to “classical” culture 113 Family structure 115 Place of birth and language spoken at home 116 Implications for policy 119 REFERENCES 121 ANNEX A .123 Annex A1 Construction of indices and other derived measures from the student context questionnaire 124 Annex A2 Detailed results from the factor analysis in Chapter 126 Annex A3 The PISA target population and the PISA samples .128 © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Annex A6 Development of the PISA assessment instruments 139 Annex A7 Reliability of the marking of open-ended items .141 ANNEX B Table of Contents Annex A4 Standard errors, significance tests and subgroup comparisons 137 Annex A5 Quality assurance 138 Data tables for the chapters .143 ANNEX C The development and implementation of PISA – a collaborative effort 157 © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Table of Contents LIST OF BOXES Box 1.1 Key features of the PISA 2003 assessment 15 Box 2.1 Interpreting sample statistics 43 LIST OF FIGURES Figure 1.1 A map of PISA countries 14 Figure 2.1 Figure 2.2 Figure 2.3 Figure 2.4 Figure 2.5 Features of the three types of problem solving 29 The PISA problem-solving scale 31 Percentage of students at each level of proficiency on the problem-solving scale 41 Multiple comparisons of mean performance on the problem-solving scale 42 Distribution of student performance on the problem-solving scale 45 Figure 3.1 Figure 3.2 Figure 3.3 Analysis of two dominant factors in student performance on the problem-solving, reading and mathematics items 52-53 Latent correlations between the four assessment areas 55 Difference between student performance in mathematics and problem solving 56 Figure 4.1 Figure 4.2 Figure 4.3 Figure 4.4 Figure 4.5 Figure 4.6 Figure 4.7 Figure 4.8 Figure 4.9 Figure 4.10 Figure 4.11 Figure 4.12 Figure 4.13 Problem-solving units and their characteristics 61 Full credit student work on Energy Needs, Question 65 Partial credit student work on Energy Needs, Question – example 66 Partial credit student work on Energy Needs, Question – example 66 No credit student work on Energy Needs, Question 66 Partial credit solution for Transit System (Response Coding Code 11) 75 Example of full credit response to Library System, Question 80 Partial credit solution for Library System, Question (Response Code 11) 81 Example of full credit response for Design by Numbers©, Question 86 Example of partial credit response for Course Design, Question 90 Example of full credit response for Children’s Camp, Question 93 Example of partial credit response for Children’s Camp, Question 93 Graph of PISA problem-solving item scale values by problem type 101 Figure 5.1 Figure 5.2 Figure 5.3 Figure 5.4 Figure 5.5 Figure 5.6 Figure 5.7 Figure 5.8 Figure 5.9 Figure 5.10 Gender differences in student performance in problem solving 105 Percentage of males and females performing below Level and at Level in problem solving 106 Gender differences in problem solving and in mathematics 108 Gender differences in problem solving and in reading 109 Parental occupational status and student performance in problem solving 111 Parental education and student performance in problem solving 113 Cultural possessions and student performance in problem solving 114 Type of family structure and student performance in problem solving 115 Place of birth and student performance in problem solving 117 Home language and student performance in problem solving 118 LIST OF TABLES Table A2.1 Table A2.2 Table A3.1 Table A3.2 Eigenvalues of the first 12 factors and total variance explained 126 Component correlation matrix 126 PISA target populations and samples 129-130 Exclusions 132 © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Response rates 135 Table 2.1 Table 2.2 Percentage of students at each level of proficiency on the problem-solving scale 144 Mean score and variation in student performance on the problem-solving scale 145 Table 3.1 Table 3.2 Factor loadings of mathematics, reading and problem-solving items 146 Difference between mean scores in mathematics and problem solving 147 Table 5.1 Gender differences in mean score in student performance on the problem-solving, mathematics and reading scales and percentage of males and females below Level and at Level of the problem-solving scale 148 International socio-economic index of occupational status (HISEI) and performance on the problem-solving scale, by national quarters of the index 149 Index of highest educational level of parents (HISCED) and performance on the problem-solving scale, by national quarters of the index 150 Index of possessions related to “classical” culture in the family home and performance on the problem-solving scale, by national quarters of the index 151 Percentage of students and performance on the problem-solving scale, by type of family structure 152 Percentage of students and performance on the problem-solving scale, by students’ nationality and the nationality of their parents 153 Percentage of students and performance on the problem-solving scale, by language spoken at home 154 Table 5.2 Table 5.3 Table 5.4 Table 5.5 Table 5.6 Table 5.7 © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Table of Contents Table A3.3 PISA 2003 and Problem Solving Introduction 12 Problem solving in PISA 2003 16 Organisation of this report 20 © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 11 Partner countries OECD countries Mean score Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 Standard deviation 5th percentile 10th percentile 25th percentile 75th percentile 90th percentile 95th percentile Mean S.E S.D S.E Score S.E Score S.E Score S.E Score S.E Score S.E Score S.E 530 506 525 529 516 517 548 519 513 448 501 505 498 469 547 550 494 384 520 533 490 487 470 492 482 509 521 408 477 490 500 371 548 361 483 529 532 479 420 425 345 411 m (2.0) (3.2) (2.2) (1.7) (3.4) (2.5) (1.9) (2.7) (3.2) (4.0) (2.9) (1.4) (2.3) (3.1) (4.1) (3.1) (1.4) (4.3) (3.0) (2.2) (2.6) (2.8) (3.9) (3.4) (2.7) (2.4) (3.0) (6.0) (3.1) (1.2) (0.6) (4.8) (4.2) (3.3) (3.9) (3.9) (2.5) (4.6) (3.3) (2.7) (2.1) (3.7) m 91 90 104 88 93 87 82 93 95 99 94 85 80 102 105 86 92 96 89 96 99 90 92 93 94 88 94 97 98 106 100 100 97 73 92 93 81 99 86 82 80 112 m (1.4) (1.7) (1.5) (0.9) (1.9) (1.5) (1.2) (2.1) (1.8) (1.7) (2.0) (1.1) (1.4) (2.1) (2.7) (2.0) (1.0) (2.0) (2.0) (1.2) (1.7) (1.7) (2.1) (2.4) (1.3) (1.6) (1.9) (4.4) (1.3) (0.8) (0.4) (2.6) (2.9) (1.7) (1.7) (4.2) (2.6) (2.1) (1.6) (1.6) (1.4) (1.9) m 371 357 340 379 356 369 409 358 351 283 343 358 364 289 362 404 339 227 372 370 322 338 311 337 322 360 358 257 312 308 328 211 376 245 326 369 395 314 279 293 213 224 m (4.1) (5.1) (5.0) (2.4) (8.6) (5.0) (4.7) (6.1) (5.9) (5.6) (5.8) (5.5) (4.5) (8.7) (8.3) (4.6) (3.7) (5.4) (5.9) (3.8) (5.5) (5.6) (7.9) (7.1) (4.8) (6.4) (5.7) (7.8) (5.6) (2.7) (1.7) (7.5) (10.5) (4.2) (7.0) (14.9) (6.4) (7.7) (4.2) (3.9) (4.3) (5.7) m 409 388 383 414 394 402 442 396 383 319 378 393 395 334 406 438 373 262 401 406 361 372 345 370 361 395 397 291 347 348 368 244 420 270 362 404 425 351 311 322 243 265 m (3.5) (4.5) (4.5) (2.8) (6.2) (4.3) (2.8) (4.8) (5.3) (5.3) (4.1) (3.3) (3.8) (6.5) (6.8) (5.2) (2.3) (5.2) (5.1) (4.2) (4.6) (4.1) (6.8) (5.9) (4.1) (4.4) (4.0) (6.6) (4.6) (2.2) (1.3) (6.1) (7.9) (3.8) (6.0) (11.1) (5.6) (7.0) (4.4) (3.4) (3.1) (5.1) m 469 443 456 471 454 459 495 459 447 383 436 450 445 406 481 494 432 317 456 468 424 428 409 430 421 451 461 343 410 418 434 302 487 312 420 468 478 413 363 369 291 334 m (2.8) (4.1) (3.3) (2.5) (4.4) (3.1) (2.5) (3.9) (4.8) (4.5) (3.8) (2.2) (3.1) (4.7) (5.7) (3.9) (2.4) (5.2) (4.9) (3.7) (3.7) (3.1) (5.7) (4.7) (3.5) (3.0) (3.3) (5.2) (4.1) (1.7) (1.1) (4.7) (6.1) (3.6) (5.4) (6.0) (3.7) (5.7) (3.9) (2.6) (2.5) (4.7) m 594 569 602 591 582 578 604 586 583 517 567 564 555 540 621 610 558 451 587 601 559 548 534 558 547 571 587 466 548 566 571 438 617 409 547 599 590 546 478 478 400 488 m (2.1) (4.0) (2.6) (1.9) (3.6) (2.8) (2.3) (3.0) (4.3) (4.6) (3.9) (2.0) (2.7) (3.0) (4.2) (3.5) (2.2) (5.1) (3.6) (2.4) (3.3) (3.0) (3.6) (3.6) (3.2) (3.1) (3.9) (7.7) (3.3) (1.3) (0.8) (5.7) (3.2) (4.1) (4.6) (9.3) (4.3) (5.1) (4.2) (4.0) (2.8) (5.5) m 644 621 653 640 634 627 650 635 632 574 622 609 601 595 675 658 610 509 636 653 615 600 586 609 599 619 637 531 604 624 625 501 664 457 599 644 633 604 530 532 446 552 m (2.7) (4.2) (2.0) (2.1) (3.9) (3.4) (2.3) (3.7) (2.7) (5.7) (4.3) (2.3) (2.8) (3.4) (4.6) (4.2) (2.6) (5.7) (3.3) (2.5) (4.2) (3.5) (3.5) (3.8) (3.9) (3.8) (4.6) (11.9) (4.0) (1.3) (0.8) (7.3) (2.9) (5.5) (4.1) (10.5) (5.4) (5.0) (4.9) (4.0) (4.1) (5.0) m 672 651 681 669 663 655 677 662 658 607 653 634 625 627 705 686 640 542 662 682 645 632 614 638 629 647 666 577 635 656 656 538 690 487 628 672 659 637 560 565 474 589 m (3.4) (4.6) (2.0) (2.4) (4.0) (3.7) (3.6) (4.5) (3.2) (5.6) (5.4) (3.6) (3.2) (3.6) (6.0) (5.5) (3.4) (6.5) (3.7) (2.8) (4.4) (4.5) (3.5) (4.2) (3.3) (3.6) (5.2) (18.6) (4.2) (1.4) (0.8) (8.3) (3.7) (5.9) (4.9) (12.0) (6.5) (5.6) (5.1) (6.0) (5.0) (5.7) m Annex B Table 2.2 Mean score and variation in student performance on the problem-solving scale Response rate too low to ensure comparability (see Annex A3) © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 145 Annex B Table 3.1 Factor loadings of mathematics, reading and problem-solving items The strength of association of each PISA item with two different presumed factors, calculated from students’ responses Problem-solving item Items Mathematics item Item loads higher on factor than on factor Factor Factor Loading Loading Items Reading item Item loads higher on factor than on factor Factor Factor Factor Factor Loading Loading Items Loading Loading BRAILLE – Question 0.400 THE BEST CAR – Question 0.279 DRUGGED SPIDERS – Question –0.136 0.642 HOLIDAY – Question 0.393 CASH WITHDRAWAL – Question 0.278 0.106 OPTICIAN – Question –0.100 0.609 SKATEBOARD – Question 13 0.391 CARBON DIOXIDE – Question 0.276 OPTICIAN – Question POPULATION PYRAMIDS – Question 0.380 CARPENTER – Question 0.275 EXCHANGE – Question BICYCLES – Question 0.372 CARBON DIOXIDE – Question 0.271 EXCHANGE – Question –0.110 0.557 BOOKSHELVES – Question 0.368 CHILDREN'S CAMP – Question 0.271 TELEPHONE – Question –0.117 0.556 NUMBER CUBES – Question 0.368 SEEING THE TOWER – Question 0.265 DRUGGED SPIDERS – Question POPULATION PYRAMIDS – Question 0.359 HEIGHT – Question 0.261 CONTAINERS – Question 0.356 CHAIR LIFT – Question 0.258 DRUGGED SPIDERS – Question CUBES – Question 0.355 CHAIR LIFT – Question 0.256 OPTICIAN – Question 0.494 SUPPORT FOR PRESIDENT – Question 0.353 NUMBER CHECK – Question 0.255 TELEPHONE – Question 0.487 CARBON DIOXIDE – Question 0.350 PIPELINES – Question 0.254 OPTICIAN – Question 0.371 LABELS – Question 0.348 RUNNING TRACKS – Question 0.252 THERMOMETER CRICKET – Question 0.347 A VIEW ROOM – Question 0.250 CASH WITHDRAWAL – Question 0.346 RUNNING TRACKS – Question 0.246 IRRIGATION – Question 0.346 THERMOMETER CRICKET – Question THE FENCE – Question 0.346 ROBBERIES – Question 15 0.188 0.171 0.576 0.576 0.541 DRUGGED SPIDERS – Question 0.521 –0.119 TELEPHONE – Question 0.521 0.324 SOUTH POLE – Question 0.102 0.291 SOUTH POLE – Question 0.125 0.279 0.236 SOUTH POLE – Question 0.169 0.261 CINEMA OUTING – Question 0.234 SOUTH POLE – Question 0.106 0.257 0.345 DESIGN BY NUMBERS© – Question 0.234 SOUTH POLE – Question TILE ARRANGEMENT – Question 0.342 STEP PATTERN – Question 0.233 EMPLOYMENT – Question 0.150 0.248 BRICKS – Question 0.338 THIRD SIDE – Question 0.232 EMPLOYMENT – Question (E) 0.174 0.235 SCIENCE TESTS – Question 0.337 LIBRARY SYSTEM – Question 0.232 SHIRTS – Question 0.106 0.234 COLOURED CANDIES – Question 0.336 CAR DRIVE – Question 0.223 EMPLOYMENT – Question 0.121 0.233 SKATEBOARD – Question 12 0.335 LITTER – Question 0.221 SHIRTS – Question 0.153 0.230 WALKING – Question 0.334 STOP THE CAR – Question 0.221 0.142 EXCHANGE RATE – Question 10 0.201 0.227 CAR DRIVE – Question 0.333 DIVING – Question 0.216 0.193 GROWING UP – Question 0.181 0.223 COURSE DESIGN – Question 0.328 CUBE PAINTING – Question 0.215 0.122 EXCHANGE RATE – Question 0.165 0.217 IRRIGATION – Question 0.321 STAIRCASE – Question 0.213 SHIRTS – Question 0.145 0.200 LIBRARY SYSTEM – Question 0.318 RUNNING TRACKS – Question 0.213 GROWING UP – Question 0.182 0.196 POPULATION PYRAMIDS – Question 0.315 DESIGN BY NUMBERS© – Question 0.207 EXCHANGE RATE – Question 11 0.193 0.193 IRRIGATION – Question 0.313 HEIGHT – Question 0.207 THE BEST CAR – Question 0.173 0.187 HOLIDAY – Question 0.310 TOSSING COINS – Question 0.205 AESOP – Question BICYCLES – Question 0.308 CHOICES – Question 0.204 DIVING – Question 0.153 0.172 TRANSPORT – Question 0.308 CUBE PAINTING – Question 0.202 0.126 MAKING A BOOKLET – Question 0.141 0.147 EARTHQUAKE – Question 0.305 GROWING UP – Question 0.201 0.198 COMPUTER GAME – Question LOTTERIES – Question 0.304 CUBE PAINTING – Question 0.197 0.169 INTERNET RELAY CHAT – Question 0.303 MAP – Question 0.196 ENERGY NEEDS – Question 0.303 TELEPHONE RATES – Question 0.194 WALKING – Question 0.301 CINEMA OUTING – Question 0.189 TEST SCORES – Question 16 0.298 FORECAST OF RAIN – Question 0.189 SKATEBOARD – Question 14 0.298 FREEZER – Question 0.188 NUMBER CHECK – Question 0.297 CUBE PAINTING – Question 0.184 POPULATION PYRAMIDS – Question 0.295 EXPORTS – Question 17 0.176 TRANSIT SYSTEM – Question 0.292 FREEZER – Question 0.170 INTERNET RELAY CHAT – Question 0.287 HEIGHT – Question 0.160 BICYCLES – Question 0.286 RUNNING TIME – Question 0.160 DESIGN BY NUMBERS© – Question 0.285 ENERGY NEEDS – Question 0.157 EXPORTS – Question 18 0.281 CAR DRIVE – Question 0.150 0.160 0.145 0.254 0.179 0.128 0.133 0.130 Note: The items are arranged in order of the magnitudes of the factor loadings Columns and show items where the factor loading is higher than the factor loading for an item, and the items are arranged in descending order of the factor loadings Column shows items where the factor loading is higher than the factor loading for an item, and the items are arranged in descending order of factor loadings Source: OECD, PISA 2003 database 146 © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Partner countries OECD countries (Math – P.S.) S.E Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States –5.6 –0.5 4.0 3.2 0.0 –2.5 –3.3 –8.4 –10.5 –3.6 –11.1 10.5 4.4 –3.8 –13.1 –8.2 –0.4 0.8 17.6 –9.3 5.4 3.7 –3.8 6.4 2.8 0.5 5.3 15.9 5.5 (0.70) (0.78) (0.78) (0.53) (0.85) (1.18) (0.68) (1.01) (0.91) (1.33) (0.82) (0.98) (0.79) (1.22) (1.06) (0.79) (1.13) (1.30) (1.33) (1.01) (1.24) (1.20) (0.88) (0.99) (0.97) (1.27) (0.83) (1.27) (0.62) Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 –15.0 2.5 –1.3 0.9 6.4 –5.2 –10.2 16.7 –7.9 14.0 11.6 m (0.99) (1.20) (1.22) (1.33) (2.80) (1.77) (1.35) (1.30) (1.23) (1.23) (1.41) m Annex B Table 3.2 Difference between mean scores in mathematics and problem solving Note: Values that are statistically significant are indicated in bold (see Annex A4) Response rate too low to ensure comparability (see Annex A3) © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 147 Annex B Table 5.1 Gender differences in mean score in student performance on the problem-solving, mathematics and reading scales and percentage of males and females below Level and at Level of the problem-solving scale Problem solving Males OECD countries Mean score Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Partner countries OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 Females Standard deviation Mean score Standard deviation S.E S.D S.E Mean S.E S.D S.E Score dif S.E % S.E 527 505 522 533 520 519 543 519 511 450 499 490 499 467 546 554 495 387 522 531 486 486 470 495 479 504 520 408 477 489 499 374 545 358 481 535 538 480 416 418 346 412 m (2.7) (3.9) (3.1) (2.0) (4.1) (3.1) (2.5) (3.8) (3.9) (4.9) (3.4) (2.2) (2.8) (5.0) (5.7) (4.0) (2.4) (5.0) (3.6) (2.6) (3.1) (3.4) (4.6) (4.1) (3.6) (3.0) (4.0) (7.3) (3.4) (1.4) (0.8) (6.0) (6.2) (3.1) (5.1) (6.6) (4.2) (5.9) (3.8) (3.9) (2.5) (4.6) m 95 94 108 93 95 88 87 97 96 104 96 89 81 110 111 88 95 97 89 99 103 96 99 95 98 90 96 102 101 109 103 106 104 72 97 98 85 104 91 84 80 116 m (1.5) (2.1) (1.9) (1.1) (2.1) (1.8) (1.5) (2.9) (2.3) (2.3) (2.3) (1.5) (1.9) (3.1) (3.6) (2.3) (1.6) (2.6) (2.2) (1.6) (2.0) (2.4) (2.5) (2.8) (1.8) (2.0) (2.6) (5.2) (1.6) (1.1) (0.6) (3.4) (3.8) (1.8) (2.8) (5.8) (4.2) (2.8) (2.0) (2.3) (1.9) (2.4) m 533 508 527 532 513 514 553 520 517 448 503 520 498 471 548 546 493 382 518 534 494 487 470 488 485 514 523 406 478 490 501 368 550 365 484 524 527 477 424 431 343 409 m (2.5) (3.8) (3.2) (1.8) (4.3) (2.9) (2.2) (2.9) (3.7) (4.1) (3.4) (2.5) (3.5) (3.5) (4.1) (4.8) (1.9) (4.7) (3.6) (3.1) (3.2) (3.0) (3.9) (3.6) (2.6) (2.8) (3.3) (5.8) (3.5) (1.3) (0.8) (4.3) (4.0) (4.0) (4.0) (5.9) (3.2) (4.4) (3.9) (3.1) (2.5) (4.2) m 87 86 101 84 91 86 77 89 92 94 92 78 78 94 99 84 88 95 89 92 94 84 86 90 89 86 92 89 95 103 97 95 90 74 87 87 77 93 80 80 79 108 m (1.8) (2.3) (2.2) (1.2) (2.6) (2.2) (1.4) (2.1) (2.2) (1.9) (2.6) (1.6) (1.7) (2.0) (3.0) (2.6) (1.6) (2.6) (2.6) (1.7) (2.7) (1.8) (2.1) (2.6) (1.5) (2.1) (1.8) (4.1) (1.9) (0.8) (0.5) (2.4) (2.6) (2.1) (2.2) (5.2) (2.9) (2.0) (2.2) (2.0) (1.6) (2.4) m –6.40 –2.88 –3.49 0.49 6.53 4.90 –9.99 –0.78 –5.71 1.94 –3.71 –30.46 0.52 –4.06 –2.41 8.15 2.37 5.08 4.45 –3.27 –8.46 –1.07 0.01 6.93 –6.04 –9.90 –2.46 2.01 –0.95 –0.67 –1.71 5.21 –5.06 –7.30 –2.57 11.52 11.22 2.30 –7.39 –12.37 2.71 2.73 m (3.3) (4.3) (4.5) (2.1) (5.0) (3.2) (3.0) (4.1) (3.9) (4.4) (3.7) (3.9) (4.2) (6.0) (5.7) (6.1) (3.3) (4.5) (4.1) (3.8) (3.6) (3.1) (3.5) (3.7) (3.1) (3.1) (4.1) (5.8) (3.0) (1.5) (0.8) (3.7) (6.3) (3.0) (4.6) (9.8) (5.5) (4.9) (4.1) (4.3) (2.6) (4.8) m 0.1 0.0 0.0 0.0 0.1 0.1 0.4 0.0 0.1 0.0 0.0 3.2 0.0 0.0 0.0 0.2 0.0 0.1 0.1 0.0 0.2 0.0 0.0 0.1 0.1 0.3 0.0 0.0 0.0 0.0 0.0 0.1 0.1 0.3 0.0 0.4 0.5 0.0 0.2 0.6 0.0 0.0 m (0.1) (0.1) (0.1) (0.0) (0.2) (0.1) (0.2) (0.0) (0.1) (0.1) (0.1) (0.8) (0.1) (0.1) (0.1) (0.3) (0.1) (0.1) (0.1) (0.1) (0.2) (0.0) (0.0) (0.2) (0.1) (0.2) (0.1) (0.1) (0.0) (0.0) (0.0) (0.1) (0.2) (0.2) (0.1) (0.7) (0.5) (0.1) (0.2) (0.4) (0.1) (0.1) m Percentage of males Mean score Partner countries OECD countries Below Level (below 405 score points) 148 Explained variance in student performance (r-squared x 100) Mean Problem solving S.E Percentage of females Standard deviation At Level (above 592 score points) S.E Mean score Standard deviation Below Level (below 405 score points) S.E 10.8 (0.7) 25.7 (1.2) 7.5 15.0 (1.2) 18.5 (1.5) 11.7 14.6 (1.1) 28.9 (1.3) 12.0 9.2 (0.6) 27.9 (0.9) 6.8 11.7 (1.3) 23.5 (1.7) 12.0 10.2 (0.9) 21.0 (1.3) 10.4 5.9 (0.8) 29.3 (1.2) 3.0 13.0 (1.4) 24.0 (1.3) 10.1 14.9 (1.3) 21.7 (1.6) 12.7 33.3 (1.9) 8.4 (1.1) 31.4 16.9 (1.3) 17.2 (1.4) 14.7 17.1 (1.1) 12.2 (0.9) 6.8 12.6 (1.2) 13.2 (0.9) 11.9 26.6 (2.1) 12.1 (1.0) 22.5 11.3 (1.3) 36.5 (2.3) 8.4 5.2 (0.8) 34.6 (1.7) 4.7 17.7 (1.1) 16.1 (0.9) 15.8 57.2 (2.1) 1.6 (0.4) 58.2 9.8 (1.2) 23.7 (1.4) 11.2 10.9 (0.9) 28.8 (1.1) 8.6 20.8 (1.1) 14.8 (1.1) 17.4 18.8 (1.2) 13.0 (0.9) 15.6 25.6 (2.0) 10.1 (1.0) 21.8 17.1 (1.6) 16.0 (1.2) 17.3 21.7 (1.3) 12.2 (1.2) 17.8 12.8 (1.1) 16.2 (1.2) 10.8 11.9 (1.1) 23.4 (2.0) 10.4 50.1 (2.9) 4.6 (1.5) 51.4 24.1 (1.3) 13.2 (1.1) 22.6 22.2 (0.5) 18.1 (0.5) 20.5 OECD total 18.0 (0.3) 19.0 (0.2) 16.0 OECD average 62.6 (2.3) 2.4 (0.7) 64.7 Brazil 9.8 (1.6) 36.1 (2.1) 5.9 Hong Kong-China 75.2 (1.8) 0.1 (0.1) 70.9 Indonesia 21.7 (1.9) 12.7 (1.3) 18.5 Latvia 9.8 (2.4) 31.2 (3.8) 10.0 Liechtenstein 6.3 (1.3) 27.7 (2.7) 6.2 Macao-China 23.7 (2.2) 14.0 (1.5) 21.3 Russian Federation 44.6 (2.0) 2.7 (0.6) 39.6 Serbia 45.1 (2.1) 2.5 (0.5) 37.2 Thailand 76.3 (1.3) 0.1 (0.1) 77.0 Tunisia 46.2 (1.9) 5.5 (0.8) 47.4 Uruguay m m m m m United Kingdom1 Note: Values that are statistically significant are indicated in bold (see Annex A4) Response rate too low to ensure comparability (see Annex A3) Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Gender difference (0.8) (1.3) (1.1) (0.5) (1.5) (1.1) (0.4) (1.0) (1.2) (1.8) (1.1) (0.9) (1.3) (1.5) (1.2) (0.9) (1.1) (2.0) (1.5) (1.0) (1.2) (1.3) (1.8) (1.6) (0.8) (1.0) (0.9) (2.8) (1.3) (0.5) (0.3) (1.9) (0.8) (2.2) (1.7) (2.5) (1.2) (1.6) (2.0) (1.7) (1.3) (2.0) m At Level (above 592 score points) 25.7 16.1 27.7 24.1 19.5 19.4 31.0 21.7 21.9 5.8 17.3 17.9 11.7 9.2 34.9 29.5 12.5 1.0 22.5 27.9 15.1 10.5 7.3 12.1 11.2 18.7 22.8 3.2 11.6 16.5 17.6 0.9 33.9 0.2 10.7 22.7 21.1 10.6 1.5 2.7 0.1 4.0 m © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Reading Gender difference in reading (M – F) Mathematics Gender difference in mathematics (M – F) Gender difference Gender difference S.E Score difference S.E Score difference S.E (1.0) (1.4) (1.1) (0.8) (1.5) (1.4) (1.2) (1.2) (1.8) (0.9) (1.4) (1.0) (1.2) (0.7) (1.5) (2.2) (0.7) (0.3) (1.4) (1.4) (1.2) (0.8) (1.1) (1.0) (0.8) (1.3) (1.4) (1.1) (0.9) (0.4) (0.2) (0.3) (1.8) (0.1) (1.1) (3.5) (2.3) (1.1) (0.3) (0.7) (0.1) (0.7) m -39 -47 -37 -32 -31 -25 -44 -38 -42 -37 -31 -58 -29 -39 -22 -21 -33 -21 -21 -28 -49 -40 -36 -33 -39 -37 -35 -33 -32 -31 -34 -35 -32 -24 -39 -17 -13 -29 -43 -43 -25 -39 m (3.6) (5.2) (5.1) (2.0) (4.9) (2.9) (2.7) (4.5) (4.6) (4.1) (3.8) (3.5) (4.6) (6.0) (5.4) (5.6) (3.4) (4.4) (3.9) (4.4) (3.7) (3.7) (3.3) (3.5) (3.9) (3.2) (4.7) (5.8) (3.3) (1.4) (0.8) (3.9) (5.5) (2.8) (4.2) (11.9) (4.8) (3.9) (3.9) (4.1) (3.6) (4.7) m 8 11 15 17 9 19 -15 15 18 23 17 11 14 6 12 19 17 15 10 11 16 3 29 21 10 -4 12 12 m (3.8) (4.4) (4.8) (2.1) (5.1) (3.2) (2.7) (4.2) (4.4) (3.6) (3.5) (3.5) (4.2) (5.9) (5.9) (6.8) (2.8) (3.9) (4.3) (3.9) (3.2) (3.1) (3.3) (3.7) (3.0) (3.3) (4.9) (6.2) (2.9) (1.4) (0.8) (4.1) (6.6) (3.4) (4.0) (10.9) (5.8) (4.4) (4.4) (4.2) (2.5) (4.2) m International socio-economic index of occupational status (highest of the father’s or mother’s) All students OECD countries Mean index Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Partner countries OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 52.6 47.1 50.6 52.6 50.1 49.3 50.2 48.7 49.3 46.9 48.6 53.7 48.3 46.8 50.0 46.3 48.2 40.1 51.3 51.5 54.6 45.0 43.1 48.8 44.3 50.6 49.3 41.6 54.6 49.2 48.8 40.1 41.1 33.6 50.3 50.7 39.4 49.9 48.1 36.0 37.5 46.1 49.6 Bottom quarter S.E Mean index (0.30) (0.52) (0.38) (0.27) (0.34) (0.45) (0.36) (0.47) (0.42) (0.72) (0.33) (0.26) (0.49) (0.38) (0.31) (0.36) (0.22) (0.68) (0.38) (0.36) (0.39) (0.34) (0.54) (0.40) (0.58) (0.38) (0.43) (0.75) (0.37) (0.15) (0.08) (0.64) (0.45) (0.61) (0.52) (0.75) (0.40) (0.38) (0.53) (0.43) (0.60) (0.48) (0.39) 31.6 27.3 29.0 31.7 32.3 29.4 28.7 27.6 29.5 26.9 30.2 31.5 28.5 26.9 33.4 28.9 27.3 22.2 30.9 30.1 35.0 26.9 26.4 29.3 26.2 30.4 29.4 23.7 32.6 28.1 28.2 21.7 25.9 16.0 29.1 30.8 25.8 30.8 28.3 22.1 18.0 25.2 28.5 Second quarter S.E Third quarter S.E Mean index S.E Mean index S.E 48.0 40.9 44.5 47.7 45.7 44.2 43.4 42.3 42.6 38.8 42.3 48.0 42.2 40.3 43.9 43.5 42.1 28.9 45.4 46.2 49.0 39.5 33.9 41.4 35.5 44.1 43.1 33.6 49.9 42.5 42.3 32.4 34.9 24.1 44.2 47.4 34.4 40.9 41.2 26.7 29.2 37.8 43.0 (0.07) (0.11) (0.13) (0.08) (0.12) (0.11) (0.16) (0.15) (0.14) (0.13) (0.08) (0.13) (0.11) (0.11) (0.04) (0.09) (0.13) (0.04) (0.15) (0.12) (0.12) (0.11) (0.08) (0.09) (0.14) (0.14) (0.14) (0.15) (0.15) (0.07) (0.08) (0.09) (0.07) (0.15) (0.16) (0.52) (0.12) (0.10) (0.12) (0.13) (0.18) (0.15) (0.14) 58.3 51.4 56.4 58.1 52.5 53.2 56.4 53.6 53.7 51.8 51.6 61.7 52.7 50.6 50.6 49.4 52.8 42.1 56.9 56.8 60.6 49.1 46.6 53.1 49.3 56.1 53.5 47.2 61.4 54.1 53.2 44.4 45.1 34.6 54.8 55.0 41.7 54.2 51.4 35.6 39.6 50.8 55.5 (0.11) (0.12) (0.13) (0.09) (0.05) (0.07) (0.14) (0.05) (0.06) (0.07) (0.11) (0.19) (0.08) (0.05) (0.08) (0.06) (0.06) (0.28) (0.20) (0.17) (0.16) (0.10) (0.19) (0.10) (0.11) (0.17) (0.08) (0.10) (0.12) (0.08) (0.09) (0.17) (0.13) (0.33) (0.14) (0.09) (0.25) (0.21) (0.11) (0.13) (0.19) (0.12) (0.11) 72.5 68.7 72.4 72.9 69.7 70.3 72.4 71.2 71.5 70.3 70.2 73.7 70.0 69.5 72.0 63.5 70.5 67.3 71.8 72.7 73.9 64.4 65.5 71.5 66.1 71.9 71.1 61.8 74.3 71.9 71.2 62.1 58.7 59.9 73.0 70.0 55.9 73.6 71.4 59.6 63.1 70.8 71.6 (0.14) (0.28) (0.16) (0.15) (0.23) (0.29) (0.18) (0.26) (0.25) (0.39) (0.20) (0.25) (0.29) (0.38) (0.25) (0.43) (0.24) (0.25) (0.25) (0.26) (0.21) (0.34) (0.53) (0.21) (0.39) (0.21) (0.27) (0.77) (0.21) (0.11) (0.13) (0.60) (0.37) (0.42) (0.30) (0.67) (0.52) (0.20) (0.38) (0.41) (0.44) (0.36) (0.19) (0.14) (0.19) (0.13) (0.11) (0.18) (0.19) (0.12) (0.20) (0.17) (0.13) (0.18) (0.20) (0.17) (0.16) (0.17) (0.20) (0.15) (0.12) (0.26) (0.19) (0.20) (0.21) (0.14) (0.17) (0.13) (0.18) (0.14) (0.29) (0.21) (0.07) (0.04) (0.31) (0.14) (0.00) (0.23) (0.63) (0.32) (0.16) (0.20) (0.14) (0.17) (0.16) (0.14) Change in the problem-solving score per 16.4 units of the inter- Explained variance national socio-economic in student performance index of occupational status (r-squared x 100) OECD countries Performance on the problem-solving scale, by national quarters of the international socio-economic index of occupational status Bottom quarter Second quarter Third quarter Top quarter Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Partner countries OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 Top quarter Mean index Annex B Table 5.2 International socio-economic index of occupational status (HISEI) and performance on the problem-solving scale, by national quarters of the index Results based on students’ self-reports Mean score S.E Mean score S.E Mean score S.E Mean score S.E Effect S.E % S.E 498 468 481 505 490 488 521 476 473 411 457 489 466 433 520 526 450 352 482 492 449 455 433 450 449 477 480 384 442 455 465 333 526 336 459 479 526 450 390 404 316 374 473 (2.5) (4.0) (3.7) (2.4) (4.1) (3.4) (2.7) (4.1) (4.6) (4.4) (4.1) (3.0) (3.5) (4.7) (5.1) (4.2) (2.9) (5.3) (4.5) (3.8) (3.8) (4.0) (5.7) (4.6) (3.8) (3.5) (3.6) (5.3) (3.6) (1.5) (4.0) (5.4) (5.6) (3.4) (4.4) (9.1) (4.9) (5.7) (3.9) (3.2) (3.4) (5.3) (3.0) 526 493 524 528 504 507 542 511 514 441 482 500 495 461 551 555 484 371 518 525 483 475 454 478 471 500 520 395 473 484 492 367 546 355 472 527 527 468 411 409 327 404 504 (2.6) (3.5) (3.3) (2.2) (3.6) (3.3) (2.6) (4.4) (3.4) (4.4) (3.6) (3.0) (3.1) (4.2) (5.0) (3.5) (3.2) (4.5) (4.1) (3.4) (3.4) (3.3) (4.6) (3.4) (3.0) (2.8) (3.6) (6.1) (3.8) (1.5) (3.6) (5.6) (4.6) (3.6) (5.1) (10.1) (5.8) (5.3) (3.8) (3.4) (2.8) (5.0) (3.3) 543 522 550 539 530 527 555 534 536 454 522 508 509 485 556 555 511 392 540 543 502 494 486 516 495 517 532 405 492 502 512 386 559 362 492 538 535 485 432 431 356 418 520 (2.3) (2.8) (2.8) (2.0) (3.3) (3.7) (2.8) (3.1) (4.1) (4.9) (3.4) (2.5) (3.1) (3.6) (4.7) (3.5) (2.6) (4.7) (3.7) (3.5) (4.2) (3.5) (3.9) (3.6) (3.6) (3.2) (4.7) (6.6) (3.8) (1.6) (3.6) (4.9) (3.5) (4.5) (4.6) (10.2) (6.5) (5.1) (3.8) (3.5) (3.9) (4.7) (3.8) 566 549 580 565 566 552 577 567 567 498 558 529 536 507 580 573 539 433 564 573 529 528 515 541 521 545 561 461 526 533 542 422 573 399 511 582 545 514 456 464 388 474 553 (2.8) (4.2) (3.1) (2.4) (4.3) (3.2) (3.1) (3.8) (3.7) (5.0) (4.3) (3.3) (3.4) (4.3) (6.0) (5.4) (3.2) (5.7) (3.8) (3.6) (3.9) (3.5) (4.2) (3.7) (3.7) (3.5) (3.4) (11.2) (3.6) (1.3) (4.1) (7.7) (4.9) (4.8) (5.5) (9.0) (6.7) (5.0) (4.6) (4.7) (4.8) (4.7) (3.6) 26.6 30.2 36.0 23.5 33.3 25.2 19.6 32.8 34.8 30.1 41.4 13.7 27.2 28.3 21.5 22.3 32.4 28.6 31.9 30.2 30.9 33.5 34.7 34.2 27.9 26.7 28.4 34.1 29.9 33.5 32.9 36.6 24.4 22.2 19.6 38.6 12.1 23.5 24.7 25.9 26.9 34.9 29.4 (1.3) (1.8) (1.7) (1.2) (2.0) (1.7) (1.4) (2.1) (1.8) (2.2) (2.3) (1.5) (1.9) (2.1) (3.2) (2.9) (1.6) (2.3) (2.1) (1.8) (1.8) (1.9) (2.1) (1.9) (1.6) (1.6) (1.6) (5.1) (1.5) (0.8) (0.4) (3.4) (2.6) (2.0) (2.1) (5.1) (3.8) (2.1) (2.0) (2.3) (2.0) (2.2) (1.5) 8.3 11.0 13.9 6.8 10.8 7.6 6.1 13.6 14.3 10.0 17.1 2.8 11.3 8.1 3.6 4.5 12.9 11.1 12.4 10.1 8.8 11.6 13.7 14.1 8.9 9.2 9.1 11.2 9.6 13.3 11.2 13.2 4.4 10.2 4.7 14.6 1.2 6.0 8.6 9.0 13.4 11.7 10.5 (0.7) (1.2) (1.2) (0.7) (1.2) (1.0) (0.8) (1.5) (1.2) (1.5) (1.6) (0.6) (1.5) (1.0) (0.9) (1.1) (1.2) (1.7) (1.4) (1.2) (0.9) (1.2) (1.5) (1.2) (1.0) (1.0) (0.9) (2.9) (0.9) (0.7) (0.2) (2.2) (0.9) (1.6) (1.1) (3.7) (0.8) (1.0) (1.3) (1.5) (2.0) (1.3) (1.1) Note: Values that are statistically significant are indicated in bold (see Annex A4) Response rate too low to ensure comparability (see Annex A3) © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 149 Annex B Table 5.3 Index of highest educational level of parents (HISCED)1 and performance on the problem-solving scale, by national quarters of the index Results based on students’ self-reports Index of highest educational level of parents (HISCED) All students OECD countries Mean index Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Partner countries OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 4.62 4.07 4.64 4.93 4.23 4.47 4.78 3.98 4.02 4.16 4.24 4.29 4.22 3.86 4.78 4.07 4.09 2.91 4.55 4.24 4.75 4.10 2.74 4.26 3.66 4.66 3.88 2.81 4.69 4.18 4.16 3.66 2.59 2.83 4.83 3.92 2.58 4.83 4.19 2.39 2.46 3.88 4.20 Bottom quarter S.E Second quarter Third quarter S.E Mean index S.E Mean index S.E Mean index 2.71 2.68 2.88 3.62 2.91 2.69 3.06 2.00 1.90 1.84 2.70 2.49 2.48 1.83 3.15 1.82 1.42 0.50 2.38 2.26 3.48 2.90 0.27 3.05 0.87 2.69 1.87 0.83 3.41 1.88 1.92 0.87 0.94 0.72 3.73 2.02 0.65 3.90 2.68 0.93 0.54 1.31 2.53 (0.03) (0.03) (0.04) (0.02) (0.01) (0.05) (0.03) (0.03) (0.04) (0.03) (0.02) (0.03) (0.06) (0.01) (0.06) (0.03) (0.04) (0.02) (0.06) (0.04) (0.03) (0.02) (0.01) (0.04) (0.01) (0.04) (0.02) (0.02) (0.06) (0.01) (0.01) (0.03) (0.01) (0.02) (0.02) (0.08) (0.04) (0.01) (0.02) (0.02) (0.02) (0.02) (0.03) 4.17 3.33 4.29 4.54 3.93 4.34 4.67 3.36 3.53 4.00 3.94 4.00 4.00 3.43 4.32 3.74 4.01 1.71 4.00 4.00 4.51 3.99 1.46 4.00 3.21 4.43 3.14 1.59 4.01 3.99 3.92 2.62 2.00 1.58 4.39 3.12 1.93 4.00 3.27 1.01 1.43 3.12 3.71 (0.01) (0.02) (0.01) (0.01) (0.01) (0.02) (0.01) (0.02) (0.02) (0.00) (0.01) (0.00) (0.00) (0.02) (0.01) (0.01) (0.01) (0.02) (0.00) (0.00) (0.02) (0.00) (0.02) (0.00) (0.03) (0.01) (0.01) (0.02) (0.00) (0.00) (0.00) (0.04) (0.00) (0.02) (0.02) (0.03) (0.02) (0.00) (0.02) (0.00) (0.02) (0.03) (0.01) 5.59 4.73 5.41 5.56 4.07 5.00 5.40 4.54 4.65 4.80 4.33 4.67 4.60 4.38 5.65 4.71 5.00 3.71 5.80 4.79 5.01 4.01 3.48 4.06 4.57 5.52 4.66 3.49 5.36 4.86 4.78 5.16 2.79 3.68 5.18 4.75 3.27 5.42 5.00 2.58 2.97 5.08 4.68 (0.01) (0.01) (0.01) (0.01) (0.01) (0.00) (0.02) (0.02) (0.02) (0.02) (0.01) (0.02) (0.02) (0.01) (0.02) (0.03) (0.00) (0.05) (0.02) (0.01) (0.00) (0.00) (0.03) (0.01) (0.02) (0.02) (0.01) (0.03) (0.02) (0.01) (0.02) (0.03) (0.01) (0.02) (0.01) (0.04) (0.07) (0.03) (0.00) (0.03) (0.03) (0.01) (0.01) max 5.55 max max max 5.85 max max 5.99 max max max 5.79 5.81 max max 5.94 5.70 max 5.91 max 5.50 5.74 5.93 max max 5.86 5.35 max max max max 4.62 5.33 max 5.83 4.46 max 5.83 5.06 4.90 max 5.89 (0.02) (0.03) (0.03) (0.02) (0.03) (0.04) (0.02) (0.04) (0.04) (0.06) (0.03) (0.02) (0.04) (0.03) (0.03) (0.04) (0.03) (0.07) (0.04) (0.03) (0.02) (0.02) (0.06) (0.03) (0.07) (0.03) (0.04) (0.09) (0.03) (0.01) (0.01) (0.07) (0.04) (0.06) (0.03) (0.07) (0.05) (0.03) (0.04) (0.05) (0.06) (0.05) (0.03) Change in the problemsolving score per unit of the index of highest educational level of parents OECD countries Performance on the problem-solving scale, by national quarters of the index of highest educational level of parents (HISCED) Bottom quarter Second quarter Third quarter Top quarter Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Partner countries OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 Mean score S.E Mean score S.E Mean score S.E Mean score S.E 508 485 495 512 481 487 530 487 475 416 449 487 473 431 519 523 466 337 502 513 469 453 440 453 449 490 480 373 453 443 459 339 528 346 469 501 525 460 404 410 323 370 489 (3.1) (4.5) (4.3) (2.0) (3.6) (3.6) (2.7) (4.8) (4.7) (4.5) (4.3) (3.5) (3.3) (4.9) (6.1) (4.5) (3.1) (4.6) (5.5) (3.7) (4.3) (4.2) (4.7) (6.0) (4.6) (3.4) (3.3) (4.6) (4.2) (1.8) (1.2) (5.0) (4.8) (3.1) (5.5) (9.3) (6.5) (5.7) (4.3) (3.3) (3.0) (5.7) (4.0) 516 507 528 526 522 515 544 519 523 448 495 502 494 474 529 544 494 372 516 537 490 485 463 489 481 516 537 388 466 489 500 375 542 352 474 520 526 460 421 409 335 400 502 (3.4) (4.4) (3.2) (2.2) (3.8) (3.6) (2.9) (3.8) (4.4) (4.2) (3.5) (3.0) (3.3) (3.8) (5.9) (3.4) (3.4) (4.4) (3.8) (3.1) (4.5) (4.1) (4.5) (3.9) (3.0) (3.3) (3.1) (5.5) (4.6) (1.4) (0.7) (5.0) (6.1) (3.8) (5.8) (12.8) (7.2) (5.2) (4.6) (3.7) (2.9) (5.2) (3.3) 543 510 552 545 525 534 554 536 528 448 501 505 503 478 566 555 500 418 543 538 506 485 486 488 484 520 526 415 486 501 510 391 562 374 481 541 538 491 409 424 350 417 510 (2.8) (4.2) (3.1) (3.0) (3.6) (3.7) (2.6) (4.2) (5.0) (5.5) (3.5) (3.7) (3.3) (4.0) (5.5) (3.6) (3.2) (5.5) (4.5) (3.3) (4.3) (3.8) (3.9) (3.7) (3.9) (4.4) (3.4) (6.3) (4.3) (2.0) (0.9) (6.4) (4.9) (3.8) (5.7) (9.0) (6.7) (6.7) (4.4) (3.8) (3.0) (5.1) (4.1) 559 531 561 554 565 542 566 551 565 482 560 529 526 497 576 582 533 413 545 568 506 524 496 540 522 520 550 456 511 531 540 385 564 375 510 563 540 505 448 458 372 458 549 (3.0) (4.7) (3.2) (2.8) (4.5) (3.8) (2.8) (4.0) (4.2) (6.3) (4.4) (2.9) (3.5) (3.9) (5.2) (5.4) (3.1) (5.7) (6.0) (4.0) (4.3) (3.5) (6.0) (3.4) (3.8) (3.6) (5.1) (11.7) (4.5) (1.5) (0.8) (7.1) (5.2) (6.0) (5.7) (11.9) (4.8) (5.8) (5.1) (5.1) (5.2) (5.6) (3.6) Note: Values that are statistically significant are indicated in bold (see Annex A4) Highest Educational level of Parents (HISCED) corresponds to the higher level of education (ISCED) of either parent Response rate too low to ensure comparability (see Annex A3) 150 Top quarter Mean index © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Effect 14.3 15.2 21.5 17.1 26.1 18.6 10.6 16.0 20.6 14.5 33.1 11.5 15.8 15.7 19.4 13.5 13.9 15.0 12.6 15.4 15.7 26.3 10.5 29.9 12.4 11.1 15.3 19.6 21.3 22.7 20.2 9.5 9.4 6.5 15.9 16.2 3.6 21.6 11.1 12.1 10.7 16.9 17.8 S.E (0.02) (0.01) (0.00) (0.01) (0.01) (0.01) (0.02) (0.01) (0.03) (0.02) (0.01) (0.01) (0.04) (0.04) (0.03) (0.04) (0.05) (0.02) (0.03) (0.03) (0.01) Explained variance in student performance (r-squared x 100) S.E % S.E (1.1) (1.6) (1.2) (0.9) (1.7) (1.5) (1.0) (1.3) (1.2) (1.6) (1.7) (1.0) (1.1) (1.4) (2.7) (1.4) (0.8) (1.3) (1.4) (1.2) (1.5) (1.9) (1.0) (2.3) (1.0) (1.1) (1.1) (2.5) (1.2) (0.5) (0.3) (1.3) (1.5) (1.2) (2.6) (3.5) (2.0) (2.0) (1.5) (1.3) (1.3) (1.4) (1.4) 5.2 4.5 9.1 4.3 11.3 7.9 2.8 7.8 13.6 5.5 18.9 3.5 7.6 5.7 5.7 6.6 7.3 10.6 5.5 5.7 2.8 9.1 6.2 13.3 7.1 3.3 6.8 14.5 7.1 12.0 10.4 4.1 2.0 2.8 2.7 7.3 0.5 5.1 3.0 7.2 5.8 8.4 6.7 (0.7) (1.0) (1.0) (0.4) (1.3) (1.2) (0.5) (1.2) (1.5) (1.1) (1.6) (0.6) (1.1) (0.9) (1.3) (1.2) (0.8) (1.6) (1.1) (0.9) (0.5) (1.2) (1.1) (1.8) (1.1) (0.6) (0.9) (3.0) (0.9) (0.5) (0.3) (1.0) (0.6) (1.0) (0.9) (3.4) (0.6) (0.9) (0.8) (1.4) (1.4) (1.3) (1.0) Index of possessions related to “classical” culture in the family home OECD countries All students Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Partner countries OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 Bottom quarter Mean index S.E Mean index –0.12 –0.05 –0.30 0.00 0.26 –0.01 0.11 –0.05 0.00 0.23 0.31 0.79 –0.26 0.19 –0.43 0.16 –0.03 –0.68 –0.31 –0.18 0.15 0.25 –0.08 0.35 0.15 0.10 –0.37 –0.11 –0.04 –0.10 0.00 –0.33 –0.44 –0.65 0.40 –0.27 –0.50 0.48 0.14 –0.21 –0.63 0.07 –0.03 (0.01) (0.03) (0.02) (0.01) (0.02) (0.03) (0.02) (0.02) (0.02) (0.03) (0.02) (0.01) (0.02) (0.02) (0.02) (0.02) (0.01) (0.03) (0.02) (0.02) (0.02) (0.02) (0.03) (0.02) (0.02) (0.02) (0.03) (0.03) (0.02) (0.01) (0.00) (0.02) (0.03) (0.02) (0.02) (0.05) (0.02) (0.02) (0.03) (0.02) (0.02) (0.02) (0.02) min min –1.00 min min –0.94 –0.97 –0.42 –1.18 –1.14 min min –0.84 –0.93 –1.17 –1.26 min –1.28 –1.28 –1.28 min –0.91 min –0.67 min –1.21 Second quarter S.E Third quarter S.E Mean index S.E Mean index S.E –0.64 –0.48 –0.94 –0.40 –0.02 –0.45 –0.28 –0.44 –0.44 –0.07 0.16 0.90 –0.85 –0.08 –1.12 –0.11 –0.51 –1.28 –0.78 –0.62 –0.30 –0.04 –0.55 0.10 –0.11 –0.28 –1.02 –0.51 –0.57 –0.62 –0.45 –0.83 –1.04 0.25 –0.85 –1.16 0.38 –0.22 –0.62 –0.22 –0.61 (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.02) (0.02) (0.01) (0.01) (0.01) (0.01) (0.00) (0.02) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.00) (0.01) (0.02) (0.01) (0.01) (0.00) (0.02) (0.01) 0.13 0.28 –0.05 0.32 0.71 0.35 0.65 0.30 0.37 0.59 0.69 1.35 0.01 0.67 –0.18 0.55 0.31 –0.65 –0.16 0.06 0.84 0.53 0.27 0.88 0.54 0.59 –0.13 0.22 0.34 0.21 0.38 –0.06 –0.22 –0.51 0.92 –0.04 –0.24 0.85 0.73 0.05 –0.47 0.38 0.40 (0.01) (0.01) (0.01) (0.01) (0.01) (0.00) (0.01) (0.01) (0.01) (0.01) (0.01) (0.00) (0.01) (0.01) (0.01) (0.01) (0.01) (0.02) (0.01) (0.01) (0.02) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.01) (0.02) (0.01) (0.01) (0.01) (0.02) (0.03) (0.01) (0.01) (0.01) (0.01) (0.01) (0.00) (0.01) 1.31 1.29 1.08 max max max max 1.22 max max max max 1.07 max 0.85 max max 0.49 0.96 1.11 max max 1.24 max max max 0.95 1.12 max 1.29 1.35 0.86 0.78 0.46 max 1.09 0.69 max max 1.02 0.50 1.32 max (0.00) (0.01) (0.01) (0.02) (0.01) (0.02) (0.02) (0.01) (0.01) (0.02) (0.02) (0.01) (0.00) (0.00) (0.00) (0.00) (0.02) (0.02) (0.01) (0.01) (0.05) (0.02) (0.00) (0.01) (0.01) (0.01) (0.01) OECD countries Performance on the problem-solving scale, by national quarters of the index of possessions related to “classical” culture in the family home Bottom quarter Second quarter Third quarter Top quarter Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Partner countries OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 Top quarter Mean index Change in the problem-solving score per unit of this index Annex B Table 5.4 Index of possessions related to “classical” culture in the family home and performance on the problem-solving scale, by national quarters of the index Results based on students’ self-reports (0.01) (0.01) (0.01) (0.02) (0.02) (0.01) (0.01) (0.02) (0.01) (0.01) (0.00) (0.02) (0.02) (0.02) (0.05) (0.03) (0.01) (0.02) (0.00) Explained variance in student performance (r-squared x 100) Mean score S.E Mean score S.E Mean score S.E Mean score S.E Effect S.E % S.E 508 477 494 510 491 474 521 477 495 412 441 482 481 434 521 521 474 364 498 507 446 455 432 459 447 470 511 377 441 453 468 358 526 358 443 515 522 442 387 413 334 372 482 (3.0) (4.0) (3.9) (2.4) (4.4) (3.2) (3.1) (4.8) (4.1) (4.7) (3.9) (3.7) (4.2) (4.4) (5.3) (3.9) (3.0) (4.7) (5.4) (3.8) (3.8) (4.1) (5.6) (5.7) (4.0) (3.5) (4.0) (5.2) (3.9) (2.0) (1.1) (5.3) (5.8) (4.4) (5.7) (10.6) (5.6) (4.8) (4.0) (3.7) (3.7) (5.5) (3.4) 519 491 506 529 518 502 542 512 505 445 501 507 488 466 527 544 481 365 513 521 480 483 454 485 476 497 511 390 458 475 489 361 534 357 486 510 522 489 414 418 331 399 492 (3.0) (3.9) (3.6) (2.5) (4.3) (3.2) (2.8) (4.3) (4.5) (4.5) (4.1) (3.6) (4.6) (4.5) (5.5) (4.3) (3.3) (4.5) (4.8) (4.1) (3.6) (3.5) (4.4) (4.3) (3.7) (3.1) (4.4) (5.7) (4.2) (1.6) (0.8) (6.0) (5.9) (4.4) (4.8) (12.8) (6.7) (5.3) (4.1) (3.7) (3.2) (4.7) (3.7) 532 513 541 538 534 535 561 539 514 462 525 516 500 482 566 560 492 381 524 534 511 501 485 506 496 528 516 414 489 502 511 372 556 364 499 521 540 491 437 428 341 423 519 (2.4) (4.2) (3.2) (2.5) (3.6) (3.2) (3.5) (3.8) (4.4) (5.1) (4.5) (3.0) (3.6) (4.2) (5.1) (3.6) (2.8) (4.9) (4.0) (5.0) (3.9) (4.5) (4.4) (3.5) (3.6) (3.7) (4.0) (6.7) (3.8) (1.2) (0.8) (6.7) (5.5) (4.6) (5.1) (11.7) (6.2) (5.0) (4.3) (3.7) (3.8) (5.9) (4.2) 561 551 571 557 547 557 568 553 561 476 537 515 524 498 576 578 528 427 557 572 525 508 509 516 511 541 547 449 523 530 534 397 575 367 503 571 546 492 444 441 374 449 545 (2.6) (4.1) (2.8) (2.6) (4.0) (3.2) (3.3) (3.6) (3.9) (5.8) (4.0) (3.2) (3.4) (3.9) (5.4) (4.8) (2.8) (6.3) (4.0) (3.6) (4.6) (4.2) (4.8) (3.4) (3.9) (3.6) (4.9) (9.5) (4.2) (1.5) (0.8) (6.7) (5.2) (4.3) (4.9) (10.8) (5.0) (6.0) (4.0) (4.3) (4.1) (4.9) (4.3) 20.1 29.1 32.6 17.5 23.8 32.6 18.2 30.8 23.9 27.8 42.0 20.1 18.6 25.2 25.9 23.1 20.5 35.7 25.7 26.8 29.5 25.5 30.7 25.0 25.7 27.9 17.2 30.7 31.4 30.3 25.3 19.1 22.9 5.9 24.8 24.1 12.7 24.3 22.4 12.4 22.5 31.2 24.7 (1.2) (1.7) (1.7) (1.3) (1.6) (1.5) (1.4) (2.2) (1.5) (2.6) (2.1) (2.0) (1.5) (1.9) (2.8) (2.4) (1.4) (3.1) (2.4) (1.7) (1.5) (2.2) (2.1) (2.4) (1.8) (1.9) (1.7) (3.9) (1.6) (0.7) (0.4) (2.3) (2.6) (1.7) (2.2) (4.9) (3.2) (2.0) (1.6) (1.9) (2.4) (2.4) (1.7) 4.9 10.3 9.7 3.9 6.4 13.7 5.1 10.3 6.6 6.6 16.8 3.8 5.1 6.0 5.1 6.6 5.1 9.1 7.0 7.1 10.0 6.0 10.6 6.4 7.0 10.0 3.0 9.0 10.9 8.1 6.4 2.9 4.2 0.4 6.2 6.4 1.7 4.3 7.4 2.0 5.1 6.9 7.8 (0.5) (1.1) (0.8) (0.5) (0.8) (1.1) (0.7) (1.2) (0.8) (1.2) (1.4) (0.7) (0.8) (0.8) (1.0) (1.2) (0.6) (1.6) (1.2) (0.9) (1.0) (0.9) (1.3) (1.1) (0.9) (1.2) (0.6) (1.7) (1.0) (0.3) (0.2) (0.7) (0.9) (0.2) (1.1) (2.6) (0.9) (0.7) (1.0) (0.6) (1.0) (1.0) (1.0) Note: Values that are statistically significant are indicated in bold (see Annex A4) “Min” is used for countries with more than 25 per cent of students at the lowest value on this index, which is –1.28 “Max” is used for countries with more than 25 per cent of students at the highest value of this index, which is 1.35 Response rate too low to ensure comparability (see Annex A3) © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 151 Annex B Table 5.5 Percentage of students and performance on the problem-solving scale, by type of family structure Results based on students’ self-reports Partner countries OECD countries Students from single-parent families Students from other types of families % of students S.E Mean score S.E % of students S.E S.E Difference S.E Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States 20.0 15.9 17.0 18.6 12.8 24.3 20.0 20.3 16.7 23.4 19.0 13.3 15.4 15.5 m 20.3 16.3 33.1 13.7 18.9 27.1 11.4 16.5 11.5 14.0 24.0 20.8 32.7 29.4 (0.5) (0.6) (0.5) (0.4) (0.5) (1.1) (0.7) (0.7) (0.6) (1.0) (0.7) (0.6) (0.7) (0.6) m (0.6) (0.5) (0.8) (0.9) (0.7) (0.7) (0.5) (0.6) (0.5) (0.5) (0.7) (0.7) (1.3) (0.9) 511 509 496 514 518 499 543 510 514 430 494 496 474 463 m 548 476 378 500 515 473 479 470 487 475 495 510 402 448 (2.7) (4.6) (4.3) (3.0) (3.8) (3.9) (3.3) (3.9) (5.7) (5.3) (4.3) (4.3) (4.1) (4.6) m (4.6) (3.9) (6.1) (5.2) (4.1) (4.0) (5.7) (5.9) (5.2) (4.8) (3.6) (3.9) (6.5) (4.2) 80.0 84.1 83.0 81.4 87.2 75.7 80.1 79.8 83.3 76.6 81.0 86.7 84.6 84.5 m 79.7 83.7 66.9 86.3 81.1 72.9 88.7 83.5 88.5 86.0 76.0 79.2 67.3 70.6 (0.5) (0.6) (0.5) (0.4) (0.5) (1.1) (0.7) (0.7) (0.6) (1.0) (0.7) (0.6) (0.7) (0.6) m (0.6) (0.5) (0.8) (0.9) (0.7) (0.7) (0.5) (0.6) (0.5) (0.5) (0.7) (0.7) (1.3) (0.9) 535 508 535 538 523 523 549 523 521 455 504 507 503 471 m 551 497 389 527 538 498 488 470 494 484 514 525 412 491 (2.1) (3.1) (2.3) (1.6) (3.2) (2.8) (1.8) (2.8) (3.4) (4.3) (2.9) (1.5) (2.4) (3.1) m (3.0) (1.5) (4.1) (3.0) (2.3) (2.9) (2.7) (3.9) (3.4) (2.9) (2.5) (3.3) (6.3) (3.0) –25 –39 –24 –5 –25 –6 –14 –7 –26 –10 –11 –29 –8 m –3 –21 –10 –28 –22 –25 –9 –7 –9 –19 –15 –10 –44 (2.5) (4.1) (3.9) (3.0) (3.6) (3.8) (3.0) (3.9) (5.1) (5.0) (4.0) (4.6) (3.9) (4.1) m (3.4) (4.4) (4.3) (5.1) (4.4) (4.2) (5.2) (4.7) (4.6) (4.8) (3.4) (3.6) (4.4) (3.5) OECD total OECD average 23.4 19.4 (0.3) (0.1) 458 481 (2.0) (1.1) 76.6 80.6 (0.3) (0.1) 492 504 (1.2) (0.6) –34 –23 (1.6) (0.9) 26.2 19.7 9.9 25.4 17.8 21.1 20.7 14.9 21.7 7.3 23.1 22.2 (0.9) (0.7) (0.5) (0.9) (2.1) (1.3) (0.6) (0.7) (0.8) (0.4) (0.6) (0.6) 368 534 347 474 514 532 479 416 420 332 409 494 (6.8) (5.6) (5.0) (5.1) (12.1) (6.7) (4.8) (4.6) (4.1) (5.1) (4.5) (3.5) 73.8 80.3 90.2 74.6 82.2 78.9 79.3 85.1 78.3 92.7 76.9 77.8 (0.9) (0.7) (0.5) (0.9) (2.1) (1.3) (0.6) (0.7) (0.8) (0.4) (0.6) (0.6) 374 552 363 486 533 533 481 421 427 348 412 515 (4.7) (4.1) (3.3) (4.0) (4.8) (2.9) (4.2) (3.4) (2.8) (2.3) (4.0) (2.6) –6 –18 –16 –12 –18 –1 –2 –6 –7 –16 –3 –21 (5.6) (4.0) (4.5) (4.1) (14.0) (7.7) (3.4) (4.4) (3.6) (5.2) (4.3) (3.3) Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 Note: Values that are statistically significant are indicated in bold (see Annex A4) Response rate too low to ensure comparability (see Annex A3) 152 Difference in problem-solving performance (single-parent families – other types of families) © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Mean score OECD countries Native students (born in the country of assessment with at least one of their parents born in the same country) Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Partner countries OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 % of students S.E Mean score S.E 77.3 86.7 88.2 79.9 98.7 93.5 98.1 85.7 84.6 92.6 97.7 99.0 96.5 97.9 99.9 100.0 66.7 97.7 89.0 80.2 94.4 100.0 95.0 99.1 96.6 88.5 80.0 99.0 85.6 91.5 91.4 99.2 56.7 99.7 90.6 82.9 23.9 86.5 91.1 99.9 99.7 99.2 92.0 (1.1) (1.0) (0.9) (1.1) (0.2) (0.8) (0.2) (1.3) (1.1) (0.6) (0.2) (0.2) (0.3) (0.3) (0.0) (0.0) (0.6) (0.3) (1.4) (1.1) (0.7) (0.0) (1.4) (0.2) (0.4) (0.9) (0.9) (0.2) (1.0) (0.3) (0.2) (0.2) (1.4) (0.1) (0.9) (2.0) (1.4) (0.7) (0.6) (0.1) (0.1) (0.2) (0.8) 535 515 540 535 523 522 549 529 534 452 502 507 499 472 548 551 507 392 532 537 495 488 475 493 484 516 538 409 483 495 505 374 556 364 483 537 536 482 423 426 346 411 511 (2.1) (3.2) (2.4) (1.6) (3.0) (2.4) (1.8) (2.5) (3.4) (4.0) (3.1) (1.4) (2.3) (3.0) (4.1) (3.1) (1.8) (4.3) (3.1) (2.5) (2.6) (2.7) (3.3) (3.2) (2.7) (2.2) (3.0) (5.9) (2.9) (1.1) (0.6) (4.7) (4.1) (3.3) (4.0) (4.5) (5.1) (4.7) (3.4) (2.7) (2.1) (3.6) (2.4) First-generation students (born in the country of assessment but whose parents were foreign-born) % of students 11.7 4.1 6.3 9.2 0.5 3.5 0.0 10.8 6.9 0.5 0.1 0.2 1.0 0.4 0.0 0.0 15.8 0.5 7.1 6.6 2.3 0.0 2.3 0.6 0.6 5.7 8.9 0.5 8.3 4.6 4.0 0.6 22.9 0.2 8.3 7.6 57.9 6.4 3.2 0.1 0.2 0.4 5.3 S.E Mean score S.E (0.6) (0.5) (0.6) (0.5) (0.1) (0.6) (0.0) (1.1) (0.8) (0.1) (0.0) (0.1) (0.2) (0.1) (0.0) (0.0) (0.6) (0.1) (1.1) (0.7) (0.4) (0.0) (0.4) (0.2) (0.1) (0.5) (0.5) (0.2) (0.7) (0.2) (0.1) (0.2) (0.9) (0.1) (0.8) (1.3) (1.5) (0.5) (0.3) (0.1) (0.1) (0.1) (0.6) 521 465 445 532 c 443 c 482 444 c c c c c c c 475 c 463 500 452 c c c c 483 480 c 465 473 479 c 572 c 487 512 533 473 417 c c c 512 (4.0) (9.9) (7.7) (4.0) c (10.5) c (6.2) (9.2) c c c c c c c (3.7) c (9.7) (7.5) (11.7) c c c c (8.9) (4.8) c (8.5) (4.5) (2.0) c (4.0) c (7.9) (17.9) (3.3) (6.7) (8.3) c c c (7.3) OECD countries Difference in problem-solving performance between native Non-native students (foreign-born and whose parents were also foreign-born) and first-generation students Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States Partner countries OECD total OECD average Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 % of students S.E Mean score S.E 11.0 9.2 5.5 10.9 0.8 3.0 1.8 3.5 8.5 6.9 2.2 0.8 2.5 1.7 0.1 a 17.4 1.8 3.9 13.3 3.4 0.0 2.7 0.3 2.8 5.9 11.1 0.5 6.1 3.9 4.6 0.2 20.4 0.1 1.1 9.4 18.2 7.0 5.6 0.0 0.1 0.4 2.7 (0.7) (0.7) (0.6) (0.8) (0.1) (0.4) (0.2) (0.5) (0.7) (0.7) (0.2) (0.2) (0.3) (0.2) (0.0) a (0.5) (0.2) (0.4) (0.7) (0.4) (0.0) (1.1) (0.1) (0.4) (0.7) (0.6) (0.1) (0.4) (0.1) (0.1) (0.1) (1.3) (0.0) (0.2) (1.6) (1.4) (0.5) (0.5) (0.0) (0.0) (0.1) (0.4) 523 453 446 533 c 464 c 445 461 412 c c c c c a 463 c 462 534 417 c c c c 434 447 c 446 454 468 c 505 c c 480 531 451 423 c c c c (4.8) (5.9) (8.6) (4.7) c (8.8) c (14.8) (7.4) (7.0) c c c c c a (3.9) c (8.8) (4.6) (10.3) c c c c (10.1) (5.8) c (8.3) (3.9) (1.9) c (5.0) c c (19.6) (8.8) (7.4) (5.8) c c c c Difference 14 50 95 c 79 c 47 90 –13 c c c c c c 33 c 69 38 43 c c c c 33 58 c 19 22 26 c –17 c –5 26 c c c S.E (4.3) (10.2) (7.5) (4.2) c (10.5) c (6.5) (9.6) (24.2) c c c c c c (4.2) c (10.4) (8.1) (11.5) c c c c (8.3) (4.7) c (8.1) (4.4) (2.0) c (3.8) c (7.7) (18.6) (6.5) (6.9) (8.0) c c c (6.8) Annex B Table 5.6 Percentage of students and performance on the problem-solving scale, by students’ nationality and the nationality of their parents Results based on students’ self-reports Difference in problem-solving performance between native and non-native students Difference 12 62 93 c 58 c 84 73 40 c c c c c a 44 c 70 78 c c c c 82 91 c 37 40 36 c 51 c c 58 31 –1 c c c c S.E (4.7) (5.8) (8.8) (4.7) c (8.7) c (14.9) (7.8) (7.4) c c c c c a (4.4) c (9.5) (5.3) (10.7) c c c c (10.4) (5.9) c (8.1) (4.0) (1.9) c (4.4) c c (20.7) (10.0) (6.2) (5.5) c c c c Note: Values that are statistically significant are indicated in bold (see Annex A4) Response rate too low to ensure comparability (see Annex A3) © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 153 Annex B Table 5.7 Percentage of students and performance on the problem-solving scale, by language spoken at home Results based on students’ self-reports Partner countries OECD countries Language spoken at home most of the time IS DIFFERENT from the language of assessment, from other official languages or from other national dialects Language spoken at home most of the time IS THE SAME as the language of assessment, other official languages or other national dialects % of students S.E Mean score S.E % of students S.E S.E Difference S.E Australia Austria Belgium Canada Czech Republic Denmark Finland France Germany Greece Hungary Iceland Ireland Italy Japan Korea Luxembourg Mexico Netherlands New Zealand Norway Poland Portugal Slovak Republic Spain Sweden Switzerland Turkey United States 8.9 9.0 4.8 11.2 0.9 3.9 1.8 6.1 7.7 3.2 0.6 1.6 0.8 1.6 0.2 0.1 25.0 1.1 4.6 9.0 4.5 0.2 1.4 1.4 1.7 6.9 9.5 1.2 9.0 (0.7) (0.7) (0.4) (0.7) (0.2) (0.5) (0.2) (0.7) (0.6) (0.4) (0.1) (0.2) (0.2) (0.2) (0.1) (0.0) (0.6) (0.3) (0.6) (0.7) (0.5) (0.1) (0.2) (0.3) (0.3) (0.7) (0.7) (0.6) (0.7) 515 458 450 524 c 475 c 458 430 401 c c c c c c 464 c 450 516 439 c c c c 456 453 c 440 (5.3) (7.8) (8.2) (4.4) c (10.2) c (9.1) (6.5) (11.0) c c c c c c (2.8) c (10.0) (6.3) (9.7) c c c c (9.3) (6.7) c (7.2) 91.1 91.0 95.2 88.8 99.1 96.1 98.2 93.9 92.3 96.8 99.4 98.4 99.2 98.4 99.8 99.9 75.0 98.9 95.4 91.0 95.5 99.8 98.6 98.6 98.3 93.1 90.5 98.8 91.0 (0.7) (0.7) (0.4) (0.7) (0.2) (0.5) (0.2) (0.7) (0.6) (0.4) (0.1) (0.2) (0.2) (0.2) (0.1) (0.0) (0.6) (0.3) (0.6) (0.7) (0.5) (0.1) (0.2) (0.3) (0.3) (0.7) (0.7) (0.6) (0.7) 533 514 539 536 523 520 549 526 531 451 502 506 498 474 551 551 507 386 530 535 495 487 472 494 482 516 534 408 484 (1.9) (3.2) (2.3) (1.6) (3.1) (2.5) (1.9) (2.5) (3.3) (3.9) (3.0) (1.4) (2.3) (3.0) (4.1) (3.1) (1.6) (4.2) (3.1) (2.4) (2.5) (2.7) (3.9) (3.2) (2.7) (2.2) (3.4) (6.0) (3.0) 18 56 89 11 c 44 c 69 101 49 c c c c c c 43 c 79 20 56 c c c c 61 81 c 44 (5.2) (7.7) (8.4) (4.3) c (10.4) c (9.3) (6.3) (11.0) c c c c c c (3.3) c (10.3) (7.0) (9.8) c c c c (9.3) (6.3) c (7.1) OECD total OECD average 4.5 4.5 (0.2) (0.1) 449 465 (4.1) (1.9) 90.7 91.2 (0.3) (0.1) 494 504 (1.1) (0.7) 46 39 (4.2) (2.0) 0.5 4.5 2.1 8.3 18.4 4.6 5.4 1.5 a 0.4 1.9 3.8 (0.1) (0.4) (0.3) (1.1) (2.2) (0.7) (1.3) (0.2) a (0.1) (0.4) (0.6) c 473 c 456 505 493 421 c a c c 489 c (9.8) c (8.8) (11.6) (13.6) (13.3) c a c c (11.7) 99.5 95.6 97.9 91.7 81.6 95.4 94.6 98.5 100.0 99.6 98.1 96.2 (0.1) (0.4) (0.3) (1.1) (2.2) (0.7) (1.3) (0.2) (0.0) (0.1) (0.4) (0.6) 372 553 362 486 543 535 482 421 426 344 414 512 (4.8) (4.0) (3.3) (3.9) (4.9) (2.8) (4.4) (3.4) (2.7) (2.2) (3.7) (2.5) c 80 c 30 38 42 61 c a c c 23 c (9.1) c (8.1) (12.6) (14.6) (12.4) c a c c (11.7) Brazil Hong Kong-China Indonesia Latvia Liechtenstein Macao-China Russian Federation Serbia Thailand Tunisia Uruguay United Kingdom1 Note: Values that are statistically significant are indicated in bold (see Annex A4) Response rate too low to ensure comparability (see Annex A3) 154 © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Mean score Difference in problem-solving performance (students speaking the same language – students speaking a different language) Annex C THE DEVELOPMENT AND IMPLEMENTATION OF PISA – A COLLABORATIVE EFFORT © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 155 Annex C Annex C: The development and implementation of PISA – a collaborative effort Introduction PISA is a collaborative effort, bringing together scientific expertise from the participating countries, steered jointly by their governments on the basis of shared, policy-driven interests A PISA Governing Board on which each country is represented determines, in the context of OECD objectives, the policy priorities for PISA and oversees adherence to these priorities during the implementation of the programme This includes the setting of priorities for the development of indicators, for the establishment of the assessment instruments and for the reporting of the results Experts from participating countries also serve on working groups that are charged with linking policy objectives with the best internationally available technical expertise By participating in these expert groups, countries ensure that the instruments are internationally valid and take into account the cultural and educational contexts in OECD Member countries, the assessment materials have strong measurement properties, and the instruments place an emphasis on authenticity and educational validity Through National Project Managers, participating countries implement PISA at the national level subject to the agreed administration procedures National Project Managers play a vital role in ensuring that the implementation of the survey is of high quality, and verify and evaluate the survey results, analyses, reports and publications The design and implementation of the surveys, within the framework established by the PISA Governing Board, is the responsibility of an international consortium, referred to as the PISA Consortium, led by the Australian Council for Educational Research (ACER) Other partners in this consortium include the Netherlands National Institute for Educational Measurement (Citogroep), The National Institute for Educational Research in Japan (NIER), the Educational Testing Service in the United States (ETS), and WESTAT in the United States The OECD Secretariat has overall managerial responsibility for the programme, monitors its implementation on a day-to-day basis, acts as the secretariat for the PISA Governing Board, builds consensus among countries and serves as the interlocutor between the PISA Governing Board and the international consortium charged with the implementation of the activities The OECD Secretariat also produces the indicators and analyses and prepares the international reports and publications in cooperation with the PISA consortium and in close consultation with Member countries both at the policy level (PISA Governing Board) and at the level of implementation (National Project Managers) The following lists the members of the various PISA bodies and the individual experts and consultants who have contributed to PISA Members of the PISA Governing Board Hong Kong-China: Esther Ho Sui Chu Chair: Ryo Watanabe Hungary: Péter Vári Iceland: Júlíus K Bjưrnsson Australia: Wendy Whitham Indonesia: Bahrul Hayat Austria: Helmut Bachmann and Jürgen Horschinegg Ireland: Gerry Shiel Belgium: Dominique Barthélémy, Christiane Blondin and Italy: Giacomo Elias and Angela Vegliante Liselotte van de Perre Brazil: Eliezer Pacheco Canada: Satya Brink and Dianne Pennock Czech Republic: Jan Koucky Denmark: JØrgen Balling Rasmussen Finland: Jari Rajanen France: Gérard Bonnet Germany: Hans Konrad Koch, Elfriede Ohrnberger and Botho Priebe Greece: Vassilis Koulaidis 156 Japan: Ryo Watanabe Korea: Kye Young Lee Latvia: Andris Kangro Luxembourg: Michel Lanners Macao-China: Lam Fat Lo Mexico: Felipe Martínez Rizo Netherlands: Jules L Peschar New Zealand: Lynne Whitney Norway: Alette Schreiner Poland: Stanislaw Drzazdzewski © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 Spain: Guillermo Gil Russian Federation: Galina Kovalyova Sweden: Karin Taube Serbia: Dragica Pavlovic Babic Switzerland: Huguette McCluskey Slovak Republic: Vladimir Repas Thailand: Sunee Klainin Spain: Carme Amorós Basté, Guillermo Gil and Tunisia: Néjib Ayed Josu Sierra Orrantia Sweden: Anita Wester Switzerland: Katrin Holenstein and Heinz Rhyn Thailand: Sunee Klainin Tunisia: Nộjib Ayed Turkey: Sevki Karaca and Ruhi Kilỗ United Kingdom: Lorna Bertrand and Liz Levy United States: Mariann Lemke and Elois Scott Uruguay: Pedro Ravela Special Advisor: Eugene Owen PISA 2003 National Project Managers Australia: John Cresswell and Sue Thomson Austria: Günter Haider and Claudia Reiter Belgium: Luc van de Poele Brazil: Mariana Migliari Canada: Tamara Knighton and Dianne Pennock Czech Republic: Jana Paleckova Denmark: Jan Mejding Finland: Jouni Välijärvi France: Anne-Laure Monnier Germany: Manfred Prenzel Greece: Vassilia Hatzinikita Hong Kong-China: Esther Ho Sui Chu Hungary: Péter Vári Iceland: Almar Midvik Halldorsson Indonesia: Bahrul Hayat Ireland: Judith Cosgrove Italy: Maria Teresa Siniscalco Japan: Ryo Watanabe Korea: Mee-Kyeong Lee Latvia: Andris Kangro Luxembourg: Iris Blanke Macao-China: Lam Fat Lo Mexico: Rafael Vidal Netherlands: Erna Gille New Zealand: Fiona Sturrock Norway: Marit Kjaernsli Poland: Michal Federowicz Portugal: Lídia Padinha Russian Federation: Galina Kovalyova Serbia: Dragica Pavlovic Babic Slovak Republic: Paulina Korsnakova Annex C Portugal: Glória Ramalho Turkey: Sevki Karaca United Kingdom: Rachael Harker, Graham Thorpe United States: Mariann Lemke Uruguay: Pedro Ravela OECD Secretariat Andreas Schleicher (overall co-ordination of PISA and Member country relations) Miyako Ikeda (project management) Claire Shewbridge (project management) Claudia Tamassia (project management) Sophie Vayssettes (statistical support) Juliet Evans (administrative support) Kate Lancaster (editorial support) PISA Expert Groups Mathematics Expert Group Jan de Lange (Chair) (Utrecht University, The Netherlands) Werner Blum (Chair) (University of Kassel, Germany) Vladimir Burjan (National Institute for Education, Slovak Republic) Sean Close (St Patrick’s College, Ireland) John Dossey (Consultant, United States of America) Mary Lindquist (Columbus State University, United States of America) Zbigniew Marciniak (Warsaw University, Poland) Mogens Niss (Roskilde University, Denmark) Kyung-Mee Park (Hongik University, Korea) Luis Rico (University of Granada, Spain) Yoshinori Shimizu (Tokyo Gakugei University, Japan) Reading Expert Group Irwin Kirsch (Chair) (Educational Testing Service, United States) Marilyn Binkley (National Center for Educational Statistics, United States) Alan Davies (University of Edinburgh, United Kingdom) Stan Jones (Statistics Canada, Canada) John de Jong (Language Testing Services, The Netherlands) Dominique Lafontaine (Université de Liège Sart Tilman, Belgium) Pirjo Linnakylä (University of Jyväskylä, Finland) Martine Rémond (Institut National de Recherche Pédagogique, France) © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 157 Annex C Science Expert Group Wynne Harlen (Chair) (University of Bristol, United Kingdom) Peter Fensham (Monash University, Australia) Raul Gagliardi (University of Geneva, Switzerland) Svein Lie (University of Oslo, Norway) Manfred Prenzel (Universität Kiel, Germany) Senta A Raizen (National Center for Improving Science Education (NCISE), United States) Donghee Shin (KICE, Korea) Elizabeth Stage (University of California, United States) Problem Solving Expert Group John Dossey (Chair) (Consultant, United States of America) Beno Csapo (University of Szeged, Hungary) Jan De Lange (Utrecht University, The Netherlands) Eckhard Klieme (German Institute for International Educational Research, Germany) Wynne Harlen (University of Bristol, United Kingdom) Ton de Jong (University of Twente, The Netherlands) Irwin Kirsch (Educational Training Service, United States) Stella Vosniadou (University of Athens, Greece) PISA Technical Advisory Group Keith Rust (Chair) (Westat) Ray Adams (ACER, Australia) Pierre Foy (Statistics Canada, Canada) Aletta Grisay (Belgium) Larry Hedges (The University of Chicago, United States) Eugene Johnson (American Institutes for Research, United States) John de Jong (Language Testing Services, The Netherlands) Irwin Kirsch (Educational Testing Service, United States) Steve May (Ministry of Education, New Zealand) Christian Monseur (HallStat SPRL, Belgium) Norman Verhelst (Citogroep, The Netherlands) J Douglas Willms (University of New Brunswick, Canada) PISA Consortium Australian Council for Educational Research Ray Adams (Project Director of the PISA Consortium) Alla Berezner (data management, data analysis) Eveline Gerbhardt (data processing, data analysis) Marten Koomen (management) Dulce Lay (data processing) Le Tu Luc (data processing) Greg Macaskill (data processing) Barry McCrae (science instruments, test development mathematics and problem solving) Martin Murphy (field operations and sampling) 158 Van Nguyen (data processing) Alla Routitsky (data processing) Wolfram Schulz (Coordinator questionnaire development data processing, data analysis) Ross Turner (Coordinator test development) Maurice Walker (sampling, data processing, questionnaire development) Margaret Wu (test development mathematics and problem solving, data analysis) John Cresswell (test development science) Juliette Mendelovits (test development reading) Joy McQueen (test development reading) Beatrice Halleux (translation quality control) Westat Nancy Caldwell (Director of the PISA Consortium for field operations and quality monitoring) Ming Chen (weighting) Fran Cohen (weighting) Susan Fuss (weighting) Brice Hart (weighting) Sharon Hirabayashi (weighting) Sheila Krawchuk (sampling and weighting) Christian Monseur (consultant) (weighting) Phu Nguyen (weighting) Mats Nyfjall (weighting) Merl Robinson (field operations and quality monitoring) Keith Rust (Director of the PISA Consortium for sampling and weighting) Leslie Wallace (weighting) Erin Wilson (weighting) Citogroep Steven Bakker (science test development) Bart Bossers (reading test development) Truus Decker (mathematics test development) Erna van Hest (reading test development and quality monitoring) Kees Lagerwaard (mathematics test development) Gerben van Lent (mathematics test development) Ico de Roo (science test development) Maria van Toor (office support and quality monitoring) Norman Verhelst (technical advice, data analysis) Educational Testing Service Irwin Kirsch (reading test development) Other experts Cordula Artelt (questionnaire development) Aletta Grisay (technical advice, data analysis, translation, questionnaire development) Donald Hirsch (editorial review) © OECD 2004 Problem Solving for Tomorrow’s World – First Measures of Cross-Curricular Competencies from PISA 2003 OECD PUBLICATIONS, rue André-Pascal, PARIS CEDEX 16 PRINTED IN FRANCE (962004131P1) ISBN 92-64-00642-7 – No 53833 2004