Introduction

With the increasing emphasis on science, technology, engineering, and mathematics

Assessing students in core subjects like mathematics and science is crucial for their success in STEM education and careers Identifying their strengths and weaknesses throughout their educational journey helps to enhance their learning outcomes and prepare them for future opportunities in these vital fields.

Addressing persistent student misconceptions, errors, and misunderstandings in higher grade levels is crucial for stakeholders in science and mathematics education, including teachers, educators, policymakers, and researchers This report examines specific misconceptions and errors related to fundamental physics and mathematics concepts, highlighting the connection between these misunderstandings and a lack of foundational knowledge from earlier grades The findings aim to enhance the teaching, learning, and reinforcement of core concepts across elementary, middle, and secondary education.

This study analyzes assessment items and student performance data from the Trends in International Mathematics and Science Study (TIMSS) and TIMSS Advanced assessments over 20 years (1995–2015) to evaluate students' understanding of key topics such as gravity and linear equations, as well as their misconceptions and errors The research focuses on five countries—Italy, Norway, the Russian Federation, Slovenia, and the United States—that participated in the TIMSS Advanced 2015 assessment and have a history of involvement in TIMSS assessments for grades four and eight By selecting these countries, the study enhances the ability to compare performance trends across different grade levels and over time, providing valuable insights into educational outcomes.

In the TIMSS 2015 assessment, France, Lebanon, Portugal, and Sweden did not participate at either the fourth or eighth-grade levels, or they had incomplete data for more than one previous assessment cycle for at least one grade A detailed summary of each country's specific assessments can be found in Chapter 3.

Leveraging TIMSS and TIMSS Advanced assessment data to investigate student misconceptions and errors offers significant benefits These assessments have been conducted on nationally representative student samples since 1995, with the latest evaluations occurring in 2015, providing a robust dataset for analysis In comparison, many research studies lack this level of comprehensive and consistent data.

The Trends in International Mathematics and Science Study (TIMSS) is a prominent research initiative by the International Association for the Evaluation of Educational Achievement (IEA), managed by the TIMSS & PIRLS International Study Centre at Boston College This study aims to evaluate trends in mathematics and science performance globally while gathering data on educational contexts that may influence student success TIMSS and TIMSS Advanced involve collaboration with national research coordinators from each participating educational system For further details, visit www.iea.nl/timss.

2 Although our study focuses on these speci fi c countries, the methodology described can be applied to an individual education system or any set of education systems.

3 TIMSS has been administered every four years, starting in 1995 (although the 1999 assessment was administered at grade eight only), and TIMSS Advanced was administered in 1995, 2008, and 2015.

Student misconceptions are often investigated through limited samples from specific regions or schools, as noted by Alonzo et al (2012) The Trends in International Mathematics and Science Study (TIMSS) allows for tracking the performance of student cohorts across three grade levels and multiple assessment years, enabling a comprehensive evaluation of misconceptions over time Additionally, TIMSS and TIMSS Advanced offer access to released assessment items and detailed student performance data, which can be utilized for research, including diagnostic item-level analysis.

The results may provide a more comprehensive picture of student performance within and across countries.

TIMSS and TIMSS Advanced data have facilitated various secondary analyses focused on student misconceptions across multiple countries, as evidenced by studies conducted by Angell (2004), Juan et al (2017), Mosimege et al (2017), and Prinsloo et al.

2017; Provasnik et al 2019; Saputro et al 2018; Văcăreţu n.d.; Yung 2006).

Following the release of the 2015 TIMSS and TIMSS Advanced results in the

United States (Provasnik et al 2016), the American Institutes for Research

(AIR) conducted in-depth secondary analyses of TIMSS and TIMSS Advanced data from the United States An initial report on the United States’ performance in

The TIMSS Advanced 2015 report highlighted specific strengths and weaknesses in advanced mathematics and physics, as well as prevalent misconceptions and errors among students (Provasnik et al., 2019) A subsequent study utilized both TIMSS and TIMSS data to further analyze these findings.

Advanced data further explored how physics misconceptions demonstrated by

TIMSS Advanced students in the United States can be traced back to misconcep- tions, or a lack of foundational understanding about physics concepts in earlier grades (unpublished work) 4

This report builds on previous research to examine misconceptions, errors, and misunderstandings in physics and mathematics We analyze patterns of these issues across different grade levels in a selected group of countries, highlighting variations by country, overall performance, and gender differences Additionally, we investigate changes in these patterns over various assessment years.

De fi ning the Terminology

Performance Objectives

Performance objectives derived from TIMSS and TIMSS Advanced items outline the specific knowledge and skills students must possess at various grade levels to successfully tackle assessment questions This report identifies four performance objectives related to gravity and nine pertaining to linear equations, each evaluated through multiple assessment items While some objectives are assessed at a single grade level, others span two levels, such as TIMSS Advanced/eighth grade or eighth/fourth grade, and certain physics objectives are evaluated across all three grade levels.

Misconceptions in Physics

Misconceptions in physics arise from students' incorrect preconceived notions, often rooted in their everyday experiences with physical phenomena This report illustrates these misconceptions through specific types of student responses, highlighting incorrect choices in multiple-choice questions and miscategorized answers in constructed-response items, where students articulate their thoughts in writing.

Errors in Mathematics

Errors in mathematics occur when students do not follow the required procedures to arrive at the correct answer These errors indicate any response that fails to achieve the correct result.

Misunderstandings in Physics and Mathematics

Misunderstandings in physics and mathematics occur when students fail to grasp the underlying concepts related to specific problems These misunderstandings are not due to procedural errors in mathematics or specific misconceptions in physics, but rather indicate a lack of comprehension of the relevant principles.

The article discusses assessments that require students to demonstrate their grasp of physics concepts through constructed-response items, highlighting that specific incorrect response types are not monitored It emphasizes that misunderstandings in physics often reflect a fundamental lack of comprehension of the subject matter.

4 1 An Introduction to Student Misconceptions … understanding and include all incorrect responses (including off-task and blank responses).

The article discusses scenarios where students encounter questions that lack a defined procedure, requiring them to apply their understanding of mathematical concepts independently It highlights that misunderstandings in mathematics can manifest through specific incorrect answers or entirely incorrect responses, including off-task and blank submissions.

Core Concepts in Physics and Mathematics

Our approach emphasizes fundamental concepts in physics and mathematics, starting from elementary school and evolving through middle and secondary education To illustrate our methodology for addressing students' misconceptions and errors, we focus on gravity in physics and linear equations in mathematics These subjects are essential topics featured in both the TIMSS assessments, highlighting their significance in educational curricula.

Advanced assessment frameworks include items related to these topics in both the grade four and eight assessments, as well as the TIMSS Advanced assessment This comprehensive approach enables us to identify misconceptions, errors, and misunderstandings across all three grade levels.

Gravity is a fundamental concept that students encounter early in their education, shaped by their everyday experiences and observations It is explored in various subjects, including physical science, earth science, and advanced physics courses in secondary school, with expectations for a deeper understanding as students progress through grades The study of gravitational force serves as an effective context for assessing students' abilities to apply concepts of force, while also helping to identify common misconceptions about force and motion across different educational levels.

According to the TIMSS 2015 frameworks (Jones et al., 2013), fourth-grade students can identify gravity as the force that attracts objects toward Earth and understand that forces can alter an object's motion By eighth grade, students are able to describe various mechanical forces, including gravitational force, and predict how these forces influence an object's motion Additionally, they recognize that gravity is responsible for keeping planets and moons in orbit and for pulling objects toward the Earth's surface.

The 2015 TIMSS Advanced physics framework, as outlined by Jones et al (2014), anticipates that by the conclusion of secondary education, students will be able to apply Newton's laws of motion to elucidate various types of motion and understand how the interaction of multiple forces affects an object's movement.

Linear equations are a fundamental aspect of mathematics education, spanning from elementary to secondary school According to the 2015 TIMSS mathematics framework, fourth-grade students learn to represent problem situations with expressions and number sentences, while eighth graders progress to writing equations and solving simultaneous linear equations By the end of secondary school, students are expected to solve linear and quadratic equations, as well as systems of equations and inequalities, applying these concepts to real-world problems This progression highlights the importance of linear equations in developing students' mathematical skills throughout their education.

Not only do students continue to study the topic of linear equations across grades, their conceptual understanding of linear equations progresses from concrete (number

Table 1.1 TIMSS 2015 and TIMSS Advanced 2015 assessment framework objectives related to gravity

TIMSS grade 4 TIMSS grade 8 TIMSS Advanced physics

• Identify gravity as the force that draws objects to Earth

Pushing and pulling forces can significantly alter an object's motion, and it is essential to compare how forces of varying strengths, whether acting in the same or opposite directions, influence an object's movement Understanding these dynamics is crucial for grasping the principles of physics and mechanics.

• Describe common mechanical forces, including gravitational, normal, friction, elastic, and buoyant forces, and weight as a force due to gravity

• Predict qualitative one-dimensional changes in motion (if any) of an object based on the forces acting on it Earth science

• Recognize that it is the force of gravity that keeps the planets and moons in orbits as well as pulls objects to Earth ’ s surface

This article discusses how to predict and determine the position, displacement, and velocity of objects based on initial conditions It emphasizes the application of Newton's laws of motion to explain the dynamics of various types of motion Additionally, it provides methods to calculate essential parameters such as displacement, velocity, acceleration, distance traveled, and time elapsed, ensuring a comprehensive understanding of motion in physics.

Understanding the forces acting on a body—whether at rest, in constant motion, or accelerating—is crucial for analyzing its motion This includes identifying frictional forces and their combined effects on the body's movement The insights gained from these concepts are essential for solving related physics problems This content aligns with the objectives outlined in the 2015 TIMSS and TIMSS Advanced frameworks, focusing on key physics principles and assessment criteria.

Source International Association for the Evaluation of Educational Achievement (IEA), Trends in International Mathematics and Science Study (TIMSS) 2015 and TIMSS Advanced 2015 assessment frameworks (Jones et al 2013, 2014)

6 1 An Introduction to Student Misconceptions … sentences at grade four) to abstract (equations and graphical representations at grade eight and the upper secondary level) as their mathematics competency progresses.

Students' performance in algebra is crucial for higher achievement in mathematics, as highlighted by Walston and McCarroll (2010) Linear equations serve as a foundational topic in algebra, being simpler than quadratic and exponential equations Mastery of linear equations is essential for understanding more complex concepts like intercepts and slope Additionally, the versatility of linear equations allows for connections to various subject areas and real-world applications, such as graphing in science and financial literacy in everyday life Focusing on linear equations is vital for evaluating students' performance in a key area that contributes to their postsecondary success.

Table 1.2 TIMSS 2015 and TIMSS Advanced 2015 assessment framework objectives related to linear equations: 2015

TIMSS grade 4 TIMSS grade 8 TIMSS Advanced mathematics

• Identify or write expressions or number sentences to represent problem situations involving unknowns

• Identify and use relationships in a well-de fi ned pattern (e.g., describe the relationship between adjacent terms and generate pairs of whole numbers given a rule)

• Solve problems set in contexts, including those involving measurements, money, and simple proportions

• Read, compare, and represent data from tables, and line graphs

• Write equations or inequalities to represent situations

• Solve linear equations, linear inequalities, and simultaneous linear equations in two variables

• Interpret, relate, and generate representations of functions in tables, graphs, or words

• Interpret the meanings of slope and y-intercept in linear functions

• Solve linear and quadratic equations and inequalities as well as systems of linear equations and inequalities

• Use equations and inequalities to solve contextual problems

Notes This outlines the portion of the objectives included in the 2015 TIMSS and TIMSS

Advanced frameworks that speci fi cally relate to the mathematics concepts and assessment items discussed in this report

Source International Association for the Evaluation of Educational Achievement (IEA), Trends in

International Mathematics and Science Study (TIMSS) 2015 and TIMSS Advanced 2015 assessment frameworks (Gr ứ nmo et al 2013, 2014)

1.3 Core Concepts in Physics and Mathematics 7

Research Questions

Our methodology comprises three key components: first, we conduct a review of the assessment framework and content mapping to pinpoint the items that measure relevant topics for each grade level; second, we evaluate diagnostic item-level performance data to identify specific performance objectives and uncover common misconceptions, errors, and misunderstandings; and third, we analyze the percentage of students exhibiting these misconceptions across different countries, grade levels, genders, and assessment years The report includes example items to illustrate the types of misconceptions and errors observed among students at each grade level.

Using item-level performance data from multiple assessment cycles of TIMSS and TIMSS Advanced (from 1995 to 2015), we addressed three research questions.

This research investigates prevalent misconceptions, errors, and misunderstandings among students in grade four, grade eight, and final year secondary school (TIMSS Advanced students), while also comparing these findings across different countries By analyzing these common challenges, the study aims to identify patterns and variations in student understanding, providing insights into educational practices and curriculum effectiveness globally.

We analyzed the frequency of misconceptions, errors, and misunderstandings regarding gravity and linear equations in various countries, using data from multiple grade levels Our study identified and compared patterns of these misconceptions across different countries and educational stages.

Research question 2: How do student misconceptions, errors, and misunderstandings differ by gender?

We analyzed student performance on each assessment item, identifying variations in misconceptions, errors, and misunderstandings based on gender Additionally, we compared these differences across various countries and grade levels.

Research question 3: How persistent are patterns in misconceptions, errors, and misunderstandings over time?

We analyzed trend items from various assessment cycles to evaluate the prevalence of specific misconceptions, errors, and misunderstandings across TIMSS assessments from 1995 to 2015 This comparison aimed to identify changes in patterns among countries over time, particularly regarding the persistence of certain educational challenges.

This report features six example items, including "restricted-use" items from the TIMSS 2015 assessments and previously released items The "restricted-use" items are intended for inclusion in international and national reports by participating countries, as well as for secondary research While the report focuses on released and restricted-use items, it also analyzes appropriate non-released (secure) items from 2015 to address misconceptions, though these are not displayed in the report All example items, both "restricted-use" and "released," are presented with permission from IEA.

8 1 An Introduction to Student Misconceptions … misconceptions at grades four or eight increase, decrease, or stay the same between

This report examines student misconceptions, errors, and misunderstandings in gravity and linear equations, while also presenting a methodology applicable to various mathematics and science topics within TIMSS and TIMSS Advanced This approach allows for the tracing of misconceptions across multiple grade levels, such as grades four and eight, or can focus on a single grade The findings provide valuable insights for instructional improvement by linking country-specific patterns of misconceptions to curriculum gaps and deficiencies.

Alonzo, A C., Neidorf, T., & Anderson, C W (2012) Using learning progressions to inform large-scale assessment In A C Alonzo, & A W Gotwals (Eds.), Learning progressions in science Rotterdam, The Netherlands: Sense Publishers.

Angell, C (2004) Exploring students ’ intuitive ideas based on physics items in TIMSS-1995 In C.

Papanastasiou (Ed.), Proceedings of the IRC-2004 TIMSS IEA International Research

Conference (Vol 2, pp 108 – 123) Nicosia, Cyprus: University of Cyprus Retrieved from https://www.iea.nl/sites/default/ fi les/2019-03/IRC2004_Angell.pdf.

Gr ứ nmo, L S., Lindquist, M., & Arora, A (2014) TIMSS Advanced 2015 mathematics framework In I.V S Mullis, & M O Martin (Eds.), TIMSS Advanced 2015 assessment frameworks (pp 9 – 15) Chestnut Hill, MA: TIMSS & PIRLS International Study Center,

Boston College Retrieved from https://timss.bc.edu/timss2015-advanced/downloads/TA15_

Gr ứ nmo, L S., Lindquist, M., Arora, A., & Mullis, I.V S (2013) TIMSS 2015 mathematics framework In I.V S Mullis, & M O Martin (Eds.), TIMSS 2015 assessment frameworks

(pp 11 – 27) Chestnut Hill, MA: TIMSS & PIRLS International Study Center, Boston College.

Retrieved from https://timssandpirls.bc.edu/timss2015/downloads/T15_FW_Chap1.pdf.

Jones, L R., Wheeler, G., & Centurino, V A S (2013) TIMSS 2015 science framework In I.V.

S Mullis, & M O Martin (Eds.) TIMSS 2015 assessment frameworks (pp 29 – 58) Chestnut

Hill, MA: TIMSS & PIRLS International Study Center, Boston College Retrieved from https:// timssandpirls.bc.edu/timss2015/downloads/T15_FW_Chap2.pdf.

Jones, L R., Wheeler, G., & Centurino, V A S (2014) TIMSS Advanced 2015 physics framework In I V S Mullis and M O Martin (Eds.), TIMSS Advanced 2015 assessment frameworks (pp 17 – 25) Chestnut Hill, MA: TIMSS & PIRLS International Study Center,

Boston College Retrieved from https://timss.bc.edu/timss2015-advanced/downloads/TA15_

Juan, A., Hannan, S., Zulu, N., Harvey, J C., Prinsloo, C H., Mosimege, M., & Beku, U (2017).

TIMSS item diagnostic report: South Africa: Grade 5 Numeracy (Commissioned by the

Department of Basic Education) South Africa: Human Sciences Research Council Retrieved from http://ecommons.hsrc.ac.za/handle/20.500.11910/11447.

7 Trends in misconceptions, errors, and misunderstandings are only reported for grade four and grade eight, as there were no trend items measuring the selected topics in TIMSS Advanced.

The TIMSS item diagnostic report for South Africa, focusing on Grade 9 Mathematics, was commissioned by the Department of Basic Education and authored by Mosimege et al (2017) This comprehensive report, published by the Human Sciences Research Council, provides valuable insights into the mathematics performance of South African students For further details, the report can be accessed at the Human Sciences Research Council's repository.

The TIMSS item diagnostic report for South Africa, focusing on Grade 9 Science, was authored by Prinsloo et al in 2017 This report, commissioned by the Department of Basic Education, is published by the Human Sciences Research Council and provides insights into the performance and understanding of science among Grade 9 learners in South Africa For more details, you can access the report [here](http://www.hsrc.ac.za/en/research-data/ktree-doc/19286).

Provasnik, S., Malley, L., Neidorf, T, Arora, A., Stephens, M., Balestreri, K., Perkins, R., & Tang,

J H (2019, in press) U.S performance on the 2015 TIMSS Advanced mathematics and physics assessments: A closer look (NCES 2017-020) Washington, DC: US Department of Education, National Center for Education Statistics.

The 2016 report by Provasnik et al presents key findings from the TIMSS and TIMSS Advanced 2015 assessments, focusing on the mathematics and science performance of U.S students in grades 4 and 8, as well as those in advanced high school courses Published by the National Center for Education Statistics, the document highlights the achievements of American students and provides insights into educational outcomes For further details, the full report is available at the U.S Department of Education's website.

Saputro, B A., Suryadi, D., Rosjanuardi, R., & Kartasasmita, B G (2018) Analysis of students ’ errors in responding to TIMSS domain algebra problem Journal of Physics: Conference Series, 1088, 012031 Retrieved from https://iopscience.iop.org/article/10.1088/1742-6596/ 1088/1/012031.

V ă c ă re ţ u, A (n.d.) Using the TIMSS results for improving mathematics learning Cluj-Napoca, Romania: Romanian Reading and Writing for Critical Thinking Association Retrieved from http://directorymathsed.net/montenegro/Vacaretu.pdf.

Walston, J., & McCarroll, J C (2010) Eighth-grade algebra: Findings from the eighth-grade round of the early childhood longitudinal study, kindergarten class of 1998 – 99 (ECLS-K). Statistics in Brief, 16, 1 – 20.

The publication edited by Yung, B H W in 2006, titled "Learning from TIMSS: Implications for Teaching and Learning Science at the Junior Secondary Level," provides valuable insights into enhancing science education for junior secondary students Released by the TIMSS HK IEA Centre and the Education and Manpower Bureau in Hong Kong, this document emphasizes the importance of effective teaching strategies and learning methodologies derived from the Trends in International Mathematics and Science Study (TIMSS) For further details, the full text can be accessed at the Education Bureau's website.

This chapter is licensed under the Creative Commons Attribution-NonCommercial 4.0 International License, allowing noncommercial use, sharing, adaptation, and reproduction in any format, provided that appropriate credit is given to the original authors and the source, along with a link to the license Images and third-party materials are included under this license unless specified otherwise For any material not covered by this license, permission must be obtained from the copyright holder if the intended use exceeds permitted guidelines.

10 1 An Introduction to Student Misconceptions …

Review of Research into Misconceptions and Misunderstandings in Physics and Mathematics

Abstract Many diagnostic methods have been used to analyze data from large-scale assessments such as the Trends in International Mathematics and

This literature review examines various diagnostic models employed to investigate student misconceptions in mathematics and science, highlighting their comparison to the methodology of the current study It synthesizes extensive prior research on student misunderstandings in physics, particularly regarding gravitational force, and in mathematics, focusing on linear equations, thereby linking established literature to the present investigation.

Keywords Diagnostic models Errors Gravity International large-scale assessment Linear equations Mathematics Misconceptions Physics

Science Student achievement Trend analysis Trends in International

Mathematics and Science Study (TIMSS)

Introduction

Traditional methods of measuring student achievement primarily assess what students know through correct answers, as seen in large-scale assessments like IEA’s TIMSS, which utilize unidimensional models such as item response theory (IRT) However, recent research has shifted towards multidimensional models that analyze both correct and incorrect responses to better evaluate specific skills, abilities, and misconceptions Understanding student misconceptions has been shown to be crucial for enhancing learning outcomes in subjects like physics and mathematics.

The literature review is organized into three sections, with the first focusing on various diagnostic models employed to investigate student characteristics, misconceptions, misunderstandings, and errors in mathematics and science.

The second and third sections of the article delve into previous studies on student misconceptions and errors in physics, specifically regarding gravitational force, and in mathematics, focusing on linear equations Additionally, these sections examine gender differences in the occurrence of these misconceptions.

Diagnostic Models Overview

Traditional psychometric models, such as Item Response Theory (IRT), primarily measure a single latent ability, which limits their effectiveness in assessing student knowledge (Bradshaw and Templin, 2014) This unidimensional approach has been criticized for its inadequacy in providing diagnostic insights (De la Torre and Minchen, 2014) As a result, there has been a growing demand for more sophisticated models that deliver detailed diagnostic information, leading to the emergence of cognitive diagnostic models (CDMs).

A Cognitive Diagnosis Model (CDM) categorizes students based on their mastery of various attributes, identifying their abilities through specific skills they have or have not acquired One prominent example of a CDM is the Diagnostic Classification Model (DCM), which employs distractor-driven assessments to evaluate both positive and negative facets of student reasoning, often utilizing multiple-choice tests to assess multidimensional skills.

In addition to the DCM, various other cognitive diagnostic models (CDMs) exist, including the rule space model introduced by Tatsuoka in 1983, the deterministic input, noisy "and" gate (DINA) model developed by Junker and Sijtsma in 2001, the noisy input, deterministic "and" gate (NIDA) model proposed by Maris in 1999, and the reparametrized unified model (RUM) created by Roussos et al These models contribute significantly to the field of cognitive assessment and diagnostic evaluation.

Cognitive Diagnostic Models (CDMs) differ in complexity, parameter assignment, and assumptions regarding random noise in test-taking (Huebner and Wang, 2011) Their multidimensional characteristics enhance their effectiveness for educational diagnoses A study by Yamaguchi and Okada (2018) demonstrated that CDMs provided a superior fit compared to Item Response Theory (IRT) models when analyzing TIMSS 2007 mathematics data.

The Scaling Individuals and Classifying Misconceptions (SICM) model, introduced by Bradshaw and Templin in 2014, merges the Item Response Theory (IRT) model with the Diagnostic Classification Model (DCM) to create a statistical tool for measuring misconceptions in education This innovative approach focuses on analyzing incorrect responses by modeling categorical latent variables that signify misconceptions rather than skills To effectively categorize these misconceptions, the authors reference established inventories, including the Force Concept Inventory developed by Hestenes et al in 1992, which assesses understanding of Newtonian concepts of force.

Large-scale assessments like TIMSS present challenges for the application of current diagnostic models, as these assessments were not originally designed to function as cognitive diagnostic tools that evaluate specific skills or abilities.

12 2 Review of Research into Misconceptions … were they designed using a CDM with pre-defined attributes (de la Torre and

Research indicates that applying specific analytical approaches to TIMSS data can yield valuable insights into test takers' performance For instance, Dogan and Tatsuoka (2008) utilized the rule space model to assess and interpret the results effectively.

Turkish performance on the TIMSS 1999 grade eight mathematics assessment (also known as the Third International Mathematics and Science Study-Repeat, or

A study on the TIMSS-R revealed that Turkish students struggle with algebraic rule application and quantitative reading skills Additionally, research by Choi et al (2015) employed a CDM approach to analyze and compare the mathematics performance of eighth-grade students from the United States and Korea on the TIMSS assessment.

Research indicates that while Cognitive Diagnostic Models (CDM) can provide insightful data regarding student understanding in the TIMSS assessment, there are recognized limitations in the application of these models.

CDMs and SICMs utilize best-fit models to effectively predict student-level proficiency and misconceptions, particularly in the context of computer adaptive tests (CATs), which ensure uniform measurement precision for all test takers (Hsu et al 2013) However, the TIMSS assessments, which lack a focus on student-level reporting and are not computer-adaptive, do not align with the requirements of CDMs and SICMs Due to the design of TIMSS, where only a subset of items is administered to each student, the ability to make comprehensive claims about student proficiency in specific skills and concepts is significantly restricted.

In contrast to research using the types of diagnostic models described above, our study used a different diagnostic approach based on item-level performance data

Analyzing frequency distributions of response categories for individual assessment items reveals the nature and extent of students' misconceptions, errors, and misunderstandings reflected in their incorrect answers This method aligns with approaches used in other TIMSS-participating countries to effectively describe student understanding and identify misconceptions based on their responses.

TIMSS and TIMSS Advanced mathematics and science items at different grade levels (Angell2004; Juan et al.2017; Mosimege et al.2017; Prinsloo et al.2017;

Research by Provasnik et al (2019), Saputro et al (2018), Văcăreţu (n.d.), and Yung (2006) highlights the significance of international assessments in evaluating student performance For instance, Angell (2004) examined the results of Norwegian students on TIMSS Advanced 1995 physics items Additionally, diagnostic reports from South Africa utilized item-level data from TIMSS 2015 to assess the mathematics performance of fifth and ninth-grade students (Juan et al 2017; Mosimege et al n.d.).

In recent studies, including those by Prinsloo et al (2017) and Saputro et al (2018), researchers analyzed student performance on TIMSS assessments to identify common errors in mathematics and science for ninth graders in Indonesia These reports highlighted that misconceptions often varied by context and could be overlooked in broader evaluations, emphasizing the need for targeted approaches to address specific misunderstandings in student responses.

1 TIMSS uses a matrix-sampling design whereby a student is administered only a sample of the assessment items; most items are missing by design for each student.

Our research emphasizes the analysis of grouped assessment items that evaluate specific physics and mathematics concepts, such as gravity and linear equations, across various grade levels By examining student performance on these items, we identify patterns in misconceptions related to gender, countries, and assessment cycles Leveraging the TIMSS assessment design, this item-level data approach offers valuable insights for making country-level inferences and understanding the evolution of student misconceptions in diverse cultural contexts over time.

Misconceptions in Physics

Extensive research has been conducted on physics misconceptions, particularly regarding gravity, among students of all educational levels, including primary, secondary, and university students, as well as pre-service teachers Studies have shown that these misconceptions often stem from intuitive beliefs or preconceived notions about physical observations and processes, highlighting the prevalence of alternative conceptions in understanding gravitational force.

When analyzing misconceptions in physics, many researchers have focused on

Students often enter the classroom with "common sense beliefs," which are intuitions about physical phenomena shaped by personal experiences (Halloun and Hestenes, 1985a,b) These beliefs frequently conflict with scientific explanations taught in formal education, making them challenging to overcome and potentially hindering the comprehension of advanced physics concepts if not addressed promptly Numerous studies have explored these misconceptions, leading to the development of diagnostic tests, notably the Force Concept Inventory, which utilizes multiple-choice questions to identify student misconceptions related to these "common sense beliefs" (Hestenes et al., 1992) Research indicates that targeted instruction aimed at addressing these misconceptions is the most effective way to help students overcome their misunderstandings (Eryilmaz, 2002; Hestenes et al., 1992; Thornton et al., 2009).

Misconceptions rooted in common-sense beliefs often clash with fundamental physics concepts, including Newton's laws Research indicates that many students mistakenly believe a force is always present in the direction of motion, a notion that persists even after formal education (Clement 1982; Hestenes et al 1992; Thornton and Sokoloff 1998) Additionally, a prevalent misunderstanding is the idea that acceleration cannot occur without velocity (Kim and Pak 2002; Reif and Allen 1992) These misconceptions frequently arise from students' difficulties in differentiating between velocity, acceleration, and force (Reif and Allen 1992; Trowbridge and McDermott 1980), particularly regarding gravitational force, which is often poorly understood.

14 2 Review of Research into Misconceptions … learned at the secondary level, with related misconceptions continuing in higher levels of education (Bar et al.2016; Kavanaugh and Sneider2007).

In addition, many students’conceptions of gravity are closely related to their conceptions of a spherical Earth (Gửnen2008; Nussbaum1979; Sneider and Pulos

In a study by Palmer (2001), interviews with sixth and tenth-grade students revealed that less than 30% correctly identified that all presented objects were influenced by gravity Notably, some students mistakenly believed that objects buried beneath the Earth's surface were not affected by gravitational forces.

Many of these misconceptions have been shown to be stable in the face of conventional physics instruction, preventing students from learning new concepts.

A study by Pablico (2010) explored high school students' misconceptions regarding force and gravity, specifically focusing on their understanding of a ball's motion when thrown upward and falling back down The findings revealed that most students in grades 9–12 incorrectly believed that the net force on the ball always aligned with its direction of motion They failed to grasp that the constant downward force of gravity is what leads to changes in the ball's motion Many students assumed that the force acted upward during the ball's ascent and that it became zero at the peak of its flight.

Students often perceive the force acting on a ball as downward when it descends, but many struggle to justify this reasoning A common misconception is that the force must align with the direction of the ball's motion, leading to confusion about the nature of forces at play.

Research indicates that there are significant gender gaps in students' understanding of physics, with females often beginning courses with lower levels of conceptual comprehension Traditional teaching methods have proven ineffective in closing this gender gap, highlighting the need for more effective instructional strategies (Cavallo et al 2004).

Docktor and Heller2008; Hake2002; Hazari et al.2007; Kost et al.2009).

Misunderstandings in Mathematics

Algebra serves as a foundational element in mathematics, acting as a gateway to higher education and various career opportunities (Kilpatrick and Izsák, 2008) Mastery of algebraic concepts is essential for success in advanced mathematics courses; however, numerous studies indicate that students often face challenges, particularly with linear equations.

Solving linear equations necessitates a blend of conceptual understanding and procedural skills Conceptual knowledge encompasses grasping principles and relationships, while procedural skills focus on executing a series of operations effectively Unlike basic arithmetic, solving linear equations requires more than just memorizing formulas; it involves comprehending the relationships between the quantities involved Therefore, students must develop a profound understanding of these concepts to succeed.

Many students struggle with misconceptions in physics, often relying on procedural knowledge rather than developing a conceptual understanding of equations Research by Kalchman and Koedinger (2005) highlights the importance of distinguishing between independent and dependent variables to grasp the meaning of slope and intercepts in various contexts Similarly, Caglayan and Olive (2010) emphasize that this lack of conceptual insight can hinder students' overall comprehension of physics principles.

Stump (2001) highlighted that high school pre-calculus students, despite formal instruction, often lack a deep conceptual understanding of "slope." Her research revealed that many students could grasp slope in functional contexts but struggled to identify it as a measure of rate of change or steepness Additionally, other studies indicated that students had difficulty distinguishing between additive and multiplicative relationships (Simon and Blume, 1994) and understanding ratio as a measure of slope (Swafford and Langrall, 2000) This gap in conceptual knowledge regarding the relationships between variables has led to widespread misunderstandings of slope and linear equations.

A student's understanding of linear equations is hindered by a lack of conceptual knowledge regarding the relationships between variables This gap affects their ability to interpret and translate the symbolic aspects of linear equations The National Council of Teachers of Mathematics (NCTM) emphasizes that students should be capable of representing and analyzing relationships through various methods, including tables, verbal rules, equations, and graphs.

Many students struggle to graph equations effectively due to a weak understanding of the connection between algebraic equations and their graphical representations (Knuth, 2000) Research indicates that even when a graphical approach could lead to better outcomes, students often hesitate to utilize graphs (Knuth, 2000; Tsamir and Almog, 2001; Dyke and White, 2004) For instance, Knuth (2000) observed that students frequently relied on alternative methods rather than embracing graphical reasoning, even in scenarios designed to promote it Additionally, Huntley et al (2007) found that third-year high school students often required prompting to apply graphical solutions, despite these methods being the most efficient for solving equations.

Students often struggle to graphically model algebraic relationships, making it challenging to convert real-life word problems into suitable algebraic equations (Adu et al 2015; Bishop et al 2008) Without targeted instruction, addressing these algebraic misunderstandings becomes increasingly difficult as students advance in mathematics Research indicates that males tend to make fewer errors than females and exhibit different types of mistakes when solving multi-step linear equations in algebra (Powell 2013).

This report enhances the existing literature on students' misconceptions in physics and mathematics by examining specific misunderstandings related to gravity and linear equations across various grade levels It highlights patterns in these misconceptions based on gender and country, emphasizing the critical need to identify and comprehend these errors to improve educational outcomes.

A review of research into student misconceptions, errors, and misunderstandings reveals the need for curriculum adjustments in secondary education By addressing these issues, we can enhance student learning and better prepare them for post-secondary education and future career opportunities.

Adu, E., Assuah, C K., & Asiedu-Addo, S K (2015) Students ’ errors in solving linear equation word problems: Case study of a Ghanaian senior high school African Journal of Educational

Studies in Mathematics and Sciences, 11, 17 – 30.

Angell, C (2004) Exploring students ’ intuitive ideas based on physics items in TIMSS-1995 In C.

Papanastasiou (Ed.), Proceedings of the IRC-2004 TIMSS IEA International Research

Conference (Vol 2, pp 108 – 123) Published Nicosia, Cyprus: University of Cyprus Retrieved from https://www.iea.nl/sites/default/ fi les/2019-03/IRC2004_Angell.pdf.

Bar, V., Brosh, Y., & Sneider, C (2016) Weight, mass, and gravity: Threshold concepts in learning science Science Educator, 25(1), 22 – 34.

Bishop, A., Filloy, E., & Puig, L (2008) Educational algebra: A theoretical and empirical approach Boston, MA, USA: Springer.

Bradshaw, L., & Templin, J (2014) Combining item response theory and diagnostic classi fi cation models: A psychometric model for scaling Psychometrika, 79(3), 403 – 425.

Caglayan, G., & Olive, J (2010) Eighth grade students ’ representations of linear equations based on a cups and tiles model Educational Studies in Mathematics, 74(2), 143 – 162.

Cavallo, Potter, and Rozman (2004) explore gender differences in learning constructs and how shifts in these constructs relate to course achievement Their study focuses on a yearlong, structured inquiry physics course designed for life science majors, highlighting the impact of gender on educational outcomes in a scientific context The research emphasizes the importance of understanding these dynamics to enhance learning experiences and improve academic performance in college-level physics.

Choi, K M., Lee, Y S., & Park, Y S (2015) What CDM can tell about what students have learned: An analysis of TIMSS eighth grade mathematics Eurasia Journal of Mathematics,

Clement, J (1982) Students ’ preconceptions in introductory mechanics American Journal of

Darling, G (2012) How does force affect motion? Science and Children, 50(2), 50 – 53.

De la Torre, J., & Minchen, N (2014) Cognitively diagnostic assessments and the cognitive diagnosis model framework Psicolog í a Educativa, 20(2), 89 – 97.

Demirci, N (2005) A study about students ’ misconceptions in force and motion concepts by incorporating a web-assisted physics program Turkish Online Journal of Educational

Docktor, J., & Heller, K (2008) Gender differences in both force concept inventory and introductory physics performance AIP Conference Proceedings, 1064, 15 – 18 Retrieved from https://doi.org/10.1063/1.3021243.

Dogan, E., & Tatsuoka, K (2008) An international comparison using a diagnostic testing model:

Turkish students ’ pro fi le of mathematical skills on TIMSS-R Educational Studies in

Dyke, F V., & White, A (2004) Examining students ’ reluctance to use graphs Mathematics

Eryilmaz, A (2002) Effects of conceptual assignments and conceptual change discussions on students ’ misconceptions and achievement regarding force and motion Journal of Research in

Gilmore, C., Keeble, S., Richardson, S., & Cragg, L (2017) The interaction of procedural skill, conceptual understanding and working memory in early mathematics achievement Journal of

G ử nen, S (2008) A study on student teachers ’ misconceptions and scienti fi cally acceptable conceptions about mass and gravity Journal of Science Education and Technology, 17(1), 70 – 81.

The study by Hake (2002) explores the correlation between individual student normalized learning gains in mechanics and various factors such as gender, high school physics background, and pretest scores in mathematics and spatial visualization This research was presented at the Physics Education Research Conference held in Boise, Idaho, emphasizing the importance of understanding how these factors influence learning outcomes in physics education The findings can be accessed for further insights on ResearchGate.

Halloun, I A., & Hestenes, D (1985a) Common sense concepts about motion American Journal of Physics, 53(11), 1056 – 1065.

Halloun, I A., & Hestenes, D (1985b) The initial knowledge state of college physics students. American Journal of Physics, 53(11), 1043 – 1048.

Gender differences significantly impact introductory university physics performance, influenced by high school physics preparation and various affective factors The study by Hazari, Tai, and Sadler (2007) published in *Science Education* highlights these disparities, revealing that students' prior experiences and emotional responses play a crucial role in their academic outcomes in physics For more insights, refer to the full article available at [Wiley Online Library](https://onlinelibrary.wiley.com/doi/abs/10.1002/sce.20223).

Henson, A H., Templin, J L., & Willse, J T (2009) De fi ning a family of cognitive diagnosis models using log-linear models with latent variables Psychometrika, 74(2), 191 – 210. Hestenes, D., Wells, M., & Swackhamer, G (1992) Force concept inventory The Physics Teacher, 30(3), 141 – 158.

Recent studies have explored various aspects of cognitive diagnosis models in educational assessment Hsu et al (2013) examined variable-length computerized adaptive testing, highlighting its effectiveness in measuring cognitive abilities Similarly, Huebner and Wang (2011) compared different methods for classifying examinees within these cognitive frameworks, contributing to the understanding of assessment accuracy Additionally, Huntley et al (2007) investigated the reasoning strategies employed by high school students while solving linear equations, offering insights into mathematical behavior and problem-solving approaches These works collectively enhance the field of educational measurement and cognitive assessment.

The TIMSS item diagnostic report for South Africa, focusing on Grade 5 Numeracy, was authored by Juan et al in 2017 and commissioned by the Department of Basic Education This report, published by the Human Sciences Research Council, provides an in-depth analysis of numeracy skills among fifth-grade students in South Africa For further details, the full report is accessible at the Human Sciences Research Council's eCommons platform.

Junker, B W., & Sijtsma, K (2001) Cognitive assessment models with few assumptions, and connections with nonparametric item response theory Applied Psychological Measurement, 25

Kalchman, M., & Koedinger, K R (2005) Teaching and learning functions In M S Donovan, &

J D Bransford (Eds.), How students learn: History, mathematics, and science in the classroom (pp 351 – 393) Washington, DC: National Academies Press.

Kavanagh, C., & Sneider, C (2007) Learning about gravity I Free fall: A guide for teachers and curriculum developers Astronomy Education Review, 5(21), 21 – 52.

Kilpatrick, J., & Izs á k, A (2008) A history of algebra in the school curriculum Algebra and Algebraic Thinking in School Mathematics, 70, 3 – 18.

Kim, E., & Pak, S J (2002) Students do not overcome conceptual dif fi culties after solving 1000 traditional problems American Journal of Physics, 70(7), 759 – 765.

Knuth, E J (2000) Student understanding of the Cartesian connection: An exploratory study. Journal for Research in Mathematics Education, 31(4), 500 – 508.

Kost, L E., Pollock, S J., & Finkelstein, N D (2009) Unpacking gender differences in students ’ perceived experiences in introductory physics AIP Conference Proceedings, 1179, 177 – 180. Retrieved from https://doi.org/10.1063/1.3266708.

Leighton, J P., & Gierl, M J (2007) De fi ning and evaluating models of cognition used in educational measurement to make inferences about examinees ’ thinking processes. Educational Measurement: Issues and Practice, 26(2), 3 – 16.

Maris, E (1999) Estimating multiple classi fi cation latent class models Psychometrika, 64, 187 – 212.

18 2 Review of Research into Misconceptions …

Mosimege, M., Beku, U., Juan, A., Hannan, S., Prinsloo, C H., Harvey, J C., & Zulu, N (2017).

TIMSS item diagnostic report: South Africa: Grade 9 Mathematics (Commissioned by the

Department of Basic Education) South Africa: Human Sciences Research Council Retrieved from http://repository.hsrc.ac.za/handle/20.500.11910/11448.

NCTM (1989) Curriculum and evaluation standards for school mathematics Reston, VA:

National Council of Teachers of Mathematics.

Nussbaum, J (1979) Children ’ s conceptions of the earth as a cosmic body: A cross age study.

Pablico, J R (2010) Misconceptions on force and gravity among high school students Louisiana

State University Master ’ s theses 2462 Retrieved from https://digitalcommons.lsu.edu/ gradschool_theses/2462/.

Palmer, D (2001) Students ’ alternative conceptions and scienti fi cally acceptable conceptions about gravity International Journal of Science Education, 23(7), 691 – 706.

Piburn, M D., Baker, D R., & Treagust, D F (1988) Misconceptions about gravity held by college students Paper presented at the annual meeting of the National Association for

Research in Science Teaching, Lake of the Ozarks, MO, USA, April 10 – 13, 1988 Retrieved from https://eric.ed.gov/?id92616.

Powell, A N (2013) A study of middle school and college students ’ misconceptions about solving multi-step linear equations Masters thesis, State University of New York at Fredonia, NY,

USA Retrieved from http://hdl.handle.net/1951/58371.

Prinsloo, C H., Harvey, J C., Mosimege, M., Beku, U., Juan, A., Hannan, S., & Zulu, N (2017).

TIMSS item diagnostic report: South Africa: Grade 9 Science (Commissioned by the

Department of Basic Education) South Africa: Human Sciences Research Council Retrieved from http://www.hsrc.ac.za/en/research-data/ktree-doc/19286.

Provasnik, S., Malley, L., Neidorf, T, Arora, A., Stephens, M., Balestreri, K., Perkins, R., & Tang,

J H (2019, in press) U.S performance on the 2015 TIMSS Advanced mathematics and physics assessments: A closer look (NCES 2017-020) Washington, DC: US Department of

Education, National Center for Education Statistics.

Reif, F., & Allen, S (1992) Cognition for interpreting scienti fi c concepts: A study of acceleration.

Roussos, L A., Templin, J L., & Henson, R A (2007) Skills diagnosis using IRT-based latent class models Journal of Educational Measurement, 44(4), 293 – 311.

Saputro, B A., Suryadi, D., Rosjanuardi, R., & Kartasasmita, B G (2018) Analysis of students ’ errors in responding to TIMSS domain algebra problem Journal of Physics: Conference Series, 1088,

012031 Retrieved from https://iopscience.iop.org/article/10.1088/1742-6596/1088/1/012031.

Shear, B R., & Roussos, L A (2017) Validating a distractor-driven geometry test using a generalized diagnostic classi fi cation model In B Zumbo, & A Hubley (Eds.), Understanding and investigating response processes in validation research (Vol 69, pp 277 – 304) Cham,

Simon, M A., & Blume, G W (1994) Mathematical modeling as a component of understanding ratio-as-measure: A study of prospective elementary teachers Journal of Mathematical

Sneider, G., & Pulos, S (1983) Children ’ s cosmographies: Understanding the earth ’ s shape and gravity Science Education, 67(2), 205 – 221.

Stein, M., Larrabee, T G., & Barman, C R (2008) A study of common beliefs and misconceptions in physical science Journal of Elementary Science Education, 20(2), 1 – 11.

Stump, S L (2001) High school precalculus students ’ understanding of slope as measure School

Swafford, J O., & Langrall, C W (2000) Grade 6 students ’ preinstructional use of equations to describe and represent problem situations Journal for Research in Mathematics Education, 31

Tatsuoka, K K (1983) Rule space: An approach for dealing with misconceptions based on item response theory Journal of Educational Measurement, 20(4), 345 – 354.

Thornton, R K., Kuhl, D., Cummings, K., & Marx, J (2009) Comparing the force and motion conceptual evaluation and the force concept inventory Physical Review Special Topics: Physics Education Research, 5(1), 1 – 8.

Thornton, R K., & Sokoloff, D R (1998) Assessing student learning of Newton ’ s laws: The force and motion conceptual evaluation and the evaluation of active learning laboratory and lecture curricula American Journal of Physics, 66(4), 338 – 352.

Trowbridge and McDermott (1980) explored student comprehension of one-dimensional velocity, revealing key insights into their understanding of this fundamental physics concept Similarly, Tsamir and Almog (2001) examined the strategies and challenges faced by students when dealing with algebraic inequalities, highlighting common difficulties in mathematical education Both studies contribute valuable perspectives on the cognitive processes involved in learning physics and mathematics, emphasizing the need for effective teaching methods to address these challenges.

V ă c ă re ţ u, A (n.d.) Using the TIMSS results for improving mathematics learning Cluj-Napoca, Romania: Romanian Reading and Writing for Critical Thinking Association Retrieved from http://directorymathsed.net/montenegro/Vacaretu.pdf.

Yamaguchi, K., & Okada, K (2018) Comparison among cognitive diagnostic models for the TIMSS 2007 fourth grade mathematics assessment PLoS ONE, 13(2), e0188691.

TIMSS and TIMSS Advanced Data

Since 1995, TIMSS and TIMSS Advanced assessments have tracked international trends in mathematics and science achievement, utilizing nationally representative samples of students in participating countries across grades four, eight, and the final year of secondary school for those enrolled in advanced courses.

21 coursework in physics and mathematics) TIMSS has been administered every four years for six assessment cycles 1 (namely in 1995, 1999, 2003, 2007, 2011, and

2015), while TIMSS Advanced has been administered at three points in time (1995,

Since 2008, the IEA has published international reports following each assessment, accompanied by international databases for secondary analyses After each assessment, some assessment items and scoring guides are made available, while at least half are kept secure for future cycles Items from both assessments may be released after one, two, or three assessment cycles.

This report used assessment items and student performance data from the TIMSS and TIMSS Advanced assessments conducted across all assessment cycles from

From 1995 to 2015, the countries participating in the TIMSS and TIMSS Advanced assessments varied with each cycle This article focuses on five countries—Italy, Norway, the Russian Federation, Slovenia, and the United States—that took part in the TIMSS Advanced 2015 assessment and have consistently participated in most TIMSS grade eight and grade four mathematics and science assessments since 1995 These countries were chosen from the nine that participated in TIMSS Advanced 2015 to ensure a comprehensive data set for addressing the research questions Notably, all selected countries participated across all three grade levels.

Table 3.1 Participation of countries in TIMSS Advanced assessments, by cycle

• Indicates participation in that assessment cycle

The TIMSS Advanced 2015 assessment included nine participating countries, with Italy, Norway, the Russian Federation, Slovenia, and the United States being selected for this study For further details on TIMSS Advanced data considerations from the years 1995, 2008, and 2015, please refer to the Appendix.

The data referenced in this article is based on the TIMSS Advanced assessments conducted in 1995, 2008, and 2015, as published by the TIMSS & PIRLS International Study Center at Boston College's Lynch School of Education The source material is copyrighted by the International Association for the Evaluation of Educational Achievement (IEA) and can be found in Mullis et al.'s 2016 publication, specifically in Appendix MA.1 and Appendix PA.1.

1 In 1999, the TIMSS assessment was only administered at grade eight.

The methodology employed to analyze student misconceptions focused on data from the 2015 assessments, ensuring that no more than one assessment cycle was missing for any grade level This selection of five countries allows for extensive comparisons across different countries, grade levels, and assessment cycles.

TIMSS evaluates mathematics and science performance among students in two key grade levels, specifically targeting all fourth-grade and eighth-grade students, or their equivalent in each participating country.

Advanced physics and mathematics populations consist of final-year secondary school students enrolled in or having previously completed TIMSS Advanced-eligible courses in physics or advanced mathematics (Martin et al., 2014) For additional details regarding the TIMSS and TIMSS Advanced populations, please refer to the Appendix.

In 2015, the TIMSS Advanced assessment targeted a select group of final-year students, representing approximately one-quarter or less of this demographic across most participating countries The study revealed that the coverage index, which measures the percentage of the relevant age cohort included in the TIMSS Advanced physics and advanced mathematics populations, was consistently lower for physics compared to advanced mathematics in all five countries analyzed Specifically, the coverage index for physics in 2015 varied, with the Russian Federation reporting around 5% coverage.

Federation and the United States, to 18% in Italy (Table3.4) There were some

Table 3.2 Participation of countries in TIMSS grade eight assessments, by cycle

• Indicates participation in that assessment cycle

∘ Indicates participation but data not comparable for measuring trends to 2015

Notes This table includes the nine countries that participated in the TIMSS Advanced 2015 assessment Five countries were selected for inclusion in this study (Italy, Norway, the Russian

Federation, Slovenia, and the United States) See Appendix for TIMSS grade eight data considerations for 1995, 1999, and 2015

Source TIMSS 1995, 1999, 2003, 2007, 2011, and 2015 assessments Copyright International

Association for the Evaluation of Educational Achievement (IEA) Publisher: TIMSS & PIRLS

International Study Center, Lynch School of Education, Boston College Retrieved from Mullis et al (2016b, Appendix A.1)

2 TIMSS Advanced-eligible courses are de fi ned as those that cover most of the topics outlined in the TIMSS Advanced physics and mathematics assessment frameworks.

Between the assessment years of 1995, 2008, and 2015, significant differences were observed in the physics coverage index of TIMSS and TIMSS Advanced Data Notably, Italy, the Russian Federation, and the United States experienced increases in the percentage of students studying physics at an advanced level, with Italy rising from 4% in 2008 to 18% in 2015 Conversely, Slovenia saw a decline from 39% in 1995 to 7-8% in both 2008 and 2015, indicating a more limited student sample In terms of advanced mathematics, the coverage index in 2015 varied, with the Russian Federation and the United States at 10-11% and Slovenia at 34%.

The TIMSS Advanced populations reveal significant gender disparities in advanced coursework, particularly in physics, where boys consistently outnumber girls across five countries In Norway and Slovenia, only around 30% of advanced physics students are female, while this figure rises to about 40% in Italy, the Russian Federation, and the United States Notably, the representation of females in physics has remained relatively stable over the assessment years Conversely, in advanced mathematics, the percentage of female students is lower than that of males in Italy and Norway, approximately 40%, yet it surpasses male participation in Slovenia.

Table 3.3 Participation of countries in TIMSS grade four assessments, by cycle

• Indicates participation in that assessment cycle

∘ Indicates participation but data not comparable for measuring trends to 2015

The TIMSS Advanced 2015 assessment included nine participating countries, with five—Italy, Norway, the Russian Federation, Slovenia, and the United States—selected for this study For additional insights, refer to the Appendix, which outlines TIMSS grade four data considerations from 1995 and 2015, noting that TIMSS was not conducted at grade four in 1999.

The TIMSS assessments conducted in 1995, 2003, 2007, 2011, and 2015, published by the International Association for the Evaluation of Educational Achievement (IEA) through the TIMSS & PIRLS International Study Center at Boston College, provide valuable insights into educational performance For detailed findings, refer to Mullis et al (2016b, Appendix A.1).

24 3 Methodology Used to Analyze Student Misconceptions …

The TIMSS Advanced assessments from 1995, 2008, and 2015 reveal significant variations in the coverage index and gender distribution among physics students across different countries In Italy, the coverage index was 18.2% in 2015, with female students comprising 46% and male students 54% Norway showed a lower coverage index of 6.5% in 2015, with a balanced gender ratio of 29% female and 71% male The Russian Federation had a coverage index of 4.9%, with female students at 42% and male students at 58% Slovenia's coverage index was 7.6%, with 30% female and 70% male representation The United States had a coverage index of 4.8% in 2015, with 39% of students being female and 61% male These findings underscore the disparities in student participation and gender representation in advanced physics courses globally.

3.1 TIMSS and TIMSS Advanced Data 25

(about 60%), and about equal to males in the Russian Federation and the United States.

This report presents findings derived from item-level statistics obtained from the TIMSS and TIMSS Advanced international databases across various assessment cycles It includes the weighted percentage of correct responses for each participating country and the distribution of student responses across different categories Item-level statistics were analyzed for each country and averaged across the five countries involved in the study, with a breakdown by gender The report features example items to illustrate these findings.

Methodology

Assessment Framework Review and Content Mapping

To determine how mathematics and science concepts progress from the lower grades in TIMSS to TIMSS Advanced, topics covered in the 2015 TIMSS

The TIMSS 2015 frameworks for grades four and eight were aligned with advanced assessment frameworks, highlighting significant content overlap in mechanics (forces and motion) within physics and algebra in mathematics This overlap allows for a sufficient number of assessment items across these grades to analyze patterns of misconceptions effectively Each grade level also had specific framework objectives that guided the selection of items for the study.

As described in Chap 1, this study focuses on two specific topics: gravity in physics and linear equations in algebra We determined the set of TIMSS 2015 and

TIMSS Advanced 2015 framework objectives that measured these topics (or pre- cursor topics) across grade levels for gravity (Table1.1) and linear equations

(Table1.2) Since the TIMSS and TIMSS Advanced frameworks have been revised over the past 20 years, content mapping also included mapping the TIMSS framework objectives in 1995, 1999, 2003, 2007, and 2011, and the TIMSS

Advanced framework objectives in 1995 and 2008, to the corresponding TIMSS

Evaluation of Item-Level Performance Data

The TIMSS and TIMSS Advanced frameworks outline specific objectives for understanding gravity and linear equations, leading to the creation of item sets for each subject This includes 16 items focused on physics and 28 items dedicated to mathematics, covering content for both grade four and grade eight levels.

Advanced assessments were assembled and reviewed First, the TIMSS Advanced

In a comprehensive evaluation of 2015 assessment items, the study analyzed performance objectives and identified specific misconceptions, errors, and misunderstandings exhibited by students in five TIMSS Advanced countries: Italy, Norway, the Russian Federation, Slovenia, and the United States Additionally, TIMSS items from various assessment cycles for grades four and eight were examined to uncover evidence of related misconceptions and misunderstandings at the lower grade levels.

The analysis of item-level performance data revealed evidence of misconceptions, errors, and misunderstandings among students For multiple-choice (MC) items, this was achieved through distractor analysis, which identifies common errors based on the incorrect options selected by students In the case of constructed-response (CR) items, response patterns were evaluated according to the scoring guides that outline the nature of student answers The TIMSS and TIMSS Advanced assessments utilize scoring guides that establish criteria for categorizing responses as correct, partial, or incorrect, employing two-digit diagnostic codes to identify specific misconceptions or errors This initial evaluation relied on item statistics, including the weighted percentage distributions of student responses across countries, sourced from the TIMSS & PIRLS International Study Center's international data almanacs.

The content analysis of items related to gravity and linear equations across grade levels revealed a set of performance objectives, comprising four in physics and nine in mathematics, specifically measured by these items These objectives, derived from TIMSS and TIMSS Advanced items, are more detailed than the broader framework objectives presented in Chapter 1 While some objectives were assessed at a single grade level, others were evaluated at two levels, such as TIMSS Advanced/grade eight or grade eight/grade four, and physics objectives were measured across all three grade levels Additionally, we identified various misconceptions, errors, and misunderstandings reflected in incorrect student responses, with each type of misconception being assessed by one to six items.

4 Additional TIMSS Advanced items from 1995 and 2008 were also evaluated for physics. Mathematics only included items from TIMSS Advanced 2015.

The TIMSS testing schedule allows for the assessment of the same group of students over several years, tracking their progress from grade four in 2007 to grade eight in 2011 and grade twelve in 2015 However, this report does not provide direct insights into changes in specific misconceptions or errors within this cohort due to limitations in the available item-level data These constraints highlight important considerations and implications for future research in this field.

28 3 Methodology Used to Analyze Student Misconceptions … error, and misunderstanding (See Sect.1.2for detailed definitions of the terms, and

Chap 4 for an overview of performance objectives, misconceptions, errors, and misunderstandings, and the set of items used in the study.)

Reporting Patterns in Percent Correct and Percent with Misconceptions, Errors, and Misunderstandings by Grade, Country, Gender, and Assessment Year

with Misconceptions, Errors, and Misunderstandings by Grade, Country, Gender, and Assessment Year

All of the analyses used to report on the percent correct and percentage of students with misconceptions, errors, and misunderstandings were conducted using the

IEA’s International Database (IDB) Analyzer (Version 4.0) Percentages function

(IEA 2018) The IDB Analyzer uses a jackknife repeated replication

The JRR procedure is utilized to compute estimates and standard errors for various statistics, including average scores and percent correct, as detailed in the Appendix for additional technical information However, standard errors are not included in the tables and figures presented in this book.

(supplementary materials providing standard errors for all estimates are available for download atwww.iea.nl/publications/RfEVol9).

Four types of analyses were used to produce the item-level statistics shown in the report.

This is the percentage of students receiving credit on each item For MC and short

CR items, each valued at one score point, indicate the percentage of students who answered correctly In the case of extended CR items, the results show the weighted percentage of students earning full credit (two points) or partial credit (one point).

In an analysis of student performance, when 10% of students achieved full credit and another 10% received partial credit on an item, the weighted percent correct was calculated to be 15% This figure combines the percentage of students earning full credit (10%) with half of those receiving partial credit (5%) Percent correct was determined for all items across each country.

(overall and by gender) When reporting percent correct on the set of items in physics and mathematics, data from the most recent assessment was used for each item.

Percentage of Students with Misconceptions, Errors, and Misunderstandings

Two different types of item-level analyses were used to determine these percentages:

The study identifies specific misconceptions and misunderstandings through particular response options in multiple-choice (MC) and constructed response (CR) items It calculates the percentage of students exhibiting these misconceptions in physics and misunderstandings in mathematics by aggregating the percentages from relevant options or scoring categories This analysis focuses on 11 items in physics, consisting of 10 MC questions and one CR item, as well as three items in mathematics.

3.2 Methodology 29 mathematics (all MC) For two of the MC items in physics, one response option measured one type of misconception and others measured a second type; two separate analyses were conducted to obtain the percentages for both types of misconceptions.

General types of misunderstandings were identified in items where no specific misconceptions or errors were tracked The assessment focused on whether students demonstrated the necessary understanding or skills for the performance objectives The percentage of students displaying a general misunderstanding was calculated based on those who answered incorrectly, including invalid responses and off-task comments, as well as those who omitted the item entirely This analysis applies to six physics items (one multiple-choice and five constructed response) and 26 mathematics items (12 multiple-choice).

In the TIMSS assessment, the majority of items were constructed response questions requiring students to explain their answers or show their work According to TIMSS scoring guides, incorrect responses are categorized under code 79, which includes crossed-out answers, stray marks, and off-task comments When analyzing omitted responses, we assumed that students who did not answer the question lacked the necessary understanding, similar to those providing illegible or irrelevant responses This approach aligns with TIMSS scale scoring, where omitted answers are treated as incorrect, ensuring an accurate representation of students' conceptual understanding Excluding these omitted responses would lead to an underestimation of the percentage of students lacking comprehension.

Appendix Tables A.1 and A.2 in the TIMSS datafiles detail the codes for various misconceptions, errors, and misunderstandings related to physics and mathematics items The analysis includes the percentage of students exhibiting these misconceptions across different countries, categorized by overall performance and gender Additionally, for trend items assessed in multiple years, the percentage of students for each assessment year is provided.

Average Percent Correct and Average Percent with Misconceptions, Errors, and Misunderstandings

The averages presented indicate the percentage of correct responses, as well as the prevalence of misconceptions, errors, or misunderstandings, across various countries for each item Typically, these figures represent the combined average from all five countries that provided data on the specific item.

The "percent omitted" excludes the "not reached" responses, which are considered missing and not factored into the calculations for percent correct or percent with misconceptions.

7 The separate analyses for the physics items that measured two different types of misconceptions were identi fi ed by two different versions (V1 and V2; see Table A.1).

30 3 Methodology Used to Analyze Student Misconceptions …

However, there were some assessment years where data were not available for all countries, and the averages were based only on three or four countries.

Statistical Comparisons

The study analyzed differences in correct response rates and the prevalence of misconceptions, errors, and misunderstandings across five countries, comparing (1) each country to the overall average, (2) female and male students within each country, and (3) trends over different assessment years Appropriate t-tests were employed for all item-level comparisons, with statistical significance indicated in the accompanying data tables and figures A difference was deemed "significant" if the p-value from the t-test was below 0.05, ensuring a 95% probability that the observed differences were genuine rather than coincidental.

When comparing percentages across five countries, it's important to note that there is overlap among the samples, as each country contributes to the overall average To address this overlap, a part-whole test was employed to ensure accurate analysis of the data.

The estimated average percentage for the five countries is denoted as est i, while est j represents the estimated percentage for an individual country The standard errors for these estimates are indicated by se i and se j, respectively Additionally, p signifies the proportion of the total represented by each country, which is set at 0.2.

When analyzing within-country gender differences, researchers can utilize two types of t-tests based on the student samples: independent t-tests, applicable when random samples of female and male students are drawn independently from the population, and non-independent t-tests, used when the samples are not independent.

For independent random samples, the independentt-test is appropriate: tẳ estfemaleestmale

The independent t-test can be computed using the formula \( \frac{(est\_female - est\_male)}{\sqrt{(se\_female^2 + se\_male^2)}} \), where \( est\_female \) and \( est\_male \) represent the estimated percentages of females and males, respectively, while \( se\_female \) and \( se\_male \) denote the standard errors associated with these estimates.

3.2 Methodology 31 from the IDB Analyzer, where the JRR procedure is used to determine the separate percentages and standard errors for females and males.

In the TIMSS and TIMSS Advanced assessments, the samples of female and male students are not independent, as they are drawn from the same schools and classrooms Consequently, the appropriate statistical analysis involves using a t-test for non-independent samples, which necessitates calculating the standard error of the difference between the percentages of female and male students.

The standard error of the difference, se(est female −est male ), takes into account the covariance (cov) between females and males for dependent samples: se est female est male ð ị ẳ

ffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffiffi se 2 est female ð ị ỵse 2 est ð male ị2cov estfemale;estmale r

To achieve accurate standard errors for the female-male percentage difference, the JRR procedure must be applied The IDB Analyzer Version 4.0 utilizes this procedure to calculate standard errors for both female and male percentages; however, it does not support jackknifing gender differences for item-level statistics, such as percent correct or misconceptions Consequently, the standard errors and t-tests generated by the IDB Analyzer serve as approximations that overlook the covariance between genders These approximations are generally reliable when covariances are minimal compared to the standard errors of female and male percentages, which is typically the case in TIMSS design, where only a limited number of students—around four per school or class—respond to each item.

To analyze gender differences in covariances, we utilized the EdSurveyR Package 8 (NCES 2018) Gap function, which employs the JRR technique to assess the percentage differences between females and males The results provide both the standard error of the difference and the covariance, highlighting significant findings in the tested cases.

EdSurvey is an R statistical package created by the American Institutes for Research (AIR) and commissioned by the National Center for Education Statistics (NCES) to facilitate the processing and analysis of large-scale education data The latest version, EdSurvey 2.0.3, is specifically designed for analyzing national and international education datasets, including TIMSS and TIMSS Advanced For additional details, visit the NCES research center software page at https://nces.ed.gov/nationsreportcard/researchcenter/software.aspx.

In our analysis of student misconceptions, we found that the item-level statistics were minimal We conducted analyses using both the IDB Analyzer and the EdSurvey R Package to compare standard errors and t-tests for gender differences Our findings revealed that the standard errors from both methods were nearly identical, differing by only 0.0001%, and the significance of the reported differences remained unchanged Consequently, we opted to utilize the IDB Analyzer's output for gender differences and applied approximate independent t-tests for all items Further details on both software packages and example outputs can be found in the Appendix.

(3) The differences between years for trend items are based on independent sam- ples Thus, the standard independentt-test was used: tẳ estyear1estyear2

To compare the percentage of students across two assessment years, we use the formula \( \frac{(se_{year1})^2 + (se_{year2})^2}{(year1 - year2)^2} \) Here, \( year1 \) and \( year2 \) represent the estimated percentages of students for each assessment year, while \( se_{year1} \) and \( se_{year2} \) denote the respective standard errors This approach allows for a statistical comparison of student performance over time.

Addressing the Research Questions

Physics Results

Mathematics Results

Summary of Results Across Physics and Mathematics