Chapter 15 - Measurement in research. The main contents of the chapter consist of the following: Essential characteristics of data, establishing data characteristics, determining objectivity, determining reliability, types of relative reliability,...
Chapter 15 Measurement in Research Essential Characteristics of Data • Collected data must have certain characteristics in order for them to be considered acceptable evidence upon which to base conclusions – Objectivity – Reliability – Validity • Establish before collecting research data Objectivity • The degree to which multiple scorers agree on • • the values of collected measures or scores a.k.a rater reliability Important because data should not be dependent on who collected it Reliability • Consistency of test scores • Scores should reflect a true indication of one’s • ability and not change over a short period of time Reliability of test scores is a necessary condition for validity Validity • The degree to which interpretations of test • • scores or measures derived from a measuring instrument lead to correct interpretations Most important characteristic of data Enables researcher to have confidence that conclusions are based on correct information Establishing Data Characteristics • Before collecting research data • Documented by other researchers • Pilot study – situation specific documentation – Uniqueness of setting and participation characteristics – Sample size does not need to be large • Technique generally involves calculating a correlation coefficient Determining Objectivity • Two or more scorers or raters should • independently score a group of people on the test Intraclass correlation coefficient (R) is calculated as an estimate of the objectivity coefficient – ANOVA technique – Select appropriate computational formula for R Determining Reliability • To determine reliability, each person must have at least two repeated scores, such as. . – – – – Scores on multiple trials of a test within a day Scores from multiple administrations on different days Scores from multiple items on a test Scores on two different forms of a test Types of Relative Reliability • Internal consistency reliability – Consistency of scores within a day • Stability reliability – Consistency of scores across days Statistical Techniques • Intraclass correlation coefficient (R) – ANOVA technique – Select appropriate computational formula for R • Cronbach’s Alpha • Kappa coefficient – Technique of determining reliability of dichotomously scored tests Determining Validity • Several types of validity evidence – – – – Logical evidence Criterion evidence Construct evidence Other evidence Logical Evidence • Based on the knowledge that a test measures • an attribute and yields scores that can be validly interpreted Insufficient as the only type of validity evidence – Combine with other forms of evidence • Subjective judgment – Jury of experts Criterion Evidence • Based on a correlation coefficient between • • scores on a test and scores on a criterion measure or standard Traditional type of validity evidence Selection of suitable criterion measure often challenging Construct Evidence • A construct is something known to exist although • it may not be precisely defined or measured Experts maintain that all validity evidence is actually construct validity evidence – Evidence that scores are interpreted correctly • Group differences approach Other Measurement Issues • • • • • • • Economy of measures Ease of administration Preparation of participants Scoring directions Characteristics of participants Experience of tester Other ... conclusions are based on correct information Establishing Data Characteristics • Before collecting? ?research? ?data • Documented by other researchers • Pilot study – situation specific documentation – Uniqueness of setting? ?and? ?participation characteristics... Technique generally involves calculating a correlation coefficient Determining Objectivity • Two or more scorers or raters should • independently score a group of people on the test Intraclass correlation coefficient (R) is calculated ... The degree to which interpretations of test • • scores or measures derived from a measuring instrument lead to correct interpretations Most important characteristic of data Enables researcher to have confidence that