Assessment in counseling chapter 3

Reliability Chapter Classical Test Theory ∗ Every observed score is a combination of true score and error Obs = T + E ∗ Reliability = s 1− s E O s = s T O Reliability ∗ Systematic versus unsystematic error ∗ Reliability only takes unsystematic error into account Reliability & Correlation ∗ Reliability often based on consistency between two sets of scores ∗ Correlation: Statistical technique used to examine consistency Positive Correlation Negative Correlation Pearson-Product Moment Correlation Coefficient ∗ Correlation coefficient: a numerical indicator of the relationship between two sets of data ∗ Pearson-Product Moment correlation coefficient is most common r= ∑ z1 z N Coefficient of Determination ∗ The percentage of shared variance between two sets of data Types of Reliability ∗ Test-Retest ∗ Alternate/Parallel Forms ∗ Internal Consistency Measures Test-Retest ∗ Correlating performance on first administration with performance on the second ∗ Co-efficient of stability Alternate/Parallel Forms ∗ Two forms of instrument, administered to same individuals Internal Consistency Measures ∗ Split-half reliability ∗ Spearman-Brown formula ∗ Kuder-Richardson formulas ∗ KR 20 ∗ KR 21 ∗ Coefficient Alpha Nontypical Situations ∗ Typical methods for determining reliability may not be suitable for: ∗ Speed tests ∗ Criterion-referenced tests ∗ Subjectively-scored instruments ∗ Interrater reliability Evaluating Reliability Coefficients ∗ Examine purpose for using instrument ∗ Be knowledgeable about reliability coefficients of other instruments in that area ∗ Examine characteristics of particular clients against reliability coefficients ∗ Coefficients may vary based on SES, age, culture/ethnicity, etc Standard Error of Measurement SEM = s − r ∗ Provides estimate of range of scores if someone were to take instrument repeatedly ∗ Based on premise that when individuals take a test multiple times, scores fall into normal distribution SEM: Example ∗ Sam’s SAT Verbal = 550 ∗ r = 91; s = 100 ∗ SEM = 100 − 91 = 100 09 = 100 × = 30 ∗ 68% of the time, Sam’s true score would fall between 520 and 580 ∗ 95% of the time, Sam’s true score would fall between 490 and 610 ∗ 99.5% of the time, Sam’s true score would fall between 460 and 640 Determining Range of Scores Using SEM Standard Error of Difference ∗ Method to determine if difference between two scores is significant ∗ Takes into account SEM of both scores Alternative Theoretical Model ∗ Generalizability or Domain Sampling Theory ∗ Focus is on estimating the extent to which specific sources of variation under defined conditions are contributing to the score on the instrument [...]... forms of instrument, administered to same individuals Internal Consistency Measures ∗ Split-half reliability ∗ Spearman-Brown formula ∗ Kuder-Richardson formulas ∗ KR 20 ∗ KR 21 ∗ Coefficient Alpha Nontypical Situations ∗ Typical methods for determining reliability may not be suitable for: ∗ Speed tests ∗ Criterion-referenced tests ∗ Subjectively-scored instruments ∗ Interrater reliability Evaluating Reliability... Determining Range of Scores Using SEM Standard Error of Difference ∗ Method to determine if difference between two scores is significant ∗ Takes into account SEM of both scores Alternative Theoretical Model ∗ Generalizability or Domain Sampling Theory ∗ Focus is on estimating the extent to which specific sources of variation under defined conditions are contributing to the score on the instrument ... Coefficients ∗ Examine purpose for using instrument ∗ Be knowledgeable about reliability coefficients of other instruments in that area ∗ Examine characteristics of particular clients against reliability coefficients ∗ Coefficients may vary based on SES, age, culture/ethnicity, etc Standard Error of Measurement SEM = s 1 − r ∗ Provides estimate of range of scores if someone were to take instrument repeatedly... that when individuals take a test multiple times, scores fall into normal distribution SEM: Example ∗ Sam’s SAT Verbal = 550 ∗ r = 91; s = 100 ∗ SEM = 100 1 − 91 = 100 09 = 100 × 3 = 30 ∗ 68% of the time, Sam’s true score would fall between 520 and 580 ∗ 95% of the time, Sam’s true score would fall between 490 and 610 ∗ 99.5% of the time, Sam’s true score would fall between 460 and 640 Determining Range ... determining reliability may not be suitable for: ∗ Speed tests ∗ Criterion-referenced tests ∗ Subjectively-scored instruments ∗ Interrater reliability Evaluating Reliability Coefficients ∗ Examine... Examine purpose for using instrument ∗ Be knowledgeable about reliability coefficients of other instruments in that area ∗ Examine characteristics of particular clients against reliability coefficients... Generalizability or Domain Sampling Theory ∗ Focus is on estimating the extent to which specific sources of variation under defined conditions are contributing to the score on the instrument