Summary of Research on Various Teacher Employment Screening Instruments 2

Summary of Research on Various Teacher Employment Screening Instruments Howard Ebmeier University of Kansas howard@ku.edu 8/18/2008 Citation Sample Independent measure Dependent Measure Correlation /Effect Size Reliability Comments AARESTAD, JANNA SERINA (1980) ANALYSIS OF TEACHER SELECTION/RETENTIONAND STUDENT ACHIEVEMENT AS RELATED TO SELECTION RESEARCH INCORPORATED'STEACHER PERCEIVER INTERVIEW Ph.D dissertation, University of Minnesota ARAMBURO, DOROTHY JAMES (1981) THE EFFICACY OF THE STRUCTURED INTERVIEWIN THE SELECTION OF SPECIAL EDUCATION TEACHERS Ed.D., dissertation, University of New Orleans N=34 grades 1-6 TPI Gain score in math achievement No relationship Dissertation not available, thus information derived from abstract TPI scores dropped across grade levels N=179 special education teachers TPI Multiple Teacher Factor Survey Dissertation not available, thus information derived from abstract Not sure what the MTFS measured but apparently completed by the teacher Antoline, Carlos Mitchell (2000) A validation study of the Teacher Perceiver Interview Ph.D dissertation, Texas Woman's University N=349 employed by district teachers Teachers group into categories via TPI (not continuous variable as hoped) Principal ratings Ball, Jane Rowlett 1992) A validation study of students' and parents' perceptions of the talent of teachers selected using the SRI Teacher Perceiver interview Ed.D dissertation, Kansas State University N=11 One school TPI Parent, student ratings 0.29 via canonical correlations among 12 TPI themes and factors of MTFS Significant difference but very small F value Only top vs bottom teachers were significantly different in post hoc tests None reported Cited in study is Gallup 0.70 reliability estimate for TPI Student evaluations were not different Very small qualitative study Concludes that teachers selected with TPI were liked by parents and students —same could probably be said of all teachers however they were selected Citation Sample Independent measure Dependent Measure Correlation /Effect Size Baskin, Maryan Kay (1994) The predictability of the Urban Teacher Selection Interview to the classroom success of elementary student teachers Ed.D dissertation, The University of Memphis N=68 student teachers Star Teacher by Haberman Practice Teaching Portfolio Bingham, Patrick Jerome (2000) The concurrent and predictive validity of elementary school teacher preemployment success indicators Ph.D dissertation, The University of Texas at Austin N=138 teachers TPI Achievement on TAAS (Texas assessment) Principal Rating The regression procedure found the Pass/Fail element of the Urban Teacher Selection Interview predicted success on the Communicatio n aspect of the evaluation instrument 0.04 (reading) 0.03 (math) 0.07 (writing) -0.05 (principal rating) Brown, Karl Kevin (2004) The impact of the Gallup Teacher Perceiver Interview on hiring teachers as perceived by selected administrators in the Alamo Heights Independent School District, San Antonio, Texas Ed.D dissertation, Texas A&M University Three groups Recommended by TPI (201), not recommended but hired (105), pre-TPI (602) N=59 all grade levels TPI Retention rates Administrator perceptions TPI Panel of experts scored teacher evaluations of principals Buresh, Richard John (2003) The predictive validity of the Teacher Perceiver Interview in selecting effective elementary teachers in amid-sized Midwestern school district Ed.D dissertation, The University of North Dakota Reliability Comments “Overall, however, the UTSI did not differentiate student teachers who would be evaluated as successful in other areas of teaching which were part of the Practice Teaching Portfolio areas like planning, classroom management, and using effective teaching strategies Findings from this study indicate that Rank on the UTSI, which is emphasized by Haberman as the basis for selecting students, was not a significant predictor of the classroom success of student teachers.” Very small correlation that was significant but only because of large N-student (N=6600) unit of analysis ANOVA across groups was significant with difference due to not recommended different than pre and recommended Retention rates of TPI recommended teachers were higher 0.039 for total Also reports each scale Contains good review of TPI previous research Reposts negative correlation with leave days Citation Sample Independent measure Dependent Measure Correlation /Effect Size CHALKER, DONALD MERLE (1981) THE TEACHER PERCEIVER INTERVIEW AS ANINSTRUMENT FOR PREDICTING SUCCESSFUL TEACHING BEHAVIOR Ed.D dissertation, Wayne State University N=40 HS urban teachers in districts TPI Quartile rankings by building principals Ratings by sample of students Dissertation not available, thus information derived from abstract Correlation numbers absent from abstract COOK, RONALD EUGENE (1981) CHIPPEWA VALLEY LOCAL VALIDATION STUDY OF THETEACHER PERCEIVER INTERVIEW PROCESS Ed.D dissertation, Wayne State University N=30 middle school teachers High and low scoring scores were used for t-test TPI Administrator ratings, student ratings CORNINE, LARRY LEE (1980) A VALIDATION STUDY OF THE TEACHER PERCEIVER INTERVIEW USING THE ILLINOIS TEACHER EVALUATION QUESTIONNAIRE Ed.D., dissertation, University of Kansas N=All Ottawa, KS secondary teachers Experimental group=secondary teachers hired after TPI Control=teachers before TPI N=587 teachers TPI Student ratings of teacher competence Significant correlations between admin ratings and TPI and student ratings and TPI Unclear, author concludes no significance but then states there was a positive t test and correlations No relationship UTPI Praxis Principal ratings 0.144 (statistically significant but large N) Praxis not significant Restricted range problems since they were all employed district teachers Chesley, Daryl (2004) The validity of teacher employment success predictors Ed.D dissertation, The George Washington University Reliability Comments Dissertation not available, thus information derived from abstract Dissertation not available, thus information derived from abstract Citation Sample Independent measure Dependent Measure Correlation /Effect Size Davis, David N=12 elementary teachers Ventures Tenn value added composite achievement scores Delli, Dane Armand (2000) The predictive validity of the Teacher Perceiver Interview on the teaching performance of classroom teachers Ph.D dissertation, The Ohio State University N=72 teachers N=124 TPI Principal ratings Absence Diemer, Janaan S (1998) Identifying effective teachers of behavior disordered students: Is the Gallup Teacher Perceiver Interview an answer? Ed.D dissertation, New Mexico State University Dunning, Christopher Michael (2005) Teacher Perceiver interview tool: A contemporary validation study within a Florida district Ed.D dissertation, University of South Florida N=33 teachers (special ed) Identified as top/bottom quadrant of TPI N=40 teachers 20 NBC certified 20 not 786 Respondents for factor analysis from college interviews TPI Supervisor Ratings Reading was positive but rest were negative (rather strange since reading and math scores were positively correlated in study) 0.37 if all themes are added together in one regression equation R**2 0.14 0.00 TPI 0.31 for NBC/nonNBC teachers (restricted range problem likely) Reliability Comments “there is little association between Ventures for Excellence scores and TVA scores” Sample size too small to conclude much of anything Little or none internal reliability in the various themes Capture in article with Young Good review of the development and research behind TPI Failed to composite score correlation Teachers deemed ineffective had higher average scores on the total and subscales that those deemed effective by TPI Cronback alpha ranged from 0.0-0.30 on the 10 scales Factor analysis of first 24 questions on TPI (three factors suggested: interpersonal (0.35), intrapersonal (0.20), extrapersonal (0.45) Reliability of total instrument (0.63) Three factor model had good confirmatory factor fit (0.90) but one factor model was slightly better 95% of board certified teachers had TPI highly recommend Expert panel supported 22/24 questions as important Citation Sample Independent measure Dependent Measure Correlation /Effect Size RACHEL DAVIS (1984) THE RELATIONSHIP BETWEEN THE TEACHERPERCEIVER INTERVIEW AND INSTRUCTIONAL BEHAVIORS OF TEACHERS OF LEARNINGDISABLED STUDENTS Ed.D dissertation, University of Virginia LD teachers N=? TPI The Classroom Planning and Management Assessment (CPMA), based upon research of effective teaching Frey, Patricia (2003) Ability of the Urban Teacher Selection Interview to identify teachers who are likely to be retained in the Buffalo Public Schools Ed.D dissertation, Seton Hall University, College of Education and Human Services N=876 teachers at all grade levels Star Teacher Retention Gillies, Terry Kim (1988) The relationship between selection variables and subsequent performance ratings for teachers in an Oregon school district Ph.D dissertation, University of Oregon N=196 first year teachers employed by one district TPI Principal ratings Howard, Mildred Anthony (1998) The relationships among teacher selection predictors and teacher performance with principals' opinions about a teacher selection process Ph.D dissertation, Georgia State University N=143 teachers (relatively new) 30 elementary principals TPI Principal evaluation of each teacher via a form reflecting values of TPI A nonsignificant correlation between the TPI total score and the CPMA total mean score 0.13 correlation (low but significant given the large N) Average of 0.17 (great variation over the years of data collection —one year was actually negative) 0.47 for all scales entered collectively Reliability Comments From abstract Some positive and significant correlations reported among various scales on the TPI and the dependent measure but overall no effect Star Teacher instrument used in initial employment decision thus possible range restriction issues Overall correlation was significant but pretty low Range restriction effect likely since were employed by district after results of TPI known Individual subscales correlated from 0.03-0.38 Regression was run for of the theme variables but not total Citation Sample Independent measure Dependent Measure Correlation /Effect Size JOHANNSEN, KENNETH LUDWIG (1981) THE RELATIONSHIP OF A UNIQUE TEACHERSELECTION METHOD TO STUDENT ACHIEVEMENT Ed.D dissertation, Peabody College for Teachers of Vanderbilt University N=40 20 teacher who interviewed using TPI and 20 who were previous hires and had not used the TPI Teachers were balanced in terms of background TPI Math gain score t-test indicated no difference between groups JONES, DAVID EDWARD, JR (1978) PREDICTING TEACHING PROCESSES WITH THETEACHER PERCEIVER INTERVIEW Educat D dissertation, Virginia Polytechnic Institute and State University Kanipe, Mary Lou Smith (1996) A study of the predictive validity of the Urban Teacher Perceiver Interview in the Knox County School System Ed.D dissertation, The University of Tennessee Koerner, Robert Jacob (2007) The relationship between the Teacher Insight score and student performance as measured by the Texas Growth Index Ed.D dissertation, University of North Texas Kreifels, Robert G (2003) A study of selected teacher interview characteristics and student achievement Ed.D dissertation, Saint Louis University Reliability Comments Dissertation not available, thus information derived from abstract Dissertation not available from KU library N=233 teachers—all grade levels 68 Principals Urban TPI Principal Evaluations N=123 3-11 grade 4500 students Teacher Insight TGI (Texas Growth Index) N=260 elementary teachers (N=17 at 4/5 grade ?) TPI (reported subscales) 5th reading 4th math 0.10 total subscales ranged from 0.0-0.18 0.045 English and reading 0.042 Math Correlation with minority teachers (0.39) indicated definite bias May be Range restriction effectsince all teachers were screened with TPI prior to employment Ceiling effects are present since only teachers hired are used Math—no significant relationships with correlations ranging from 0.20 to –0.03 Reading 0.43 to 0.25 with no significant results The sample size reported for the correlations was 17, which I cannot figure out No relationship between ITBS and TPI found Citation Sample Independent measure Dependent Measure LASHER, HUDSON FRED (1976) THE TEACHER PERCEIVER INTERVIEW AS ANINSTRUMENT FOR IDENTIFYING CHARACTERISTICS OF PROSPECTIVETEACHERS Ed.D dissertation, University of Wyoming LEAHY, RICHARD EDGAR (1984) TEACHER CHARACTERISTICS AND STUDENTACHIEVEMENT: AN EXPLORATION OF THE EMPLOYMENT INTERVIEW Ph.D dissertation, University of Minnesota Loyd, David Eugene (2004) The effect of a screening instrument to determine teacher effectiveness as measured by academic gain scores Ed.D dissertation, The University of Tennessee Students in education program TPI N=242 teachers in 38 grades Urban TPI Tenn Value Added achievement in math, reading, language MAUSER, PHILLIP J (1986) THE PREDICTIVE VALIDITY OF THE TEACHERPERCEIVER INTERVIEW TO TEACHER EFFECTIVENESS Ed.D dissertation, Oklahoma State University MILLS, JONATHAN KENDALL (1986) THE TEACHER PERCEIVER INSTRUMENT AND ITSCORRELATION WITH OBSERVER RATINGS OF TEACHER-PUPIL RELATIONSHIPS (TEACHER INTERVIEWS, CALIFORNIA, TEACHER EVALUATION) Ed.D dissertation, Loma Linda University MOODY, WALTER EUGENE (1973) A COMPARISON OF TEACHER PERCEIVER INTERVIEWGUIDE AND MINNESOTA TEACHER ATTITUDE INVENTORY SCORES WITH SUPERVISORRANKING OF SPECIAL EDUCATION TEACHERS Ed.D dissertation, University of Kansas N=77 all levels TPI Principal Evaluation N=? Seventh-day Adventist schools TPI Principal evaluation using ORS McDaniel Observer Rating Scale Correlation /Effect Size Reliability Comments Of little comparative use Compared college student scores with average national TPI scores Document could not be located at KU No significant main effect ANOVA F values were pretty low thus the correlation probably was close to zero 0.02 Sample was divided into categories based on UTPI score and ANOVA conducted Concluded no relationship Dissertation not available, thus information derived from abstract Excellent background review of the TPI development and older studies Early studies indicated 0.7-0.8 KR internal reliability Document could not be located at KU Citation Sample Independent measure Martin, Linda J (2008) Searching for effective teachers: A statistical analysis of the Ventures for Excellence Teacher Interview questionnaire Ed.D dissertation, Seattle Pacific University N=315 VE Interview Mills, Scott A (2007) An investigation of the predictive validity of a structured teacher-interview instrument Ph.D dissertation, Purdue University N=61 grades 3-5 sample was used for other purposes N=38 teachers had VE scores VE interview Neal, Stephen O (1997) A comparison of principals' perceptions of teacher candidates and selected talent profiles Ed.D dissertation, Northern Illinois University OVERMAN, WILLIAM F (1981) THE IMPORTANCE OF THE TEACHER PERCEIVERINTERVIEW AS A TEACHER SELECTION CRITERION TO ADMINISTRATORS INRELATIONSHIP TO THEIR LEADERSHIP STYLE Ed.D dissertation, University of Northern Colorado Ostlund, Charles N (2006) The Teacher Perceiver Interview as apredictor of teacher retention in special education Ed.D dissertation, University of Virginia Papiernik, Henry Joseph (1988) An analysis of rating proficiency in the use of the teacher perceiver interview in the selection of teachers Ed.D dissertation, Temple University Dependent Measure Northwest Evaluation Association’s Measure of Academic Progress in reading, LA, and math TPI Correlation /Effect Size 0.36 (Reading) 0.33 (LA) 0.27 (math) 7%-13% of variance Reliability Comments 0.757 Cronback “Construct validity of the instrument was not confirmed” Three factors were discovered: interpersonal sensitivity, engagement beliefs and strategies, and flexible thinking Unclear if the teachers having VE scores were younger than those with no scores Appears to be a lot of data lost as many correlations were based on only 100 or so students Compared principal and teacher self-ratings on the 12 TPI themes Did not report a great deal of overlap Dissertation not available, thus information derived from abstract Qualitative study of how administrators perceived the value of the TPI and how they used the results N=339 special education teachers TPI Interview N=127 principals up for recertification on the TPI TPI Rates of retention of teachers TPI explained almost no variance (full Dissertation not available, thus information derived from abstract) Full Dissertation not available, thus information derived from abstract Original proficiency gained from the TPI training is maintained through recertification program So what was learned? Isn’t that obvious? Citation Potthoff, Dennis Eugene (1991) A qualitative case study of the relationship between the scores of selected students on the SRI/UNL Teacher Perceiver Interview and their subsequent student teaching success Ph.D dissertation, The University of Nebraska - Lincoln, Ricker, Ann Marie McNeela (1999) Teacher selection: Chicago Region One elementary neighborhood school administrators since school reform Ed.D dissertation, Northern Illinois University Robison, Scott E (2003) An exploratory study of the utility of a structured interview in the employment of classroom teachers Ph.D dissertation, Purdue University Sample Independent measure Dependent Measure Correlation /Effect Size Reliability Comments Qualitative study of elementary education majors Interview scores and classroom behavior were different Single building Qualitative study of what principals deem important in a candidate N=14 (half scored well the other half did not) 2-4 grade teachers N=263 students VE Interview Northwest testing data, student surveys, supervisor’s assessment SIMMONS, BERTIE AUSTIN (1984) AN ANALYTICAL AND COMPARATIVE STUDY OF THESELECTION RESEARCH INTERVIEW AS AN INSTRUMENT FOR SELECTING TEACHERS FORA LARGE, URBAN SCHOOL DISTRICT Educat D dissertation, Texas Southern University N=164 teachers TPI Principal rating Employment stability SIMMONS, JAMES ELWOOD (1976) A STUDY TO TEST THE TEACHER PERCEIVERINTERVIEW AS AN INSTRUMENT THAT WOULD SELECT VOCATIONAL AGRICULTUREINSTRUCTORS THAT DEVELOP POSITIVE RAPPORT WITH THEIR STUDENTS Ph.D dissertation, The University of Nebraska - Lincoln Sawyer, Gayle Buckheister (2005) A study using the Star Teacher Selection Interview to predict the successful performance of teachers in South Carolina's Program of Alternative Certification for Educators Ph.D dissertation, University of South Carolina N=42 TPI N=33 first year teachers from alternative certification program Star Teacher 5/14 teachers moved to the other group after one year and reinterviewing using the VE Comparison of high and low groups was significant in LA and Math but not reading Comparison groups was not significant for student surveys Principal evaluations were handled in a qualitative manner Divided the teachers into groups based on supervisor’s rating followed by ANOVA Also did a correlation which was NS Student evaluations-rapport Administrator ratings No significant relationship via ANOVA 0.372 correlation that was not significant 0.40 with student ratings 0.37 with administrator ratings Principals evaluation No predictive value Dissertation not available, thus information derived from abstract Not all the TPI scales correlated but the total did Citation Sample Independent measure Dependent Measure Correlation /Effect Size Reliability Comments Schmidt, Wayne D (1993) The Teacher Perceiver Interview as a predictor of teacher performance Ph.D dissertation, University of Missouri -Columbia N=154 hired and 144 not hired TPI PBES-Principal evaluation TPI was a good predictor if candidate was hired (TPI use as screening device thus results were obvious) Simmons, Anne Carol Brown (1996) A validation study of the use of the Teacher Perceiver Interview (TPI) in a selected southeastern urban high school Ph.D dissertation, Georgia State University N=55 teachers TPI THOMPSON, ALBERT RAY (1982) THE TEACHER PERCEIVER INTERVIEW AS RELATEDTO STUDENT ACHIEVEMENT Ed.D dissertation, University of Northern Colorado Watson, Robert Lewis (1991) A study comparing the profile of selected University of Wyoming student teachers with the SRI Gallup Teacher Perceiver Interview Ed.D dissertation, University of Wyoming ZARANEK, RICHARD JAMES (1983) A CORRELATIONAL ANALYSIS BETWEEN THETEACHER PERCEIVER INTERVIEW AND TEACHER SUCCESS IN THE CHIPPEWA VALLEYSCHOOL SYSTEM (MICHIGAN) Ed.D dissertation, Western Michigan University Gatti, Sharon (1990) Predictors of future teacher effectiveness for nontraditional teacher certification candidates Ph.D dissertation, Colorado State university N=27 4th and 5th grade teachers TPI TPAQ-Principal evaluation— measure GTOI-Principal evaluation Student scores on math, reading, and la on the CTBS 0.015 for TPI and Principal Evaluation N=155 unknown TPI total score not related to achievement Dissertation not available, thus information derived from abstract 40 student teachers TPI Compared to TPI national sample N=21 elementary N=29 secondary teachers TPI Supervisor Evaluation Non-traditional student teachers TPI plus a host of other measures Florida Performance Measurement System (FPMS)— I assume completed by cooperating teacher GTOI was not usable since little if any variation TPAQ study was qualitative in nature Study of little use Of little value in assessing the TPI R=0.46 Elementary teachers R=-0.12 Secondary teachers TPI not a significant predictor Range restriction effect present Job satisfaction also included in results Note absolute differences between elementary and secondary teachers Not very relevant for this study Dissertation not available, thus information derived from abstract 10 Citation Sample Independent measure Dependent Measure Correlation /Effect Size English, John Jarvis (1983) The Relationship between Structured Selection Criteria and the Assessment of Proficient Teaching Ed.D dissertation, University of Virginia N=60 elementary and MS teachers TPI themes significantly correlated Dissertation not available, thus information derived from abstract Trinka, Rosanne (1983) The Relationship between Administrative Ratings, Selection Criteria, and Demographic Variables of Teachers Employed by Baltimore County Public Schools, Ed.D University of Maryland Klussmann, Ducan F (2004) The Impact of Teachers Selected by the Star Teacher Interview Process on Student Achievement Ed.D dissertation, Seton Hall N=290 second-forth year teachers TPI Host of other measures Assessment for Professional Development— based on effective teaching research (observational data?) Principal teacher evaluations TPI correlates only for 4th year group Dissertation not available, thus information derived from abstract Examined correlations separately for 2,3,4 year teachers N=87 reading teachers N=88 math teachers 1351 student scores N=124 full version N=72 Short version Star Teacher Reading and math achievement scores No statistical relationship Dissertation not available, thus information derived from abstract TPI (both versions) Principal ratings and attendance 0.25 rating Full Version 0.18 rating short version -0.24 attendance FV 0.09 attendance SV Young, I Philip, and Delli, Dane (2002) The Validity of the Teacher Perceiver Interview for Predicting Performance of Classroom Teachers Educational Administration Quarterly, 38, 5, p586-612 Ryan, Patricia M., and Alcock, Martha A (2002) Personal and Interpersonal Attributes in Selecting teachers, Action in Teacher Education, 24, 1, p58-67 Sentz, Erma I (1991) Predicting success in Student Teacher from Teacher Perceiver Screeners and PreProfessional Skills Tests, ED329517 Reliability 0.57 LV 0.27 SV Individual scales ranged from 0.550.00 on LV Comments Examined outcomes via two data sets (using long and short versions) Summary of the characteristics of the TPI, Star, Praxis, National Board for Professional Teaching Data were collected throughout the 198990 school year at St Cloud State University TPI Student teaching success Results indicated that the Teacher Perceiver Screener showed promise for use in predicting student teachers' abilities to motivate and communicate with students and to apply their skills in using specific teaching methods in classrooms 11 Citation Sample Report: AISD-ORE-83.43 (1984) No Panaceas: A Brief Discussion of Teacher Selection Instruments, Austin Independent School District, Tx Office of Research and Evaluation Muller, Gale M (1978) In Defense of the Teacher Perceiver, Phi Delta Kappan, 59, 10, p 684-5 Haefele, Donald L (1978) The Teacher Perceiver Interview: How Valid, Phi Delta Kappan, 59, 10, p 683-4 Miller, J D and others (1977) A Preliminary Investigation of the Teacher Perceiver Instrument for Teacher Selection, RDCTE-TR-5077 Independent measure Dependent Measure Correlation /Effect Size Reliability TPI Comments Little support of TPI Drozd, Dee (2006) Poster presented at the 21rd annual conference of the Society for Industrial and Organizational Psychology Study Drozd, Dee (2006) Poster presented at the 21rd annual conference of the Society for Industrial and Organizational Psychology Study N=180 teachers across 12 school districts (all levels) Teacher Insight Principal and student ratings N=621 teachers from 25 districts (all levels) Teacher Insight Principal and student ratings Drozd, Dee, Kirk, Paul, Hays, Theodore (2008) Two Validation Studies of a School Teacher Selection System, Poster presented at the 23rd annual conference of the Society for Industrial and Organizational Psychology— Study 190 alternative certified teachers placed into groups based on their student’s academic progress in gain scores Teacher Insight Student gain scores in math and reading 0.25/0.28 correlations with students and principals 0.21 with students 0.12 with principal rating 0.33 0.81 Cronback on entire instrument 0.77 Cronback Conclusions 1)empirical bases for claims of the various Perceiver systems are weak 2)evidence that the 60 question format of the TPI is partially predictive of student ratings of new teachers but not of outcomes of good teaching 3) no evidence the TPI is superior to classical interview Initial tested version was 35 items No significant age or race bias Revised 35 item version 34 item Insight instrument 12 Citation Sample Drozd, Dee, Kirk, Paul, Hays, Theodore (2008) Two Validation Studies of a School Teacher Selection System, Poster presented at the 23rd annual conference of the Society for Industrial and Organizational Psychology— Study N=72 elementary and middle school teachers Metzger, Alan and Meng-Jia Wu (2008) Commercial Teacher Selection Instruments, Review of Educational Research (in press) Meta analysis of 24 studies on the TPI TPI Allshouse, Tim J (2003) A study of the concurrent validity of the "Knowledge of Content Scale" from the American Association of School Personnel Administrators' (AASPA) Interactive Computer Interview System (ICIS) Ed.D dissertation, University of Kansas Beutel, Jennifer L (2006) The development and field test of an employment interview system designed to identify the highly qualified special education teacher Ed.D dissertation, The University of Kansas Cowan, Patrick Lawrence (1999) A comparison of the predictive power of competency-based and personalitybased structured interviews in identifying successful teachers Ed.D dissertation, University of Kansas N=21 Content experts N=20 Not content experts ICIS Knowledge of Content Scale N=40 special education teachers ICIS—Special Education 27 principals rated transcript interviews using both personality and jobrelated questions Job vs personality questions Published in Journal of Personnel Evaluation in Education Independent measure Dependent Measure Correlation /Effect Size Percent of students in a teacher’s class making a gain on SAT10 and Florida Comprehensive Assessment Test 0.563 however from taking only the top and bottom quadrant Scores in 1st and 4th quadrant were deleted 0.28 from 24 studies Reliability Comments No mention of how many middle scores were deleted Significant ttest differences between groups 0.97 on Knowledge of Content Scale Special education supervisor rating 0.489 for supervisors 0.93 on total tool Principal ratings of transcripts of interviews Both types of questions significantly predicted teacher quality Meta analysis of 24 correlational studies Concluded that a modest relationship exists Raises questions about what the TPI actually measures Compared individuals with strong content background in a given field with excellent teachers who did not have a strong background in content field Describes the development of the version for special education teachers The study sought to compare personality and jobbased structured interview formats The objective was to determine which of the two was better in terms of predicting the ultimate success of newly hired teachers One, the job-related questions had an overall classification accuracy of 86.3% while the personality questions accurately classified only 75% of the cases Published in Journal of Personnel Evaluation in Education 13 Citation Sample Independent measure Dependent Measure Correlation /Effect Size Reliability Comments Cox, David R (2006) A study of the reliability of the Aptitude/Suitability for Teaching in Urban Schools Scale from the American Association of School Personnel Administrators (AASPA) Interactive Computer Interview Instrument (ICIS) Ed.D dissertation, The University of Kansas N=30 urban teachers ICIS-Urban Principal and central office ratings of teachers 0.586 on urban scale 0.89 on urban scale of ICIS Published in Journal of Personnel Evaluation in Education 18: 201-218 N=45 ICISParaprofessional Evaluations within district by supervisor 0.80 for instrument 0.958 for instrument Paper presented at AASPA in 2007 N=44 22 special education administrators and 22 general administrators N=37 secondary school counselors ICIS-Special Education Accuracies of scores on training video ICIS-Counselor Supervisors rating 0.52 0.90 for total and all subscales N=31 clerical staff ICIS—Office Staff Supervisors rating 0.45 0.84 for total Monte Carlo simulations ICIS Published in Journal of Personnel Evaluation in Education 18: 201-218 Dillon, Amy (2006) The development of an interview system to identify quality paraprofessionals Ed.D dissertation, The University of Kansas Dugan, Erin P (2007) The reliability of building administrators' and special education administrators' ratings of special education teacher applicants using an Interactive Computerized Interview System Ph.D dissertation, University of Kansas Green, Mary Elizabeth (2005) The development and field testing of an interview instrument designed to identify quality secondary school counselors Ed.D dissertation, The University of Kansas Special education administrators were no better than regular administrators in identifying quality teacher responses Published in Journal of School Counseling Published in Journal of School Counseling Hale, Tina M (2006) The development and field test of an interview system to identify quality school district office personnel Ed.D dissertation, The University of Kansas Longenecker, Dale (2006) An investigation of the metric properties of the interactive computer interview system Ed.D dissertation, The University of Kansas This research concluded that the branching format allowed for significantly more accurate results from candidates with inconsistent responses, while not elongating the interview process for all candidates 14 Citation Sample Independent measure Dependent Measure Correlation /Effect Size Reik, Michael J (2007) An examination of concurrent validity of the Interactive Computer Interview System (ICIS) using communication arts and math achievement on the Missouri Assessment Program (MAP) as the outcome measure Ed.D dissertation, University of Kansas Smith, Vicki W (2006) The development and field testing of a school psychologist employment interview instrument Ed.D dissertation, The University of Kansas N=40 third and fourth grade teachers ICIS Missouri MAP achievement tests 0.454 N=41 ICIS-School Psychologist Supervisors rating 0.51 for total instrument 0.87 for total instrument Results of this study found that practitioners who obtained higher average scores on the interview were rated significantly higher by their supervisors N=30 ICIS-Urban Supervisors rating 0.59 for total instrument 0.89 N=50 elementary teachers ICIS Examined scores across age and experience “Statistical analyses indicated a significant correlation between the ICIS-Urban scales Aptitude/Suitability for Urban Teachers (UT), Knowledge of Teaching (KT) and Knowledge of Students (KS) and the district rating of the teacher but a weak correlation between the ICIS-Urban scales of Knowledge of Content (KC) and Working with Others (WO)” “This indicates a statistically significant relationship exists between experience, age and scores earned on the ICIS tool However, given such a low r-square value, the usefulness of these results is questionable Based on the results of this analysis, it appears age and experience matter but probably not sufficiently to warrant adjusting the interview scores based on these factors.” Published in Journal of Applied School Psychology Springston, Scott M (2006) A study of the reliability and concurrent validity of the ICIS-Urban teacher interview instrument Ed.D dissertation, The University of Kansas Published in Evaluation of School Personnel Journal Stevenson, Gary L (2005) A study comparing the scores of experienced and inexperienced teachers using the American Association of School Personnel Administrator's Interactive Computer Interview System Ed.D dissertation, The University of Kansas Reliability Comments Significant correlation between achievement and ICIS total scores ¾ subscales also produced significant correlations Knowledge of Content was only no-significant subscale at p 001 level 15 Citation Sample Independent measure Dependent Measure Correlation /Effect Size Weishaar, Michael (2007) A study of the predictive validity of the Interactive Computer Interview System (ICIS) Ph.D dissertation, University of Kansas N=40 ICIS Stanford Achievement test No correlation Reliability Comments “This research concluded that the ICIS interview did not effectively differentiate between teachers whose students performed at different levels on a given standardized test This could have been due to a number of factors One such factor is the time lapse between the time the standardized tests were given and the time the interviews were conducted.” (3 years) 16 ... Meng-Jia Wu (20 08) Commercial Teacher Selection Instruments, Review of Educational Research (in press) Meta analysis of 24 studies on the TPI TPI Allshouse, Tim J (20 03) A study of the concurrent... ICIS—Office Staff Supervisors rating 0.45 0.84 for total Monte Carlo simulations ICIS Published in Journal of Personnel Evaluation in Education 18: 20 1 -21 8 Dillon, Amy (20 06) The development of. .. Performance of Classroom Teachers Educational Administration Quarterly, 38, 5, p586-6 12 Ryan, Patricia M., and Alcock, Martha A (20 02) Personal and Interpersonal Attributes in Selecting teachers, Action