Developing a validity argument for the english placement test at btec international college DANANG CAMPUS

THE UNIVERSITY OF DANANG UNIVERSITY OF FOREIGN LANGUAGE STUDIES VÕ THỊ THU HIỀN DEVELOPING A VALIDITY ARGUMENT FOR THE ENGLISH PLACEMENT TEST AT BTEC INTERNATIONAL COLLEGE DANANG CAMPUS MASTER THESIS IN LINGUISTICS AND CULTURAL STUDIES OF FOREIGN COUNTRIES Da Nang, 2020 THE UNIVERSITY OF DANANG UNIVERSITY OF FOREIGN LANGUAGE STUDIES VÕ THỊ THU HIỀN DEVELOPING A VALIDITY ARGUMENT FOR THE ENGLISH PLACEMENT TEST AT BTEC INTERNATIONAL COLLEGE DANANG CAMPUS Major: ENGLISH LANGUAGE Code: 822.02.01 MASTER THESIS IN LINGUISTICS AND CULTURAL STUDIES OF FOREIGN COUNTRIES SUPERVISOR: VÕ THANH SƠN CA, Ph.D Da Nang, 2020 ii TABLE OF CONTENT Statement of authorship i Table of content ii List of figures iv List of tables .v Acknowledgments vi Abstract vii CHAPTER INTRODUCTION .1 1.1 INTRODUCTION TO TEST VALIDITY 1.2 THE STUDY 1.3 SIGNIFICANCE OF THE STUDY .3 CHAPTER LITERATURE REVIEW 2.1 STUDIES ON VALIDITY DISCUSSION 2.1.1 The conception of validity in language testing and assessment .5 2.1.2 Using interpretative argument in examining validity in language testing and assessment 2.1.3 The argument-based validation approach in practice so far 10 2.1.4 English placement test (EPT) in language testing and assessment .16 2.1.5 Validation of an EPT 16 2.1.6 Testing and assessment of writing in a second language .18 2.2 GENERALIZABILITY THEORY (G-THEORY) 20 2.2.1 Generalizability and Multifaceted Measurement Error 21 2.2.2 Sources of variability in a one-facet design 21 2.3 SUMMARY .22 CHAPTER METHODOLOGY 23 3.1 RESEARCH DESIGN .23 3.2 PARTICIPANTS 23 3.2.1 Test takers .24 3.2.2 Raters 25 iii 3.3 MATERIALS .25 3.3.1 The English Placement Writing Test (EPT W) and the Task Types 25 3.3.2 Rating scales 25 3.4 PROCEDURES 26 3.4.1 Rater training 26 3.4.2 Rating .28 3.4.3 Data analysis 28 3.5 DATA ANALYSIS 28 3.5.1 To what extent is test score variance attributed to variability in the following: a task?; b rater? 28 3.5.2 How many raters and tasks are needed to obtain the test score dependability of at least 85? .29 3.5.3 What are vocabulary distributions across proficiency levels of academic writing? 29 CHAPTER RESULTS 31 4.1 RESULTS FOR RESEARCH QUESTION 31 4.2 RESULTS FOR RESEARCH QUESTION 33 4.3 RESULTS FOR RESEARCH QUESTION 38 CHAPTER DISCUSSION AND CONCLUSIONS 40 5.1 GENERALIZATION INFERENCE 40 5.2 EXPLANATION INFERENCE 42 5.3 SUMMARY AND IMPLICATIONS OF THE STUDY 43 5.4 LIMITATIONS OF THE STUDY AND SUGGESTION FOR FUTURE RESEARCH 43 REFERENCES 45 APPENDIX A EPT W RATING SCALE QUYẾT ĐỊNH GIAO ĐỀ TÀI (Bản sao) iv LIST OF FIGURES Number Name of Figure of Figure 2.1 2.2 An illustration of inferences in the interpretative argument (adapted from Chapelle et al 2008) Bridges that represent inferences linking components in performance assessment (adapted from Kane et al., 1999) Page 2.3 Evidence to build the validity argument for the test 15 3.1 Participants 24 5.1 5.2 Generalization inference in the validity argument for the PT W with assumptions and backing The explanation inference in the validity argument for the PT test with assumption and backing 40 42 v LIST OF TABLES Number Name of table of table 1.1 The structure of the EPT W Page Summary of the inferences, warrants in the TOEFL validity 2.1 argument with their underlying assumptions (Chapelle et al., 12 2010, p.7) 2.2 A framework of sub-skills in academic writing (McNamara, 1991) 19 3.1 Texts and word counts in the two levels of the EPT sub-corpora 30 4.1 Variance components attributed to test scores 32 4.2 Dependability Estimate 37 4.3 Distribution of vocabulary across proficiency levels 38 vi ACKNOWLEDGMENTS I would like to express my deeply sincere appreciation for my supervisor, Dr Vo Thanh Son Ca who inspired and gave me her devoted instruction throughout the period until the project work was conducted From the onset, my research topic was quite wide, but with her support and guidance, I have learned how to combine theory and practice in use Thanks to her instruction and willingness to motivate and guide me with many questions and comments, I came to be deeply aware of the important role of doing research in this area of language testing and assessment More than that, I welcomed her dedicated and detailed support since her quick feedback and comments on my drafted work I mean that she observed thoroughly every step of my work, helped me make significant improvements Without my supervisor’s dedicated support, the research would not have been completed Moreover, I would like to take this chance to thank my family and friends, who always take care, assist and encourage me I could not have completed my dissertation, without the support of all these marvelous people vii ABSTRACT Foreign or second language writing is one of the important skills in language learning and teaching, and universities use scores from writing assessments to make decisions on placing students in language support courses Therefore, in order for the inferences based on scores from the tests are valid, it is important to build a validity argument for the test This study built a validity argument for the English Placement Writing test (EPT W) at BTEC International College Danang Campus Particularly, this study examined two inferences which are generalization and evaluation by investigating the extent to which tasks and raters attributed to test score variability, and how many raters and tasks are needed to get involved in assessment progress to obtain the test score dependability of at least 85, and by investigating the extent to which vocabulary distributions were different across proficiency levels of academic writing To achieve the goals, the test score data from 21 students who took two writing tasks were analyzed using the Generalizability theory Decision studies (D-studies) were employed to investigate the number of tasks and raters needed to obtain the dependability score of 0.85 The 42 written responses from 21 students were then analyzed to examine the vocabulary distributions across proficiency levels The results suggested tasks were the main variance attributed to variability of test score, whereas raters contributed to the score variance in a more limited way To obtain the dependability score of 0.85, the test should include 14 raters and 10 tasks or 10 raters and 12 tasks In terms of vocabulary distributions, low level students produced less varied language than higher level students The findings suggest that higher proficiency learners produce a wider range of word families than lower proficient counterparts CHAPTER INTRODUCTION This chapter presents the introduction to test validity and the purpose of this thesis The chapter concludes with the significance of this thesis 1.1 INTRODUCTION TO TEST VALIDITY Language tests are needed to measure students’ ability in English in college settings One of the most common tests developed is entrance tests or placement tests which are used to place students into appropriate language courses Thus, the use of test scores cannot be denied as a very important role The placement test at BTEC International College is used as an example for building this research study and helping to build up validity argument with further research purposes It will have certain impact on students, administrators, and instructors at BTEC International College Da Nang Campus First, the test score helps students know whether they are ready for collegiate courses taught in English Second, the test score helps administrations at English programs place students into appropriate English language use class level The information on students’ ability would help instructors with their instruction or lesson planning Besides, students would also value the importance of language use ability for their success in college so that they pay more attention to their improvement of academic skills With the importance role of entrance tests, test validity is the focus of this study Test validity is the extent to which a test accurately measures what it is supposed to be measure and validity refers to the interpretations of test score entailed by proposed uses of tests which is supported by evidence and theory (American Educational Research Association, American Psychological Association, & National Council on Measurement in Education, 1999) In other words, validation is a progress in which test developers and/or test users gather evidence to provide “a sound scientific basis” for interpreting test scores Validity researchers emphasize on quality, rather than quantity of 37 Table 4.2 Dependability Estimate G study Alternative D studies Source of variation Person (p) 3 10 14 10 Nt = 10 12 1.063 1.063 1.063 1.063 1.063 1.063 1.063 0.009 0.0045 0.003 0.003 0.0009 0.0299 0.0009 0.299 0.1495 0.0598 0.0498 0.0598 0.0299 0.0249167 0.71 0.355 0.236667 0.23667 0.071 0.05071429 0.071 1.264 0.632 0.2528 0.210667 0.2528 0.1264 0.105333 0.004 0.001 r Tasks (t) p Raters (r) Nr = t Pr pr Pt pt Rt 0.0002667 0.0002222 0.00008 0.00002857 0.0000333 rt Prt, e 0.277 0.06925 0.01847 0.015389 0.00554 0.00197857 0.0023083 2.251 1.05625 0.507937 0.462726 0.32934 0.17909286 0.1786413 0.68 0.7 prt,e Rel Ep2Rel 0.32 0.5 0.76 0.8558 0.8561 38 4.3 RESULTS FOR RESEARCH QUESTION What are vocabulary distributions across proficiency levels of academic writing? Table 4.3 Distribution of vocabulary across proficiency levels Vocabulary distributions EPT L1 EPT L2 (1) Total number of types 450 524 (2) Total number of tokens 1260 1686 (3) Lexical diversity (type-token ratio) 0.36 0.31 0.56 (701/1260) 0.52(859/1670) (5a) K1 tokens 0.887 0.896 (5b) K2 tokens 0.52 0.53 (5c) AWL tokens 0.05 0.15 (6) Total number of word families 372 379 (4) Lexical density (5) Lexical sophistication Note TTR = mean of type-token ratio per text; Lexical density = number of content words divided by total number of words; K1 = the most frequent 1000 words of English; K2 = the second most frequent thousand words of English; AWL = academic word list As presented in Table 4.3, there was a progression in the number of types and tokens in different levels of writing proficiency The number of types and tokens was low in low-level written responses (450 types and 1260 tokens for EPT L1) but high in higher level responses (524 types and 1686 tokens for EPT L2) This suggests that written output by higher-level learners was longer and more varied because at higher-levels, learners acquire greater linguistic knowledge This result is in accordance with Shaw and Weir’s (2007) findings for Main Suit written learner output in which the authors suggested that as learners developed in proficiency, they produced a wider range of vocabulary in terms of both tokens and types 39 Lexical diversity was measured by mean of type-token ratio (TTR) per text The responses from both levels EPT L1 and EPT L2 used TTR in small different way (the TTRs 0.36 for EPT L1 and 0.31 for EPT L2) For lexical sophistication, the large percentage of words in the responses by the two groups from themost frequent 1000 words of English (K1 tokens) and their frequency decreased as proficiency levels increased, with 0.36 in EPT L1 and 0.31 in EPT L2 In contrast, K2 tokens and AWL increased across score levels EPT L1 responses included 0.52 K2 tokens and 0.05 AWL, and EPT L2 responses included 0.53 K2 tokens and 0.15 AWL However, there was no statistically significant difference in lexical diversity, lexical density, and lexical sophistication across two levels of proficiency As for word families, there were 372 word families in the EPT L1 and 379 in the EPT L2 There was statistically significant difference in the proportion of word families across two groups of learners This suggests that the higher-level used more word families in their written discourse, suggesting that the higher-level written responses were linguistically and cognitively than the lower-level output Overall, although there is not much difference in lexical density, lexical diversity, and lexical sopihistication among the EPT levels, when combined with the other individual measures, such as types, tokens, and word families, the higherlevel learners’ written dicourse come across as being more complex than the lowerlevel learners’ written language 40 CHAPTER DISCUSSION AND CONCLUSIONS The purpose of this study was to build a validity argument for the EPT W test The study focused on two inferences of generalization and explanation (Chapelle et al., 2008) For the generalization inference, this study investigated the extent to which tasks and raters attributed to score variability and how many tasks and raters are needed to get involved in assessment to obtain the test score dependability of at least 85 For the explanation inference, this study analyzed discourse of written responses of two groups of students across different proficiency levels in terms of the extent to which vocabulary distributions are different across proficiency levels of academic writing This chapter presents a summary and discussion of the findings of each question 5.1 GENERALIZATION INFERENCE CONCLUSION/CLAIM: observed scores on EPT Writing test reflects what expected scores would be over the relevant parallel versions of tasks and test forms and across raters WARRANT: observed scores are estimates of expected scores over the relevant parallel versions of tasks and test forms and across raters ASSUMPTION 1: sufficient number of tasks are included on the test to provide stable estimates of test takers’ performance BACKING 1: the test with two tasks provided 50% dependable estimates of test takers’ performances Generalization inference ASSUMPTION 2: sufficient number of raters are included on the test to provide stable estimates of test takers’ performance BACKING 2: the test with two raters provided 50% dependable estimates of test takers’ ability GROUNDS/DATA: observed scores Figure 5.1 Generalization inference in the validity argument for the PT W with assumptions and backing 41 The first assumption is sufficient number of tasks included on the test to provide stable estimates of test takers’ performance Backing for the first assumption was supported by generalizability analysis The results showed that the PT W with two tasks provided 50% performance of test takers The second assumption is sufficient number of raters included on the test to provide stable estimates of test takers’ performance Backing for the second assumption was also supported by G-theory The findings suggested that the test with two raters provided 50% performance of test takers Decision studies were employed to figure out the number of tasks and raters needed to obtain the dependability index of 0.85 In order to reach to the dependability score of 0.85, we would need to have 14 raters and 10 tasks or 10 raters and 12 tasks However, due to our practicality and resources at BTEC, we could not meet the demand of 14-rater-&-10-task or 10-rater-&-12-task test scale Different from the other high-stakes test such as TOEFL iBT or IELTS, the EPT W is a medium-stakes test which was intended to place students into language support courses That is, PT at BTEC is a medium-stake test due to its compulsoriness If the students could not pass the third level according to the rating scale (see Appendix A), they would not enter the majoring at BTEC Moreover, the college regulations would not force students that did not pass the PT to drop out They would have another chance to retake the test Therefore, given the nature of the EPT W test and the availability of resources at BTEC, it would be appropriate to decrease the dependability score to 0.7 With this dependability score, we would need raters and tasks or raters and tasks These numbers of tasks and raters seem practical, given the current resources at BTEC 42 5.2 EXPLANATION INFERENCE CONCLUSION/CLAIM: expected scores on the PT writing reflects test takers’ academic writing proficiency WARRANT: expected scores are attributed to a construct of academic language proficiency ASSUMPTION 1: The linguistic knowledge, processes, and strategies required to successfully complete tasks vary in keeping with theoretical expectations Explanation inference BACKING 1: vocabulary distributions were different across proficiency levels GROUNDS/DATA: test takers’ written discourse Figure 5.2 The explanation inference in the validity argument for the PT test with assumption and backing The assumption underlying the warrant of explanation inference is the linguistic knowledge, processes, and strategies required to successfully complete tasks vary in keeping with theoretical expectations Backing for this assumption was supported by discourse analysis of test takers’ written responses in terms of vocabulary frequency The analyses of written discourse of EPT W suggested that there was variation in terms of lexical frequency distributions in students’ language between language proficiency levels The findings based on two EPT sub-corpora groups consisting of 42 texts and 3920 tokens suggested that the single word-based measures such as the number of types, tokens, and tokens increased as the proficiency levels increased, and higher proficiency learners produces a wider range of word families than lower proficient counterparts 43 5.3 SUMMARY AND IMPLICATIONS OF THE STUDY Overall, the results supported the two assumptions of generalization inference in a moderate manner That is, more evidence is needed to be investigated under generalization inference for the PT W test For the explanation inference, the only assumption investigated in this study was supported by qualitative evidence, although this is limited with quantitative analysis This study has several implications First, practical implication is that the two-task-&-two-rater test could reach a dependability estimate of 50% To obtain a higher score, we need to enlarge the scale of test with more tasks and raters included The number of tasks and raters can be decided by decision makers at the college, with a reference to the findings of this study Second, the research has methodological implications in that the use of mixed methods approach can be employed to maximize the backing for the assumptions of the inferences Third, the study suggests that raters and tasks were the main components attributed to the test score variability That implicates rater training was important in performance assessments such as writing or speaking tests 5.4 LIMITATIONS OF THE STUDY AND SUGGESTION FOR FUTURE RESEARCH As being the very first investigation into the validity issue of the EPT Writing test at BTEC, there exist a number of limitations on the results produced in the analyses of the study First, due to the scope of the study, only two inferences (generalization and explanation) were investigated A validity argument with strong evidence should be backed up with evidence from six inferences such as domain description, evaluation, generalization, explanation, extrapolation and utilization For each inference, there were many assumptions underlying the warrant that supported the inference The second limitation here was that just only one or two assumptions were conducted to give example of backing sought to support assumptions For instance, 44 the generalization that was based for the first and second research questions has three assumptions: 1) a sufficient number of tasks are included on the test to provide stable estimates of test takers’ performances, 2) Configuration of tasks on measures is appropriate for intended interpretation, 3) Appropriate scaling and equating procedures for test scores are used, 4) Task, test, and rating specifications are well defined so that parallel tasks and test forms are created This study focused on only the first assumptions (Chapelle et al., 2008) That led to the open suggestion for future research in the field of building validity argument for PT test with many other inferences supported by other assumptions Third, related to the linguistic field, the study examined the vocabulary distributions across the proficiency levels by calculating the texts and tokens based on two different sup-corpora groups In terms of linguistic knowledge, other aspects of language should be analyzed such as: grammar, semantic, pragmatics, etc In other words, more linguistic features should be analyzed in future research 45 REFERENCES American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1999) The standards for educational and psychological testing American Educational Research Association American Educational Research Association, American Psychological Association, & National Council on Measurement in Education (1985) The standards for educational and psychological testing American Educational Research Association Bachman, L F (1990) Fundamental considerations in language testing Oxford university press, 188-197 Bachman, L F & Palmer, A S (1996) Language testing in practice: Designing and developing useful language tests Oxford university press 1996 Borsboom, D & Mellenbergh, G J (2004) The concept of validity, 111(4), 1061-1071 Brown, C R., Moore, J L., Silkstone, B E & Botton, C (1996) The construct validity and context dependency of teacher assessment of practical skills in some pre-university level science examinations 3(3) 377-392 Brown, J D (1989) Improving ESL Placement tests using two perspective TESOL Quarterly, 23(1), 65-83 Brow, J D (1996) Testing in language programs New in language programs New Jersey: Prentice Hall 10 Chapelle, C A., Jamieson, J., & Hegelheimer, V (2003) Validation of a webbased ESL test Language Testing, 20(4), 409-439 11 Chapelle, C A., Enright, M K., & Jamieson, J M (2008) Building a validity argument for the test of English as a foreign language 12 Chapelle, C A., Enright, M K, & Jamieson, J M (2010) Does an argumentbased approach to validity make a difference? Educational measurement 46 issues and practice, 1(29), 3-13 13 Crooks, T J., Kane, M T., & Cohen, A S (1996) Threats to the valid use of assessments Assessment in education: Principles, policy & practice,.3(3), 265-286 14 Cronbach, L J., & Meehl, P E (1955) Construct validity in psychological tests Psychological Bulletin, 52(4), 281-232 15 Douglas, D (ed.) (2003) English language testing in U.S colleges and universities (2nd ed.) Washington, D.C: Association of International Educators 16 Douglas, D (2009) Understanding Language Assessment London: Hodder 17 Fulcher, G (1997) An English language placement test: Issues in reliability and validity Language testing, 14(2), 113-139 18 Kane, M (1992) An argument-based approach to validity Psychological bulletin, 112(3), 527-535 19 Kane, M (2001) Current concerns in validity theory Journal of educational measurement, 38(4), 319-342 20 Kane, M (2002) Validating high stakes testing programs Educational measurement: Issues and practice, 21 (1), 31-41 21 Kane, M (2004) The analysis of interpretive arguments: some observations inspired by the comments Measurement: Interdisciplinary research and perspective, 2(3), 192-200 22 Kane, M (2006) Validation In R Brennon (ed.), Educational measurement (4th ed.) Westport, CT: American Council on Education and Praeger, 17-64 23 Kane, M., Crooks, T & Cohen, A (1999) Validating measures of performance Educational measurement: Issues and practice, 18(2), 5-17 24 Lee, Y J & Greene, J (2007) The predictive validity of an ESL placement test: a mixed methods approach Journal of mixed methods research, 1(4), 366-389 47 25 Messick, S (1989) Meaning and values in test validation: The science and ethics of assessment America Educational Research Association & SAGE Publications, 18(1), 5-11 26 Mislevy, R L (2003) Argument substance and argument structure in educational assessment CSE Technical Report 605 Los Angeles: Center for the study of evaluation 27 Shavelson, R., & Webb, N (1991) Generalization theory: A primer Sage Publications 28 Raimes (1994) Testing writing in EFL exams: The learners’ viewpoint as valuable feedback for improvement Procedia - Social and behavioral sciences, 199 (2015), 30-37 29 Lines (2004) Guiding the reader (or not) to re-create coherence: observations 30 on postgraduate student writing in an academic argumentative writing task Journal of English for academic purposes, 16(2014), 14-22 31 Toulmin, S., Rieke, R., & Janik, A (1984) An introduction to reasoning New York: Macmillan 32 Usaha, S (2000) Effectiveness of Suranaree University’s English placement test Suranaree University of Technology Retrieved on 12th September 2010 from http://hdl.handle.net/123456789/2213 33 Wall, D., Claphan, C & Alderson, J C (1994) Evaluating a placement test Language testing, 3(11), 321-344 APPENDIX A EPT W RATING SCALE Criteria Task Does not attend achievement Level (Top Notch Level (Top Notch Fundamental) 1) 0-4.5 4.5-5.0 Writes nothing or pts Level (Top Notch 3) 5.0-5.5  Answer is barely  Answer related to the task Level (Top Notch 2) 5.5-6.5  Attempts to address  Attempts to address sometimes presents the task but answer the task but usually related ideas often write no English does not cover all key points address all key points words Grammatical range Does not attend and  Barely correct accuracy Writes nothing or grammatical pts write no English structures words tenses uses  Sometimes correct uses  Attempts to use a  Usually Uses only a simple variety of structures, range of structures; sentences (the first but only rare use of Attempts and subject always) and subordinate clauses correct verbs forms of sentences but these tend Have complex to be less frequent accurate than simple grammatical errors sentences  Have frequent grammatical errors Criteria Lexical Does not attend resource pts Level (Top Notch Level (Top Notch Fundamental) 1) 0-4.5 4.5-5.0  Only uses a few  Uses isolated words Level (Top Notch 2) 3) 5.0-5.5 limited  Uses a Level (Top Notch 5.5-6.5 basic  Uses a limited range of words and vocabulary which may range of vocabulary, Writes nothing or expressions with no be used repetitively or but this is minimally write no English control words formation of word which may be adequate for the task and/or inappropriate for the spelling task Has a good control of  Has limited control word formation of word formation Coherence Does not attend and cohesion pts  Rarely writes any Has message due very to control of information Writes nothing or lack of cohesive organisational write no English devices words close and of features  A lot of spelling Frequently and such connection cohesive devices among sentences punctuation spelling  Present little  Presents s and information with ideas but these are some organisation but as not arranged there may be a lack coherently and there of overall progression is no clear progression  Uses has in the response and  Uses some devices cohesive effectively, basic but cohesion within Criteria Level (Top Notch Level (Top Notch Fundamental) 1) 0-4.5 4.5-5.0 Level (Top Notch 2) Level (Top Notch 3) 5.0-5.5 5.5-6.5 errors that hinder punctuation errors cohesive devices but and/ comprehension hinder these that comprehension may or between be sentences may be inaccurate or repetitive faulty or mechanical  Sometimes has  May be repetitive spelling and because of lack of punctuation errors that referencing may comprehension and hinder substitution  Has some spelling and punctuation errors but does not hinder comprehension ... only 42 examinations were rated) The raters read each performance entirely and then gave analytical ratings and a holistic rating for each task based on the rating rubric 3.4.3 Data analysis SPSS.. .THE UNIVERSITY OF DANANG UNIVERSITY OF FOREIGN LANGUAGE STUDIES VÕ THỊ THU HIỀN DEVELOPING A VALIDITY ARGUMENT FOR THE ENGLISH PLACEMENT TEST AT BTEC INTERNATIONAL COLLEGE DANANG CAMPUS Major:... recommendations 2.1.3 The argument- based validation approach in practice so far Several recent validation studies in language testing and assessment have attempted to take the argument- based approach