The quality of test questions in sample vietnam national high school english examination 2017

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES FACULTY OF ENGLISH LANGUAGE TEACHER EDUCATION GRADUATION PAPER THE QUALITY OF TEST QUESTIONS IN SAMPLE VIETNAM NATIONAL HIGH SCHOOL ENGLISH EXAMINATION 2017 Supervisor: Nguyễn Thị Ngọc Quỳnh, PhD Student: Lê Hoàng Kim Khuê Course: QH2013.F1.E2 HÀ NỘI – 2017 ĐẠI HỌC QUỐC GIA HÀ NỘI TRƯỜNG ĐẠI HỌC NGOẠI NGỮ KHOA SƯ PHẠM TIẾNG ANH KHÓA LUẬN TỐT NGHIỆP CHẤT LƯỢNG CÂU HỎI TRONG ĐỀ MINH HOẠ TRUNG HỌC PHỔ THÔNG QUỐC GIA NĂM 2017 MÔN TIẾNG ANH Giáo viên hướng dẫn: TS Nguyễn Thị Ngọc Quỳnh Sinh viên: Lê Hồng Kim Kh Khố: QH2013.F1.E2 HÀ NỘI – 2017 Signature of Approval: _ Supervisor’s Comments & Suggestions _ _ _ _ _ Acceptance I hereby state that I: Lê Hoàng Kim Khuê, class QH2013.E2, being a candidate for the degree of Bachelor of Arts (TEFL) accept the requirements of the College relating to the retention and use of Bachelor’s Graduation Paper deposited in the library In terms of these conditions, I agree that the origin of my paper deposited in the library should be accessible for the purposes of study and research, in accordance with the normal conditions established by the librarian for the care, loan or reproduction of the paper Signature Lê Hoàng Kim Khuê Hanoi, May 4th, 2017 Acknowledgements I would like to commence expressing my profound gratitude to my supervisor, Dr Nguyễn Thị Ngọc Quỳnh, who gave me enthusiastic instructions, precious support and critical feedback on the construction of the study She has given me great opportunities to gain the very first, yet valuable hands – on experiences in this realm of language testing Her guidance has always been one of decisive factors in the completion of this thesis My sincere thankfulness to Dr Nathan Carr for willingly assisting me in every stage of conducting this paper and encouraging me to delve further into this intriguing field He has inculcated in me the habit of meticulousness I am also grateful for his comments and direction I would also like to articulate my gratitude to my thesis examiners, Dr Dương Thu Mai and MA Ngô Xuân Minh If it weren’t for their reading, comments and evaluation on my progress reports, my thesis would not have been completed My deep sense of thanks to all the teachers and seniors at Hanoi Amsterdam High School, who participated in this research, for their cooperation Last but not least, my heartfelt thanks to my family and friends I am extremely grateful to my mother for her continued encouragement and support, especially in my darkest moments I would also like to appreciate Ms Bùi Thiện Sao for her assisstance with the theoretical knowledge and data analysis throughout the research i Abstract This paper is primarily conducted to evaluate the test reliability and the quality of test questions in Sample Vietnam National High School English Examination (VNHSEE) 2017 The test is categorized as both a norm – referenced placement test and criterion – referenced achievement test Classical Test Theory (CTT) was applied to examine item analysis and estimate reliability coefficients of the sample test Pilot testing was carried out with a convenience sample of 200 grade – 12 students at Hanoi – Amsterdam High School Subsequently, descriptive analysis, item analysis, reliability coefficient computation and distractor analysis of the sample test were carried out With respect to the difficulty and discrimination level, VNHSEE 2017 was rather easy to the sample group As a norm – referenced test serving the purpose of placement, the sample test generally separated selected examinees rather effectively However, with the aim of an achievement test, VNHSEE 2017 distinguished low – end students better than high – end students In addition, the test achieved notably high internal reliability, which illustrates its consistency of scoring and classification The amendments to test format could be attributable to the enhancement of its reliability and dependability coefficients Overall, Sample VNHSEE 2017 possessed good quality of test questions However, distractor analysis results highlighted some problematic distractors in need of revision Item design was reported to be the main culprit of implausible options Discussions and suggestions for improvement regarding specific cases were discretely presented below In conclusion, VNHSEE 2017 with revised test format have directly influenced English teaching and learning process, with its inevitable washback effects Some implications drawn from current findings, hopefully, propose some suggestions for developing test quality and improving English pedagogical methodologies ii Table of Contents Acknowledgements i Abstract ii Table of Contents iii List of Tables and Figures vi List of Acronyms and Abbreviations vii CHAPTER 1: INTRODUCTION 1.1 Overview 1.1.1 Vietnam National High School Examination 1.1.2 Vietnam National High School English Examination 2017 1.2 Problem Statement and Rationale for the Study 1.3 Significance of the Study 1.4 Objectives of the Study and Research Questions 1.5 Organization of the Study CHAPTER 2: LITERATURE REVIEW 2.1 Multiple Choice Questions 2.1.1 Definition and Use of Multiple Choice Questions 2.2 Classical Test Theory 11 2.3 Item Analysis in Classical Test Theory 12 2.3.1 Overview of Item Analysis for Selected Response Tasks 12 iii 2.3.2 Item Analysis and Bachman & Palmer’s Qualities of Test Usefulness Framework (1996) 13 2.3.2.1 Bachman & Palmer’s Qualities of Test Usefulness Framework (1996) 13 2.3.2.2 Item Analysis and Bachman & Palmer’s Qualities of Test Usefulness Framework (1996) 17 2.3.3 Norm – referenced Testing Item Analysis 18 2.3.3.1 Item Difficulty 18 2.3.3.2 Item Discrimination 18 2.3.3.3 Reliabity 20 2.3.4 Criterion – referenced Testing Item Analysis 22 2.3.4.1 Item Difficulty 22 2.3.4.2 Item Discrimination 22 2.3.4.3 Dependability 24 2.3.5 Distractor Analysis 25 CHAPTER 3: RESEARCH METHODOLOGY 27 3.1 Research Setting 27 3.2 Participants 27 3.3 Data Collection 28 3.3.1 Data Collection Instrument 28 3.3.2 Data Collection Procedure 28 3.4 Data Analysis 29 3.4.1 Reasons to Choose Classical Test Theory (CTT) 29 3.4.2 Data Analysis Procedure 31 3.4.2.1 Research Question 1: What is the reliability of the VNHSEE 2017? 31 iv 3.4.2.2 Research Question 2: Which distractors in Sample VNHSEE 2017 are in need of revision? 34 CHAPTER 4: RESULTS AND DISCUSSION 35 4.1 Descriptive Statistics 35 4.2 Reliability of Sample VNHSEE 2017 36 4.2.1 Item Parameters 36 4.2.1.1 Norm – referenced Testing (NRT) Item Parameters 36 4.2.1.2 Criterion – referenced Testing (CRT) Item Parameters 39 4.2.2 Reliability of Sample VNHSEE 2017 44 4.2.3 Dependability of Sample VNHSEE 2017 45 4.3 Distractor Analysis 47 CHAPTER 5: CONCLUSION 55 5.1 Summary of Major Findings 55 5.2 Implications 56 5.3 Limitations and Suggestions for Further Research 56 References 58 Appendices 63 v Carr, N T (2011) Designing and Analyzing Language Tests Oxford: Oxford Press University Choi, I C (1994) Content and Construct Validation of a Criterion-Referenced English Proficiency Test Language Institute, Seoul National University Choi, I C (2015) The Practical Foundations and Practical Applications of Language Assessment Department of English Language Education, Korea University Choi, I C (1989) Past, Present & Future of Language Testing University of Illinois at Urbana – Champaign Cooke, M Irby, D M Sullivan, W., & Ludmerer, K M (2006) American Medical Education 100 Years after the Flexner Report New England Journal of Medicine, 355(13), 1339-1344.
doi: 10.1056/NEJMra055445 Crocker, L & Algina, J (2008) Introduction to Classical & Modern Test Theory Mason, Ohio: Cengage Learning Cronbach, L J (1951) Coefficient Alpha and the Internal Structure of Tests Psychometrika, Vol 16, No De Champlain, A F (2010) A primer on classical test theory and item response theory for assessments in medical education Medical education, 44(1), 109-117 Downing, S M (2005) The effects of violating standard item writing principles on tests and students: the consequences of using flawed test items on achievement examinations in medical education Advances in Health Sciences Education, 10(2), 133-143 
 Educational Testing Service (2008) Reliability and Comparability of TOEFL iBT Scores TOEFL iBT Research, 1(3), – Embretson, S E., & Hershberger, S L (1999) (Eds.) The new rules of measurement Mahwah, NJ: Lawrence Erlbaum Associates Fowell, S L & Bligh, J G (1998) Recent developments in assessing medical students Postgraduate medical journal 74(867), 18 – 24 Retrieved from: US National Library of Medicine National Institutes of Health (PMCID: PMC2360796) 59 Freedle, R., & Kostin, I (1993) The prediction of TOEFL reading item difficulty: Implications for construct validity Language Testing, 10(2), 133 – 170 doi: 10.1177/026553229301000203 Freedle, R., & Kostin, I (1999) Does the text matter in a multiple-choice test of comprehension? The case for the construct validity of TOEFL’s minitalks Language Testing, 16(1), – 32 doi: 10.1177/026553229901600102 Glass, G V., & Stanley, J C (1970) Statistical methods in education and psychology Englewood Cliffs, NJ: Prentice-Hall Haladyna, T M & Downing, S M (1989) Validity of a Taxonomy of Multiple-Choice Item-Writing Rules Applied Measurement in Education, 2(1), 51-78 doi: 10.1207/s15324818ame0201_3 Haladyna, T M Downing, S M., & Rodriguez, M C (2002) A Review of Multiple Choice Item Writing Guidelines for Classroom Assessment Applied Measurement in Education, 15 (3), 309 – 334 Hambleton, R K., & Jones, R W (1993) Comparison of classical test theory and item response theory and their applications to test development Educational Measurement: Issues and Practice, 12(3), 38 – 47 Hambleton, R K Swaminathan, H., & Rogers, H J (1991) Fundamentals of item response theory Newbury Park, CA: Sage Kostin, I (2004) Research Report: Exploring Item Characteristics That Are Related to the Difficulty of TOEFL Dialogue Items Princeton, NJ: ETS McAlpine, M (2002) A Summary of Methods of Item Analysis (ed CAA Center, University of Luton) Retrieved from: http://caacentre.lboro.ac.uk/dldocs/BP2final.pdf McInerney, D M (2013) Education Psychology: Construct Learning (6th ed.) Australia: Pearson Mislevy, R J (2007) Validity in design Educational Researcher, 36(8), 463 – 469 Doi: 10.3102/0013189X07311660 Nasir, M (2014) Application of Classical Test Theory and Item Response Theory to 60 Analyze Multiple Choice Questions University of Calgary (Doctoral dissertation) Retrieved from: University of Calgary’s Dissertations and Theses’ Collections Ngo, T K T (2010) Evaluating the reliability and validity of an English achievement test for third-year non-major students at the University of Technology, Ho Chi Minh National University and some suggestions for changes (Unpublished MA thesis) University of Languages and International Studies, Hanoi, Vietnam Nitko, A J (2001) Educational assessment of students (3rd ed.) Upper Saddle River, NJ: Prentice Hall Oller J W., (1979) language Tests at School: A Pragmatic Approach London: Longman Palmer, N Bexley, E., & James, R (2011) Selection and Participation in Education University Retrieved from: Higher http://melbourne- cshe.unimelb.edu.au/ data/assets/pdf_file/0004/1490980/Selection_and_Participati on_in_Higher_Education.pdf Taylor, L., & Weir, C J (Eds.) (2012) IELTS Collected Papers 2: Research in reading and listening assessment Cambridge: Cambridge University Press Tran, H P (2014) Validating a National University Entrance Examination English test: a case from Vietnam (Doctoral dissertation) The University of Melbourne, Australia Tran, H P Griffin, P & Nguyen, C (2010) Validating the university entrance English test to Vietnam National University: A conceptual framework and methodology Procedia - Social and Behavioral Sciences, 2(2), 1295 – 1304 doi: 10.1016/j.sbspro.2010.03.190 Tran, T Q (2011) The content validity of the current English achievement test for second year non major students at Phuong Dong University (Unpublished Master thesis) University of Languages and International Studies, Hanoi, Vietnam United Nations Educational, Scientific and Cultural Organization (2015) The Transition from Secondary Education to Higher Education Case Studies from Asia and the Pacific France: UNESCO Vietnamnet (2017) 75% đăng ký đại học, thí sinh chọn thi khoa học xã hội áp đảo [75% of the candidates have registered for universities, the number of examinees 61 choosing Social subjects outnumbers that choosing Science subjects.] Retrieved from http://vietnamnet.vn/vn/giao-duc/tuyen-sinh/tuyen-sinh-dai-hoc-2017-thi-sinh-dangky-vao-dai-hoc-nhieu-hon-367797.html Ministry of Education and Training (2014) Decision No 3538: Approval of Vietnam National High School Examination from 2015 Hanoi: Ministry of Education and Training Vu, T H (2014) An evaluation of some aspects of the validity of a reading achievement test (the 3b end – of – term reading test) for second year mainstream students in the school year 2013 – 2014 at FELTE, ULIS, VNU (Unpublished Bachelor Thesis) University of Languages and International Studies, Hanoi, Vietnam Wang, L., Huang, X., & Schnell, J (2013) Using Burke’s dramatistic pentad to interpret Chinese “Gaokao” high stakes testing and stressing – Paralleled testing in the U.S as cross cultural context KOME – An international Journal of Pure Communication Inquiry, 1(2), 55 – 63 Hoffman, B (1967) Multiple-choice tests Physics Education 2, 247–51 Dudley, H A F (1973) Multiple-choice tests: time for a second look? Lancet 2, 195-196 Retrieved from: https://www.cambridge.org/core/services/aop-cambridge- core/content/view/142BCA6A2275694B08E741B74A1A7ED7/S003329170000249 Xa.pdf/div-class-title-psychiatristsandapos-education-in-psychology-jackdaw-orsponge-a-href-fn01-ref-type-fn-span-class-sup-1-span-a-div.pdf Wood, W B (1977) Heart and vascular systems : 1500 multiple choice questions and referenced answers (2nd ed.) Flushing, NY: Med Examination Publ Co Hughes, A (2003) Testing for language teachers Cambridge: Cambridge University Press Messick, S (1989) Meaning and values in test validation: The science and ethics of assessment Educational Researchers, 18(2), – 11 62 Appendices Appendix Sample Vietnam National High School English Examination 2017 I am Lê Hoàng Kim Khuê, from QH2013.F1.E2, University of Languages and International Studies, Vietnam National University I am conducting an investigation into “The quality of test questions in Sample Vietnam National High School English Examination 2017” Your participation in this pilot test administration would be of significant value to the final product of the research Your personal information is kept confidential Results of this test does not affect your academic performance Please complete the test as honestly as possible since that is the only way to ensure the significance and success of this research Thank you for your cooperation Sample Vietnam National High School English Examination 2017 consists of five pages Please ensure that you receive a complete paper The test should take no longer than 60 minutes to complete Dictionary, electronic devices and consultation are not allowed General information Name: Class: Have you done this test before? (Yes/No) 63 64 65 66 67 68 Appendix Content Distribution of the Sample NHSE 2017 Test Questions 1 Phonetic 11 Grammar 21 Reading 31 Vocabulary 41 Reading 2 Phonetic 12 Grammar 22 Vocabulary 32 Vocabulary 42 Reading 3 Phonetic 13 Vocabulary 23 Vocabulary 33 Vocabulary 43 Reading 4 Phonetic 14 Vocabulary 24 Vocabulary 34 Vocabulary 44 Vocabulary 5 Grammar 15 Vocabulary 25 Vocabulary 35 Vocabulary 45 Reading 6 Grammar 16 Vocabulary 26 Reading 36 Reading 46 Reading 7 Grammar 17 Vocabulary 27 Reading 37 Reading 47 Reading 8 Grammar 18 Vocabulary 28 Reading 38 Reading 48 Reading 9 Grammar 19 Vocabulary 29 Grammar 39 Reading 49 Reading 20 Reading 30 Grammar 40 Reading 50 Reading 10 Grammar 69 Appendix Item Difficulty (IF), Point Biserial (pb(r)), B – index and φ coefficient at cut Item scores of 80%, 60%, 50% and 40% of the Sample NHSE 2017 IF pb(r) λ = 80 λ = 60 λ = 50 λ =.40 B(.80) φ(.80) B(.60) φ(.60) B(.50) φ(.50) B(.40) φ(.40) 86 14 20 28 13 16 - 01 - 01 - 15 - 10 95 08 06 14 07 13 - 02 - 03 - 05 - 06 79 48 33 41 49 52 28 24 40 23 86 33 20 28 23 29 23 23 20 14 73 52 44 49 49 48 37 29 16 08 75 60 48 55 55 55 59 48 44 24 79 62 41 51 57 61 60 52 75 44 94 35 12 26 24 44 25 37 02 02 76 44 35 42 37 38 36 30 37 20 10 92 36 16 28 26 40 22 28 18 15 11 65 39 36 38 39 35 28 20 34 17 12 64 62 57 59 58 52 50 37 68 33 13 81 64 39 50 57 63 71 63 51 31 14 87 41 25 37 23 29 29 31 57 40 15 80 65 40 50 58 62 81 71 58 34 16 75 55 39 44 51 51 59 48 79 43 17 85 41 19 26 36 44 47 46 55 37 18 65 42 43 45 35 32 27 20 24 12 19 83 53 28 37 46 52 48 45 70 44 20 94 39 13 27 23 41 21 29 55 53 21 91 32 13 22 23 34 30 37 26 21 22 79 50 40 49 35 37 43 37 39 23 70 23 84 42 30 40 26 30 29 28 45 28 24 53 44 40 40 41 35 45 32 56 27 25 55 35 34 34 33 28 36 25 31 15 26 57 45 51 51 36 31 30 22 34 16 27 83 40 19 25 28 32 37 34 88 56 28 36 24 24 25 24 22 22 16 12 06 29 80 39 28 35 29 31 41 35 40 24 30 80 54 26 33 50 54 57 49 85 50 31 82 51 33 43 37 42 56 51 52 32 32 91 46 19 32 28 42 30 37 44 36 33 63 44 32 33 55 49 62 45 58 29 34 68 64 56 60 59 54 63 48 72 37 35 57 52 48 48 49 43 50 35 60 29 36 89 38 15 23 25 35 40 45 50 38 37 82 55 38 49 50 56 47 43 25 15 38 73 50 44 49 52 51 41 33 16 08 39 61 56 51 53 57 50 51 36 55 27 40 66 25 20 21 24 22 33 24 26 13 41 59 41 40 40 46 40 48 34 36 17 42 83 58 28 37 54 62 64 60 79 49 43 66 51 48 51 40 37 57 42 70 35 44 76 46 37 44 40 41 49 40 54 30 45 73 66 50 56 63 61 69 55 78 42 46 65 47 45 47 33 30 47 35 60 30 47 77 52 35 42 44 45 54 45 82 46 48 71 68 51 56 70 66 70 54 75 39 49 80 64 37 47 67 72 65 58 76 45 50 57 47 48 48 46 40 42 30 51 25 71 Appendix Distractor Analysis Resulst of the VNHSEE 2017 Item Response Frequencies Option Point – biserials %A %B %C %D A pb(r) 01 86 11 01 - 03 14 - 09 - 02 95 03 01 01 08 - 15 05 03 08 79 08 05 - 26 48 - 26 - 21 04 06 86 05 - 18 - 10 33 - 27 05 73 13 08 - 18 52 - 24 - 33 01 12 75 09 - 07 - 39 60 - 35 06 04 09 79 - 29 - 27 - 39 62 02 03 94 02 - 18 - 24 35 - 18 02 76 13 10 05 44 - 10 - 53 10 02 02 92 04 - 15 - 09 36 - 31 11 04 28 02 65 - 09 - 31 - 06 39 12 03 07 27 64 - 21 - 40 - 38 62 13 81 06 05 09 64 - 41 - 18 - 43 14 87 08 03 02 41 - 27 - 22 - 18 15 80 08 06 06 65 - 46 - 34 - 23 16 08 06 09 75 - 24 - 06 - 52 55 17 04 09 01 85 - 38 - 16 - 16 41 18 05 65 18 09 - 08 42 - 08 - 44 19 83 10 04 04 53 - 37 - 12 -.35 20 03 04 94 00 - 27 - 27 39 N/A 21 01 91 04 05 - 15 32 - 07 - 31 22 05 05 10 79 - 30 - 10 - 34 50 23 03 03 84 09 - 27 03 42 - 33 24 53 25 08 15 44 - 06 - 23 - 37 72 B pb(r) C pb(r) D pb(r) 25 14 55 21 09 - 48 35 11 - 14 26 36 57 05 02 - 32 45 - 20 - 20 27 04 09 05 83 - 32 - 17 - 20 40 28 02 03 36 58 - 21 - 30 24 - 02 29 01 02 80 18 - 17 - 07 39 - 35 30 80 14 02 05 54 - 37 - 16 - 32 31 82 09 04 05 51 - 33 - 18 - 29 32 91 02 07 00 46 - 12 - 46 N/A 33 15 10 63 12 - 28 - 20 44 - 18 34 10 15 68 06 - 40 - 23 64 - 30 35 02 15 25 57 - 20 - 21 - 31 52 36 02 03 89 05 - 30 - 22 38 - 06 37 03 10 82 05 - 10 - 45 55 - 25 38 10 09 06 73 - 18 - 37 - 13 50 39 22 61 11 05 - 42 56 - 17 - 17 40 02 19 10 66 01 03 - 35 25 41 05 23 59 13 - 06 - 06 41 - 47 42 08 83 01 08 - 38 58 - 16 - 34 43 66 19 04 09 51 - 27 - 08 - 33 44 04 13 06 76 - 23 - 28 - 24 46 45 73 16 09 01 66 - 38 - 48 N/A 46 17 07 65 08 - 20 - 33 47 - 13 47 07 13 77 03 - 46 - 25 52 - 05 48 09 71 10 09 - 34 68 - 32 - 33 49 02 10 80 09 - 14 - 47 64 - 36 50 17 10 18 57 - 43 - 22 - 02 47 Note: Correct answers are underlined 73 ... evaluating the reliability of Sample Vietnam National High School English Examination 2017 (VNHSEE 2017) by investigating the quality of its test questions 1.1.1 Vietnam National High School Examination. .. might be in need of In short, this research is designed as an investigation into the quality of test questions in the Vietnam National High School English Examination 2017 1.3 Significance of the. .. Vietnam National High School English Examination 2017 In October, 2016, MOET published the Sample Vietnam National High School English Examination (VNHSEE) 2017 with some changes in test format,