Achievement tests are of extensive use at different levels of education due to their distinguished characteristics. Researchers define the notion of achievement tests in various ways
VIETNAM NATIONAL UNIVERSITY, HANOI COLLEGE OF FOREIGN LANGUAGES DEPARTMENT OF POSTGRADUATE STUDIES NGUYEN THI VIET HA A STUDY ON THE RELIABILITY OF THE FINAL ACHIEVEMENT COMPUTER-BASED MCQS TEST 1 FOR THE 4 TH SEMESTER NON - ENGLISH MAJORS AT HANOI UNIVERSITY OF BUSINESS AND TECHNOLOGY (đánh giá độ tin cậy của bài thi trắc nghiệm THứ NHấT TRÊN MáY TíNH cuối kỳ 4 dành cho sinh viên năm thứ hai không chuyên ngành tiếng anh trờng đại học kinh doanh và công nghệ hà nội) Minor Programme Thesis Field: Methodology Code: 601410 HANOI, 2008 1 VIETNAM NATIONAL UNIVERSITY, HANOI COLLEGE OF FOREIGN LANGUAGES DEPARTMENT OF POSTGRADUATE STUDIES NGUYễN THị VIệT Hà A STUDY ON THE RELIABILITY OF THE FINAL ACHIEVEMENT COMPUTER-BASED MCQS TEST 1 FOR THE 4 TH SEMESTER NON - ENGLISH MAJORS AT HANOI UNIVERSITY OF BUSINESS AND TECHNOLOGY (đánh giá độ tin cậy của bài thi trắc nghiệm THứ NHấT TrÊN MáY TíNH cuối kỳ 4 dành cho sinh viên năm thứ hai không chuyên ngành tiếng anh trờng đại học kinh doanh và công nghệ hà nội) Minor Programme Thesis Field: Methodology Code: 601410 Supervisor: Nguyễn Thu Hiền. M.A HANOI, 2008 2 VIETNAM NATIONAL UNIVERSITY, HANOI COLLEGE OF FOREIGN LANGUAGES DEPARTMENT OF POSTGRADUATE STUDIES CANDIDATE S STATEMENT’ I hereby state that I: Nguyen Thi Viet Ha, Class 14A, being a candidate for the degree of Master of Arts (TEFL) accept the requirements of the College relating to the retention and use of Master of Arts Thesis deposited in the library. In terms of these conditions, I agree that the origin of my thesis deposited in the library should be accessible for the purposes of study and research, in accordance with the normal conditions established by the librarian for the care, loan or reproduction of the thesis. Signature Date i ACKNOWLEDGMENTS In the completion of this thesis, I have received a great deal of backup. Of primary importance has been the role of my supervisor, Ms. Nguyen Thu Hien, M.A, Teacher of Department of English and American Languages & Cultures, College of Foreign Language, Vietnam National University, Hanoi. I am deeply grateful to her for her precious guidance, enthusiastic encouragement and invaluable critical feedback. Without her dedicated support and correction, this thesis could not have been completed. I am deeply indebted to my dear teacher, Mr. Vu Van Phuc, M.A, Head of Testing Center, College of Foreign Languages, VNU, who provided me with a lot of useful suggestion and assistance towards my study. I would also like to express my sincere thanks to all teachers and colleagues in English Department, HUBT, for their help in conducting the survey, sharing opinions and making suggestions to the study. Especially, my thanks go to Ms. Le Thi Kieu Oanh, Assistant of English Department, HUBT for her willingness to offer test score data. I wish to show my special thanks to the students of K11 at Hanoi University of Business and Technology who have actively participated in the survey Finally, it is my great pleasure to acknowledge my gratitude to beloved members of my family, especially my husband who constantly encouraged and helped me with my thesis. ii ABSTRACT The main aim of this minor thesis is to evaluate the reliability of the final Achievement Computer-based MCQs Test 1 for the 4th semester non-English majors at Hanoi University of Business and Technology. In order to achieve this aim, a combination of both qualitative and quantitative research methods were adopted. The findings indicate that there is a certain degree of unreliability in the final achievement computer-based MCQs test1 and there are two main factors that cause the unreliability including test item quality and test- takers performance. ’ Having carefully considered a thorough analysis of the collected data, the author made some suggestions in order to improve the quality of the final achievement test and the MCQs test 1 for the non-majors of English in the 4 th semester in Hanoi University of Business and Technology. Firstly, the test objectives, sections and skill weight should be adjusted to be more compatible with the course objectives and the syllabus. Secondly, a testing committee should be set up for the construction and development of a multi choice item bank including test items which are of good p-value and discrimination value. iii LIST OF ABBRIVIATIONS 1. CBT: Computer-based testing 2. HUBT: Hanoi University of Business and Technology 3. MC: Multi choice 4. MCQs: Multi choice questions 5. ML Pre- : Market Leader Pre-intermediate 6. KD: Kuder- Richardson 7. SD: Standard deviation iv LIST OF TABLES AND CHARTS 1. Table 1 Types of tests 2. Table 2 Scoring format for each semester 3. Table 3 The syllabus for 4 th semester (for non English majors– ) 4. Table 4 Time allocation for language skills and sections 5. Table 5 Specification grid for the final computer-based MCQs test 1 6. Table 6 Main points in the grammar section 7. Table 7 Main points in the vocabulary section 8. Table 8 Topics in reading section 9. Table 9 Items in the functional language sections 10. Table 10: Test reliability coefficient 10. Table 11: p-value of items in 4 sections 11. Table 12: Discrimination value of items in 4 sections 12. Table 13: Number of test items with acceptable p-value and discrimination value in 4 sections 13. Table 14: Suggested scoring format 14. Table 15: Proposed test specifications 12. Chart 1 Students response on test content’ 13. Chart 2 Students response on item discrimination value’ 14. Chart 3 Students response on time length ’ 15. Chart 4 Students response arbitrariness ’ 16. Chart 5 Students response on relation between test score and their ’ achievement v TABLE OF CONTENT CANDIDATE S STATEMENT’ i ACKNOWLEDGEMENT ii ABSTRACT iii LIST OF ABBREVIATION iv LIST OF TABLES AND CHARTS v TABLE OF CONTENT vi Chapter 1: INTRODUCTION 1 1.1. Rationale for the study 1 1.2. Aims and research questions 2 1.3. Theoretical and practical significance of the study 2 1.4. Scope of the study 2 1.5. Method of the study 2 1.6. Organization of the paper 3 Chapter 2: LITERATURE REVIEW 4 2.1. Language testing 4 2.1.1. What is a language test? 4 2.1.2. The purposes of language tests 4 2.1.3. Types of language tests 5 2.1.4. Criteria of a good language test 5 2.2. Achievement test 6 2.2.1. Definition 6 2.2.2. Types of achievement test 6 2.2.3. Considerations in final achievement test construction 7 2.3. MCQs test 7 2.3.1. Definition 7 2.3.2. Benefits of MCQs test 8 2.3.3. Limitations of MCQs test 10 2.3.4. Principles on designing a good MCQs test 11 2.4. Reliability of a test 11 2.4.1. Definition 11 2.4.2. Methods for test reliability estimate 12 2.4.3. Measures to improve test reliability 15 2.5. Summary 15 Chapter 3: The Context of the Study 16 3.1. The current English learning, teaching and testing situation at HUBT 16 3.2. The course objectives, syllabus and materials used for the second non- majors of English in Semester 4. 17 3.2.1. The course objectives 17 3.2.2. Business English syllabus 17 3.2.3. The course book 19 3.2.4. Specification grid for the final achievement Computer-based MCQs test in Semester 4. 19 Chapter 4: Methodology 21 vi 4.1. Participants 21 4.2. Data collection instruments 21 4.3. Data collection procedure 21 4.4. Data analysis procedure 22 Chapter 5: RESULTS AND DISCUSSIONS 23 5.1. The compatibility of the objectives, content and skill weight format of the final achievement computer-based MCQ test 1 for 4 th semester with the course objectives and the syllabus 23 5.1.1 The test objectives and the course objectives 23 5.1.2. The test item content in four sections and the syllabus content 24 5.1.3. The skill weight format in the test and the syllabus 26 5.2. The reliability of the final achievement test 27 5.2.1. Reliability coefficient 27 5.2.2. Item difficulty and discrimination value 27 5.3. The attitude of students towards the MCQs test 1 29 5.4. Pedagogical implications and suggestions on improvements of the existing final achievement computer-based MCQs test 1 for the non- English majors at HUBT. 34 5.5. Summary 38 Chapter 6: CONCLUSION 39 6.1. Summary of the findings 39 6.2. Limitations of the study 40 6.3. Suggestions for further study 40 REFERENCES 41 APPENDICES I APPENDIX 1 Grammar, Reading, Vocabulary and Functional language check list II APPENDIX 2 Survey questionnaire (for students at HUBT) IV APPENDIX 3 Students test scores’ VII APPENDIX 4 Item analysis of the final achievement computer-based MCQs test 1- 150 items, 349 examinees XII APPENDIX 5 Item indices of the final achievement computer-based MCQs test 1 XVII vii Chapter 1: Introduction 1.1. Rationale of the study Testing plays a very important role in teaching and learning process. Testing is one form of measurement which is used to point out strengths and weaknesses in the learned abilities of the students. Through testing, especially tests scores we may discover the performance of given students and of teachers. As far as students are concerned, test scores reveal what they have achieved after a learning period. As for teachers, test scores indicate what they have taught to their students. Based on test results, we may make improvement in teaching, learning and testing for better instructional effectiveness. Another reason for the selection of testing a matter of study lies in the fact that the current language testing at Hanoi University of Business and Technology (HUBT) has been under a lot of controversy among students and teachers. Testing is mainly carried out in the form of two objective tests on computers (named test 1 and test 2) which are administered at the end of each semester. The scores that a student gets on these tests are the main indicators of his or her performance during the whole semester. There are different comments on the results of these tests, especially the test 1 for the second-year non-English majors. Some subject teachers claim that these tests do not truly reflect the students’ language competence. Others say that these tests are appropriate to what students have learnt in class and compatible with the course objectives and therefore reliable. Also, among the students, do opposite ideas exist. Many think that these tests are more difficult than what they have learnt and studied for the exam, others say that these test items are easy and relevant to what they have been taught. Therefore finding out whether the tests are closely related with what the students have been learnt and what the teachers have taught, also, whether these tests are of reliability is indispensable. For the two reasons mentioned above, the author would like to undertake this study entitled A study on the reliability of the final achievement Computer-based MCQs Test“ 1 for the 4th semester non-English majors at Hanoi University of Business and Technology” with the intention to examine rumors about this test. In addition, the author hopes that the study results help to raise awareness among teachers as well as those who are interested in this field. At the same time, study results, in some extent, can be applied to improve the current testing situation in HUBT. 1.2. Aims and research questions 1 [...]... final achievement Computer-based MCQs test 1 in the 4 th semester for the second-year non- English majors at HUBT 1. 4 Theoretical and practical significance of the study Theoretically, the study proves that testing is crucial in order to measure and evaluate the quality of learning and teaching Also, test reliability is one of the most important criteria for the evaluation of a test Practically, the study. .. (Hughes, 19 89) Among the above ways of estimating reliability, test- retest and parallel methods require at least two test administrations while the inter-rater and internal consistency methods need only a single administration For the reason of convenience and satisfaction, KD20 and KD 21 are often chosen more than the others and are considered the two most common formulae (Alderson J.S et al., 19 95) Concerning... specification has to include information on the length, the structure of each part of the test, the type of materials with which the candidates will have to engage, the source of materials, the extent to which authentic materials may be altered, the response format and how responses are scored They are usually written before the tests and then the test is written on the basis of the specifications After the. .. test scores and test items analysis, quantitative method is used 1. 6 Organization of the paper The study is composed of 6 chapters Chapter 1- Introduction briefly states the rationale, aims and research questions, scope of the study, theoretical and practical significance of the study, method of the study and organization of the paper 2 Chapter 2- Literature review discusses relevant theories of language... research questions are set for exploration: 1 Are the objectives, content and skill weight format of the final achievement computer-based MCQs test 1 compatible with the course objectives, the syllabus content and skill weight format ? 2 To what extend is the test 1 reliable? 3 What is the student’s attitude towards the final achievement Computer-based MCQs test 1? 1. 3 Scope of the study The existing final. .. construction are dealt with in section 2.3 The final Section - 2.4 is concerned with test reliability, methods for estimating test reliability, and ways to make language tests more reliable 15 Chapter 3: The Context of the Study 3 .1 The current English learning, teaching and testing situation at HUBT There are over 15 00 second-year non- majors of English at HUBT English is their required subject for foreign... correlation of the ratings of one rater with those of another The following formula is used in calculating reliability: nrA,B Rtt = (Henning, 19 87) 1 + (n -1 ) r A, B Rtt: inter-rater reliability n: the number of rater who combines estimates from the final mark for the examiner rA,B: the correlation between the raters, or the average correlation among all rater if there are more than two * Internal consistency... given a score immediately Second, the computer can deliver tests that are tailored to the particular abilities of the candidate This type of test, as also called computer-adaptive test, can provide far more information about the testees’ ability 2.3.3 Limitations of MCQs tests: Despite the fact that MCQs tests bring lots of benefits, especially, to test administrators, there are several problems associated... of language testing, final achievement test, Computer-based MCQ tests and test reliability Chapter 3- The context of the study deals with English learning, teaching and testing situation at HUBT, course book, syllabus and check list for the test Chapter 4- Methodology presents participants, data collection instruments, data collection and data analysis procedure Chapter 5– Results and Discussions presents... requirement for designing a final achievement test Testers should avoid basing the test on their own teaching rather than on the syllabus or course book in order to establish and maintain a certain standard In addition, Mc Namara (2000) stated that test writers should draw out a test specification before writing a test Test specification is resulted from the process of designing test content and test method Test