The reliability of the english level a written test contents at the foreign language center of cantho in service university

VIETNAM NATIONAL UNIVERSITY – HOCHIMINH CITY UNIVERSITY OF SOCIAL SCIENCES AND HUMANITIES  THE RELIABILITY OF THE ENGLISH LEVEL-A WRITTEN TEST CONTENTS AT THE FOREIGN LANGUAGE CENTER OF CANTHO IN-SERVICE UNIVERSITY A thesis submitted in partial fulfillment of the requirements for the degree of Master of Arts (TESOL) Submitted by Phạm Thị Bích Ngân Supervisor: LÊ THỊ THANH THU, Ed D Hochiminh City, June 2008 CERTIFICATE OF ORIGINALITY I hereby certify that the thesis entitled “THE RELIABILITY OF THE ENGLISH LEVELA WRITTEN TEST CONTENTS AT THE FOREIGN LANGUAGE CENTER OF CANTHO IN-SERVICE UNIVERSITY” is my own work This thesis has not previously been submitted for the award of any degree or diploma in any other institutions Hochiminh City, June 18th, 2008 Phạm Thị Bích Ngân i RETENTION AND USE OF THE THESIS I hereby state that I, PHẠM THỊ BÍCH NGÂN, being the candidate for the degree of Master of TESOL, accept the requirements of the University relating to the retention and use of Master’s Theses deposited in the Library In terms of these conditions, I agree that the original of my thesis deposited in the Library should be accessible for purposes of study and research, in accordance with the normal conditions established by the Library for the care, loan or reproduction of theses Hochiminh city, June 18th, 2008 PHẠM THỊ BICH NGÂN ii ACKNOWLEDGEMENTS I would like first to express my deepest gratitude to my thesis supervisor, Dr Lê Thị Thanh Thu for her valuable and critical comments, whole-hearted guidance, support, and a lot of her time consuming to proofread and give valuable feedback in the preparation and completion of this thesis I am especially grateful to the organizers of this master course, Dr Trương Thị Kim Chuyên, Head of the Department of Post Graduate Studies of Hochiminh City University of Social Sciences and Humanities, Dr Nguyễn Thị Kiều Thu, Head of the Department of English Linguistics and Literature of the university, and their staff members My special thanks also go to Dr Vũ Thị Phương Anh, who has helped me have the idea to conduct this research Without her instructions and encouragement at the initial stage, I might not have carried out this study I would also like to send my greatest thanks to all of the members of the Board of managers of Cantho In-service University and my colleagues here, especially the members of English Department They have given me good conditions and assistances to complete the thesis as well as the post graduate course Last, but not least, my deepest thanks go to my husband and little daughter for their invaluable support, encouragement, and sacrifice during my long absence from home with great sympathy Their love and sympathy have made a great contribution to the completion of my post-graduate course and the thesis as well Hochiminh, June 2008 Phạm Thị Bích Ngân iii LIST OF ABBREVIATIONS CTFLC : the Foreign Language Center of Cantho In-service University CTIU : Cantho In-service University MOET : Ministry of Education and Training CED : Continuing Education Department FLC : Foreign Language Center iv ABSTRACT This research was conducted to investigate whether the English Level-A written test versions, which are issued by Continuing Education Department (CED) belonging to Ministry of Education and Training (MOET) and supplied to some foreign language centers (FLCs) throughout Vietnam, of which the foreign language center of Cantho Inservice University (CTFLC) is a case, are reliable As for the fact that among the four factors affecting the reliability of a test: administration procedures, scoring procedures, candidates and the test content itself, the contents of the English Level-A written test versions, in the particular case of CTFLC, were emerged as the most problematic ones To achieve this aim, the contents of the two randomly chosen Level-A written test versions among the ones administered in 2006 and 2007, named as Test-06 and Test-07, were analyzed and evaluated, focused successively on the four factors constituting the reliability of test content: the test (1) layout, (2) rubrics, (3) format, and (4) items These analyses and evaluations were based on both the theories drawn from the in-depth studies of famous specialists in testing field and MOET’s requirements on the course objectives Then each of the above four mentioned factors of Test-06 was compared and contrasted with that of Test-07 to see whether they were in equivalent level of difficulty and could be used interchangeably As for the nature of the study, an evaluation study, it involved basically with a qualitative method The results of the study pointed out that although the writer(s) of these written test versions paid certain attention to theories when designing them, they might have paid less attention to MOET’s requirements on Level-A course objectives and test format, which led to the fact that all the four factors constituting test content reliability: the (1) layout, (2) rubrics, (3) format, and (4) items of both Test-06 and Test-07 were somewhat problematic and needed improving Particularly, the layout of Test-06 as well as of Test-07 did not look very attractive as the items in some sections in these two test versions were closely typed to one another, so they seemed to be “eye-confusing” or illegible; there were also some items in these test versions were typed in such a way that all parts of the items did not v appear together, but were divided across two pages, which, to Brown and Hudson (2002), was not a good display Though the test rubrics, especially test instructions, of some subtests were simple, clear and easily understandable, there were still some subtest instructions which were not quite easy for elementary candidates to follow as these instructions were constructed in the language structures above the level being tested, and were rather inexplicit or vague Also, neither Test-06 format nor Test-07 format completely matched MOET’s required format, so some differences in the format between the two test versions appeared; as a result, the unbalance in weighting between Test-06 and Test-07 occurred Finally, some faults in items selection and construction led to the fact that quite a lot of test items constructed in the grammar structures that were not included in the course syllabus appeared in Test-06 and some in Test-07, which caused the different levels of difficulty or complexity between the two test versions Consequently, the reliability of each test version and the equivalent level of difficulty or complexity between Test-06 and Test-07 were hardly found Some suggestions to test writers to improve the reliability for the upcoming LevelA written test versions were therefore given out with the hope of giving a small contribution to enhance the reliability of the English Level-A tests at CTFLC in the future in particular and in other language centers and schools in general vi TABLE OF CONTENTS CERTIFICATE OF ORIGINALITY i RETENTION AND USE OF THE THESIS ii ACKNOWLEDGEMENTS iii LIST OF ABBREVIATIONS iv ABSTRACT v CONTENTS vii LIST OF TABLES xi LIST OF APPENDICES xii Chapter 1: Introduction 1.1 Problem identification 1.2 Purpose of the study 1.3 Research questions 1.4 Limitation 1.5 English Level-A Tests 1.5.1 Objectives 1.5.2 Test format 1.5.3 Preparation for a Level-A test 1.5.4 Administration 1.5.5 Marking 1.5.6 Candidates 1.5.7 Examiners 10 1.6 Significance of the study 10 1.7 Organization of the study 10 vii Chapter 2: Literature Review 12 2.1 Language testing 12 2.1.1 The importance of testing in language teaching and learning 12 2.1.2 Types of tests 13 2.2 Test qualities 16 2.2.1 Validity 17 2.2.2 Reliability 20 2.2.3 Factors interfering test reliability and their solutions 21 2.2.3.1 Meaningful variance 21 2.2.3.2 Error variance (or measurement error) 22 2.2.3.2.1 Administration procedures 22 2.2.3.2.2 Solutions to administration procedures 22 2.2.3.2.3 Scoring procedures 24 2.2.3.2.4 Solutions to scoring procedures 26 2.2.3.2.5 The test itself 31 2.2.3.2.6 Solutions to the test 34 2.2.3.2.7 Candidates 36 2.2.3.2.8 Solutions to candidates 36 2.2.4 The interaction between reliability and validity 37 2.3 Conclusion 38 Chapter 3: Methodology 43 3.1 Research questions 43 3.2 Methodology rationales 43 3.3 Research methodology 45 3.3.1 Published advice and recommendations on testing field 45 3.3.2 Guiding documents from MOET 46 3.3.2.1 English Level-A course objectives 46 3.3.2.2 Required format for English Level-A written test versions 47 3.3.3 Level-A written test versions’ contents to be examined 49 viii Chapter 4: Data analysis and findings 51 Evaluation on test layout 51 4.1.1 Test-06 layout 52 4.1.2 Test-07 layout 52 4.1.3 Test-06 versus Test-07 53 4.1.3.1 Similarities 53 4.1.3.2 Differences 53 Evaluation on test rubrics 55 4.2.1 Test-06 rubrics 55 4.2.2 Test-07 rubrics 57 4.2.3 Test-6 versus Test-07 57 4.2.3.1 Similarities 58 4.2.3.2 Differences 58 Evaluation on test format 59 4.3.1 Test-06 format 59 4.3.2 Test-07 format 61 4.3.3 Test-06 versus Test-07 62 4.3.3.1 Similarities 63 4.3.3.2 Differences 63 Evaluation on test items 69 4.4.1 Test-06 items 71 4.4.2 Test-07 items 75 4.4.3 Test-06 versus Test-07 79 4.4.3.1 Similarities 80 4.4.3.2 Differences 80 Summary 81 4.1 4.2 4.3 4.4 4.5 ix APPENDIX 3.1B: Grammatical Summary for Level-A course 101 102 103 APPENDIX 3.1C: MOET’s requirements on the contents of the English test versions of all levels 104 APPENDIX 3.1D: MOET’s Level-A required format 105 APPENDIX 3.2: Test-06 106 107 108 109 APPENDIX 3.3: Test-07 110 111 112 113 114 A Section items d is believed c is believing d we’re decorating c were decorated c have realized d had realized a to test c to have tested not b to be testing d to have been tested This machine ought … before it left the factory but it apparently was a realized b realize take my wallet with me Only after I had reached the cinema did I … that I had forgotten to b was decorated a decorated yesterday because of the holiday Every shop, no matter whether it was big or small, … beautifully b believed a believes sends it to relatives in a foreign country Nobody is quite sure what he does with the money, but it … that she I am directed to thank you … your letter of 6th June and to inform you that the matter will be investigated contents √ √ Passive voice √ √ √ √ √ Complex sentence Unreal conditional sentence APPENDIX 4.1: Items that broke MOET’s requirements in PAPER 1, Test-06 √ Subject and verb inversion Future perfect tense 115 B A Section 21 14 13 items d would have been c have been d were b will be d would have being polluted c are being polluted b of c as d into According to the passage, things truly happening in medicine and science today are … things that happen in some fairy tales a to Rarely had we have such a fine opportunity … this one b will be polluting a will have been polluted world … by all the rubbish poured into them By the end of the century, it is likely that the oceans of the c would a are but this seems a pious wish If motorists more careful, there would be fewer accidents, b will be a were house easily If you hadn’t forgotten your key, you … able to get into the contents √ Passive voice √ √ √ √ √ Complex sentence √ √ Unreal conditiona l sentence APPENDIX 4.1: Items that broke MOET’s requirements in PAPER 1, Test-06 √ Subject and verb inversion √ Future perfect tense ... rare inappropriateness and/or circumlocution Almost all grammatical patterns inaccurate Frequent grammatical patterns inaccuracies Some grammatical patterns inaccuracies Almost no grammatical patterns... such as the course objectives, the English Level- A test format, the preparation for a Level- A test, the test administration, the marking, the candidates’ background, as well as the examiners at. .. educational programs; and (2) as indicators of abilities or attributes that are of interest in research on language, language acquisition, and language teaching” (Bachman, 1997:54) Also, to McNamara