Evaluating the validity of the final achievement test for second-year non-major students at Electronic-Electrical Engineering Department, Namdinh University of Technology Education

VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES FACULTY OF POST – GRADUATE STUDIES SUBMITTED BY: TRẦN THỊ THU HƯƠNG A thesis submitted in partial fulfillment of the requirements for the degree of Master of Arts EVALUATING THE VALIDITY OF THE FINAL ACHIEVEMENT TEST FOR SECOND – YEAR NON – MAJOR STUDENTS AT ELECTRONIC – ELECTRICAL ENGINEERING DEPARTMENT, NAM DINH UNIVERSITY OF TECHNOLOGY EDUCATION (Đánh giá độ giá trị kiểm tra cuối kỳ cho sinh viên không chuyên tiếng Anh năm thứ hai khoa Điện – Điện tử, Trường Đại học Sư phạm Kỹ thuật Nam Định) M.A MINOR THESIS Field: Language Teaching Methodology Code: 60 14 10 HANOI, 2011 VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES FACULTY OF POST – GRADUATE STUDIES SUBMITTED BY: TRẦN THỊ THU HƯƠNG A thesis submitted in partial fulfillment of the requirements for the degree of Master of Arts EVALUATING THE VALIDITY OF THE FINAL ACHIEVEMENT TEST FOR SECOND – YEAR NON – MAJOR STUDENTS AT ELECTRONIC – ELECTRICAL ENGINEERING DEPARTMENT, NAM DINH UNIVERSITY OF TECHNOLOGY EDUCATION (Đánh giá độ giá trị kiểm tra cuối kỳ cho sinh viên không chuyên tiếng Anh năm thứ hai khoa Điện – Điện tử, Trường Đại học Sư phạm Kỹ thuật Nam Định) M.A MINOR THESIS Field: Language Teaching Methodology Code: 60 14 10 Supervisor: Phạm Lan Anh, M.A HANOI, 2011 vi TABLE OF CONTENTS DECLARATION .i ACKNOWLEDGEMENTS ii ABSTRACT iii LIST OF FIGURES, TABLES AND CHARTS iv LIST OF ABBREVIATIONS v TABLE OF CONTENTS vi CHAPTER 1: INTRODUCTION 1.1 Rationale 1.2 Scope of the study 1.3 Aims of the study 1.4 Methods of the study 1.5 Research questions 1.6 Design of the study CHAPTER 2: LITERATURE REVIEW 2.1 Relationship between teaching, learning and assessment 2.2 Purposes of formative and summative assessments 2.3 Achievement tests and their characteristics 2.3.1 Achievements tests 2.3.2 Characteristics of a good EGP test 11 2.3.3 Characteristics of a good ESP test 14 2.4 Face validity 15 2.4.1 Definition 15 2.4.2 Relationship between reliability and validity 16 2.4.3 Reasons for choosing face validity 17 2.5 Some measures to increase face validity 18 vii CHAPTER 3: THE STUDY 20 3.1 English learning and teaching at Nam Dinh University of Technology Education 20 3.1.1 Students’ backgrounds 20 3.1.2 The English teaching staff 20 3.1.3 Objectives of the English course 21 3.1.4 Checklist of the course book 22 3.1.5 Objectives of the final test 23 3.1.6 Difficulty level and discrimination of the final test 24 3.2 English testing at Nam Dinh University of Technology Education 24 3.2.1 Testing situation 24 3.2.2 The current final achievement test 25 3.3 Research methods 26 3.3.1 Survey questionnaire 26 3.3.2 Interview and informal discussion 26 3.4 Data analysis of survey questionnaires and interviews 26 3.4.1 Data analysis of the administration of the test 27 3.4.1.1 Data analysis of the format of the test 27 3.4.1.2 Data analysis of the logistics of the test 28 3.4.2 Data analysis of face validity of the test 29 3.4.2.1 Data analysis of general opinion about the test 30 3.4.2.2 Data analysis of reading comprehension task 31 3.4.2.3 Data analysis of grammar knowledge task 33 3.4.2.4 Data analysis of translation task 34 3.5 Discussion and findings 36 3.5.1 Similarities in teachers and students’ perception 36 3.5.1.1 Test administration 36 viii 3.5.1.2 Face validity 36 3.5.1.2.1 General opinion about the test 36 3.5.1.2.2 Reading comprehension task 36 3.5.1.2.3 Grammar task 37 3.5.1.2.4 Translation task 37 3.5.2 Differences in teachers and students’ perception 37 3.5.2.1 Test administration 37 3.5.2.2 Face validity 37 3.5.2.2.1 Grammar task 37 3.5.2.2.2 Translation task 37 3.6 Suggestions to improve the final achievement test 38 CHAPTER 4: CONCLUSION 41 REFERENCES 42 APPENDICES I APPENDIX I APPENDIX V APPENDIX IX APPENDIX XIII iv LIST OF FIGURES, TABLES AND CHARTS Figures Figure 1: Three Considerations for Test Choice Figure 2: The Scope of Impact of Language Tests Figure 3: Relationship between reliability and validity Tables Table 1: ESP syllabus content allocation Table 2: Specification of test 12 Table 3: Teachers and students’ comments on the format of the test Table 4: Teachers and students’ comments on the administration of the test Table 5: Teachers and students’ comment on the whole test Table 6: Teachers and students’ comment on students’ reading comprehension ability, theme and instruction of the reading comprehension task Table 7: Teachers and students’ comment on the grammar task Table 8: Teachers and students’ comment on the translation task Charts Chart 1: Percentage of teachers and students’ comment on what language ability the test mainly intend to measure Chart 2: Percentage of students and teachers’ opinions on what test items can measure their true ability Chart 3: Percentage of the results which students get in the test Chart 4: Percentage of what test tasks students cannot Chart 5: Percentage of teachers and students’ comment on the length of the reading text Chart 6: Students’ comment on whether or not the reading text is difficult Chart 7: Teachers’ comment on which types of reading skills are expressed in the reading comprehension task Chart 8: Teachers and students’ comment on which students’ ability this translation task requires v LIST OF ABBREVIATIONS NUTE: Nam Dinh University of Technology Education EGP: English for General Purposes ESP: English for Specific Purposes CHAPTER 1: INTRODUCTION 1.1 Rationale Learning English in Vietnam today is quite popular and its popularity is increasing day by day This is due to the fact that Vietnam has recently adopted an open-door policy which encourages broadening and improving its relationship and cooperation with other countries in many aspects of life such as diplomatic, economic, cultural, scientific and technological areas English is not only a means but also a key to gaining access to latest scientific and technological achievements for a developing country like Vietnam, where modern science and technology is badly needed Recognizing the necessity of this global language, most of the schools, colleges and universities in Vietnam consider English as the main, compulsory subjects that students must learn In the English language learning and teaching process, evaluating a test is significant Test is a part of teaching and learning process because it also provides feedback about the achievement of teaching and learning objectives for those who are involved in the education system Moreover, the use and necessity of knowing a person’s language ability is through a language test In education, especially at the Faculty of Foreign Languages at Nam Dinh University of Technology Education (NUTE), testing the students’ achievement toward teaching objectives is needed Without an achievement test, it is difficult to see how rational educational decisions can be made NUTE is a technological university and students’ learning English ability is really low Evaluating a test is also one of the methods to improve students’ learning process and results Although there were many theses mentioned this problem but at NUTE it is still new and very necessary because how to evaluate the test after each semester still receives little attention and up to now the process of test analysis after each examination hasn’t been fully investigated Consequently, students’ results are getting worse and worse As a teacher myself, I see that we, teachers at NUTE, just stop at experienced level of test making procedure, test administration, test marking procedure during and after examination We not care testing evaluation from other teachers and students, so the results of test are not still improved Moreover, test’s validity can be seen as an attempt for improving the test quality Being a measure of students’ achievement toward learning objectives, final examinations must be valid Validity is one of the characteristics of a qualified test Therefore, “Evaluating the validity of the final achievement test for second – year non – major students at Electronic – Electrical Engineering Department, Nam Dinh University of Technology Education” is chosen with the hope that the study will be helpful with the author, the teachers, the test-takers and everyone who is concerned with language testing in general and validity of an achievement test in particular Due to limit time in collecting students’ scores, this study is different from the previous study The author only focuses on the face validity of this test The author hopes that the result of the study can then be applied to improve the current test and to create a new really reliable item bank It is also intended to encourage both teachers and learners in their teaching and learning 1.2 Scope of the study The scope of this thesis is limited to a research on teachers’ and test-takers’ evaluation of the existing achievement test in terms of its face validity for the second-year non-English major students at Electronic – Electrical Engineering Department, NUTE due to the limitations in time, ability and availability of data Moreover, it is impossible for the author to cover all used final achievement tests as well as design a sample achievement test for second-year students Instead, only a test specification for test 12 in semester is presented 1.3 Aims of the study Following the scope of the research above, the aims of this research are: To indentify the English teachers and students’ evaluation of the final existing achievement test (test 12) at NUTE in terms of face validity To provide suggestions for test designers 1.4 Methods of the study In order to achieve the above aims, the study has been carried out as follows: First, the author goes to library to read theory about assessment and testing, achievement test with characteristics of a good achievement test and test validity with a special focus on face validity and some measures to increase it From her critical reading, many reference materials have been gathered, analyzed, and synthesized to draw out a theoretical basis to evaluate the current test being used for the 3rd semester students in terms of its face validity Then, qualitative methodologies involving data collected through survey questionnaires and interviews were employed from both teachers and students at NUTE 1.5 Research questions This study is implemented to find answers to the following research questions: What are the teachers’ and test takers’ (students’) perceptions of the final 3rd semester English achievement test at NUTE in terms of its face validity? What are suggestions to improve face validity of the final 3rd semester English achievement test at NUTE? 1.6 Design of the study The thesis is divided into four major chapters: Chapter 1: Introduction presents basic information such as: the rationale, the scope, the aims, the method, the research questions and the design of the study Chapter 2: Literature review reviews theoretical backgrounds on evaluating a test, which includes relationship between teaching, learning and assessment, purposes of formative and summative assessments, achievement tests, characteristics of good EGP and ESP tests, face validity and some measures to increase face validity Chapter 3: The study is the main part of the thesis showing the context of the study and the detailed result obtained from collected tests and findings in response to the research questions Then, the author gives some solutions to improve the final achievement test Chapter 4: Conclusion offers conclusions and proposes some suggestions for further research on the topic VI A Adequate B Not enough C Abundant Do you have measures to prevent cheating in the exam? A Yes B No The atmosphere of the rooms for testing …………………………………………… A Tense B usual C pleasant 10 Can you give more suggestions for the improvement of logistics of the test? A Yes B No If yes, please specify …………………………………………………………………… B Face validity of the test I General opinions 11 Which of the following statements you agree? A the test content reflects syllabus objectives B the test content doesn’t reflect syllabus objectives 12 According to you, does this test measure students’ true ability ? A Yes B No If “No”, could you please recommend briefly how to improve it? ……………………………………………………………………………………………… 13 According to you, this English test mainly intends to measure the students’ ………… A Vocabulary knowledge D Grammar knowledge B Translation ability E All of these above C Reading skill 14 According to you, the difficulty level of the test ……………………………………… A very high D low B high E very low C normal 15 According to you, should you adjust parts of the test? A Yes B No If “yes”, please specify …………………………………………………………………… 16 Please give your common opinion about the final test A appropriate to students level B not appropriate to students level Please specify ………………………………………………………………………… VII 17 Can you give more suggestions for the improvement of the existing test for the 3rd semester students? A Yes B No If “yes”, please specify ……………………………………………………………………… II Reading comprehension task 18 Does the reading comprehension task really test students’ reading comprehension? A Yes B No Please specify ……………………………………………………………………………… 19 Does theme in the reading text have in the syllabus ? A Yes B No 20 What you think of the length of the reading text? A too long B long C short D too short E enough 21 What you think of the difficulty level of the reading text? A High B Normal C Low 22 Do you think the origin of the reading text is reliable? Why? A Yes B No Please specify ……………………………………………………………………………… 23 The instruction of the reading comprehension task are ………………………………… A Very clear B Clear C Unclear 24 According to you, is the reading text’s content updated? A Yes B No 25 According to you, which types of reading skills does the task in the reading text express? A skimming B scanning C detailed D others …………………………………………………………………………………… III Grammar task 26 The grammar task given in the test is ………………………………………………… A completely related to what the students have been taught B partly related to what the students have been taught C not related to what the students have been taught VIII 27 The instruction of the grammar task is ……………………………………………… A Very clear B Clear C Unclear 28 What you think of the difficulty level of the grammar task? A High B Normal C Low 29 What you think of the content of the grammar task? A varied B not varied Your opinion ……………………………………………………………………………… IV Translation task 30 What does this translation task require students’ ability? A translation ability B grammar and vocabulary C writing skill 31 The instruction of the translation task is ……………………………………………… A Very clear B Clear C Unclear 32 Is the content of the translation task appropriate to the students’ ability? A Yes B No Your opinion ……………………………………………………………………………… 33 According to you, are there too many new words in the translation task? Why? A Yes B No Please specify ……………………………………………………………………………… 34 According to you, are structures in the translation task simple to students? Why? A Yes B No Please specify ……………………………………………………………………………… IX APPENDIX TRƯỜNG ĐẠI HỌC SPKT NAM ĐỊNH KHOA NGOẠI NGỮ ĐỀ THI KẾT THÚC HỌC PHẦN TÊN HỌC PHẦN: Tiếng Anh chuyên ngành TRÌNH ĐỘ ĐÀO TẠO: Đại học NGÀNH: Các ngành điện Thời gian: 60 phút Đề số 12 I Read the text and decide if these statements are true (T) of false (F) (2.5 pts) In the first category are substances which provide an easy path for an electric current All metals are conductors However, some metals not conduct well Maganin, for example, is a poor conductor Copper is a good conductor, therefore it is widely used for cables A non-metal which conducts well is carbon Salt water is an example of a liquid conductor A material which does not easily release electrons is called an insulator Rubber, nylon, porcelain and air are all insulators There are no perfect insulators All insulators will allow some flow of electrons However, this can usually be ignored because the flow they permit is so small Semiconductors are midway between conductors and insulators Under certain conditions, they allow a current to flow easily but under others, they behave as insulators Germanium and silicon are semiconductors Mixtures of certain metallic oxides also act as semiconductors These are known as thermistors The resistance of thermistors falls rapidly as their temperature rises They are therefore used in temperature-sensing devices …… All good conductors are non – metals …… Copper does not allow an electric current to flow easily …… Rubber provides an uneasy path for an electric current …… Porcelain is not a perfect insulator …… The resistance of a thermistor is higher at low temperature than at high temperature X II Give the correct passive forms of the verbs in brackets (2.5 pts) A pn junction is formed if two crystals of a semiconductor material, one of p-type and one of n-type (join) together Radio frequency waves (use) to carry audio frequency waves over long distances through the air For a long time, hi-fi recording (produce) on vinyl gramophone records Electrolysis (can use) to break down water into hydrogen and oxygen The electric relay works on the principle that if the magnetic field is cut by another conductor, an electromotive force (will enduce) in that conductor III Translate into Vietnamese (3.0 pts) A typical radio tuner circuit consists of an inductor and capacitor connected in parallel The side of the aerial inductance coil can be kept small by winding it on a ferrite rod core The RF waves fed to the tuner cause the circuit to oscillate The impedance of the circuit is smaller and the oscillator is greatest at a particular frequency known as the resonant frequency This frequency is determined by the values of the inductance and the capacitance By using a variable capacitor, the circuit can be tuned to the required radio frequency and the selected RF wave passed on to the RF amplifier IV Translate into English (2.0 pts) Động điện máy biến đổi điện sang thành Khi dòng điện chạy qua dây dẫn, làm nóng dây dẫn Một máy dao động không cho biết độ lớn tín hiệu mà cịn cho biết thay đổi tín hiệu theo thời gian Đĩa ghi âm sử dụng hệ thống ghi âm mô phỏng, hệ thống lưu mẫu hình việc tạo đường rãnh liên tục đĩa nhựa vinyl Nam Định, ngày 31 tháng 12 năm 2010 TRƯỞNG BỘ MÔN XI TRƯỜNG ĐẠI HỌC SPKT NAM ĐỊNH KHOA NGOẠI NGỮ ĐÁP ÁN ĐỀ THI KẾT THÚC HỌC PHẦN TÊN HỌC PHẦN: Tiếng Anh chuyên ngành TRÌNH ĐỘ ĐÀO TẠO: Đại học NGÀNH: Các ngành điện Thời gian: 60 phút Đề số 12 Questions Content Points I Read the text and decide if these statements are true (T) of false (F) 2.5 F 0.5 F 0.5 T 0.5 T 0.5 T 0.5 II Give the correct passive forms of the verbs in brackets 2.5 are joined 0.5 are used 0.5 have been produced 0.5 can be used 0.5 will be enduced 0.5 Translate into Vietnamese 3.0 Mạch điện phím điều chỉnh radio thông thường gồm phần cảm 0.5 III tụ điện mắc song song Dây ăng ten tự cảm giữ kích cỡ nhỏ cách 0.5 lõi pherit Sóng tần số vơ tuyến nạp vào phím điều chỉnh làm cho mạch dao động 0.5 Ở tần số cụ thể, hay gọi tần số cộng hưởng, trở kháng 0.5 mạch nhỏ dao động lớn Các giá trị tự cảm điện dung định tần số 0.5 Bằng cách sử dụng tụ xoay, mạch điều chỉnh theo tần 0.5 số vô tuyến theo u cầu sóng tần số vơ tuyến lựa chọn tiếp đến thiết bị khuyếch đại sóng tần số vơ tuyến XII IV Translate into English 2.0 An electric motor is a machine for converting electrical energy into 0.5 mechanical energy When a current flow through a conductor, it may heat the conductor 0.5 An oscilloscope not only shows the size of the signal, but also how the 0.5 signal varies with time Records use an analogue recording system, which stores patterns by 0.5 cutting a continuous groove in a vinyl disk Nam Định, ngày 31 tháng 12 năm 2010 TRƯỞNG BỘ MÔN XIII APPENDIX 3.4 Data analysis of survey questionnaires and interviews 3.4.1 Data analysis of the administration of the test 3.4.1.1 Data analysis of the format of the test Table 3: Teachers and students’ comments on the format of the test Options A Questions B T S T No % No % Font Times New Roman C S T S No % No % No % No 100 99 69.7 0 32 22.5 0 11 7.8 100 98 0 38 26.7 0 4.3 % and size 12 of the test Copies of the test … The quantity of questions 28.6 86 60.6 71.4 43 30.3 0 13 9.1 28.6 86 60.6 71.4 56 39.4 - - - - 69 of the test … Can you give more suggestions for the improvement of test format? 3.4.1.2 Data analysis of the logistics of the test Table 4: Teachers and students’ comments on the administration of the test Options A Questions Time allowance for B T S C T S T No % No % No % No % 28.6 82 57.7 71.4 50 35.2 100 118 81.3 0 24 16.9 100 142 100 0 100 142 100 0 28.6 40 28.1 71.4 S No % No % 10 7.1 - - - - 0 0 0 - - - - 90 63.4 0 12 8.5 this test … Are rooms for testing big enough? The quantity of the test paper … Do teachers have measures to prevent cheating? The atmosphere of XIV the rooms for testing … 10 Can you give more 28.6 3.5 71.4 137 96.5 - - - - suggestions for the improvement of logistics of the test? 3.4.2 Data analysis of face validity of the test 3.4.2.1 Data analysis of general opinion about the test 80% 74% 71% 70% 60% 50% teac her 40% s tudent 29% 30% 20% 10% 0% 0% 10% 9% 7% 0% 0% voc abulary trans lation reading 0% grammar all Chart 1: Percentage of teachers and students’ comment on what language ability the test mainly intend to measure teacher student translation into Vietnamese 21% no translation into E nglish 43% 44% teacher student 15% grammar yes 57% 20% reading 0% 10% 20% 30% 40% 50% 0% 10% 20% 30% 40% 50% 60% Chart 2: Percentage of students and teachers’ opinions on what test items can measure their true ability XV s tudent 35% mark 1-6 mark 7-10 65% Chart 3: Percentage of the results which students get in the test s tudent 14% 16% reading grammar 16% trans lation into E nglis h trans lation into V ietnames e 54% Chart 4: Percentage of what test tasks students cannot Table 5: Teachers and students’ comment on the whole test Options A Questions 14 Should you adjust parts B T S T No % No % No 0 94 66.2 7 100 107 75.4 S % No % 100 48 33.8 35 24.6 of the test? 15 Is the test suitable with students? XVI 3.4.2.2 Data analysis of reading comprehension task Table 6: Teachers and students’ comment on students’ reading comprehension ability, theme and instruction of the reading comprehension task Options A Questions B T 16 Does the S C T S T S No % No % No % No % No % No % 28.6 35 24.6 71.4 107 75.4 - - - - 100 117 82.4 0 25 17.6 - - - - 0 0 52 36.6 100 90 63.4 reading task really test students’ reading comprehension? 17 Does theme in the reading text have in the syllabus? 18 The 0 instruction of the reading text is … 90% 86% 80% 70% 60% 53% 50% teacher 40% 30% s tudent 26% 20% 10% 10% 0% 0% 0% too long long 14% 9% s hort 0% 3% too s hort enough Chart 5: Percentage of teachers and students’ comment on the length of the reading text XVII s tudent 38% yes no 62% Chart 6: Students’ comment on whether or not the reading text is difficult teacher 0.6 0.5 0.4 0.3 0.2 57% teacher 43% 0.1 0 s kimming s canning detailed other Chart 7: Teachers’ comment on which types of reading skills are expressed in the reading comprehension task XVIII 3.4.2.3 Data analysis of grammar knowledge task Table 7: Teachers and students’ comment on the grammar task Options A Questions T No 22 The B S T S T S % No % No % No % No % 57.1 75 52.8 42.9 61 43 0 4.2 34 99 69.7 % No C grammar task given in the test is … 23 The 0 6.4 100 23.9 0 15 10.7 71.4 100 70.4 14.3 30 21.1 85.7 112 78.9 - instruction of the grammar task is … 24 What you 28.6 27 18.9 think of the difficulty level of the grammar task? 25 What you think of the content of the grammar task? - - - XIX 3.4.2.4 Data analysis of translation task 90% 80% 70% 60% 50% 40% 30% 12% 49% s tudent 71% 20% 10% 0% teacher 39% 29% 0% trans lation ability grammar and vocabulary writing s kill Chart 8: Teachers and students’ comment on which students’ ability this translation task requires XX Table 8: Teachers and students’ comment on the translation task Options A Questions 27 The B T S C T No % No % No 0 30 21.1 7 100 114 80.3 28.6 108 76.1 28.6 62 S % No T % S No % No % 100 103 72.5 0 6.4 28 19.7 - - - - 71.4 34 23.9 - - - - 71.4 80 56.3 - - - - instruction of the translation tasks is … 28 Is the content of the translation tasks appropriate to the students’ ability? 29 Are there too many new words in the translation tasks? 30 Are structures in the translation tasks simple to students? 43.7