A study on the validity of the current final English test for the 2nd semester non-English majors at Hanoi University of Industry = Nghiên cứu về tính xác thực

LIST OF TABLES AND CHARTS Table 1: A framework for language assessments Table 2: The syllabus for the second semester Table 3: Factors to consider in writing items and tasks Table 4: Key

FACULTY OF POST-GRADUATE STUDIES

NGUYEN VAN BAC

A STUDY ON THE VALIDITY OF THE CURRENT FINAL ENGLISH TEST FOR THE 2ND SEMESTER NON-ENGLISH MAJORS AT HANOI UNIVERSITY OF INDUSTRY

(Nghiên cứu về tính xác thực của bài thi tiếng Anh cuối học kỳ thứ hai hiện nay dành cho sinh viên không chuyên tiếngAnh tại trường

Đại học Công Nghiệp Hà Nội)

M.A Minor Programme Thesis

Field: Methodology Code: 60 14 10

Hanoi, 2010

FACULTY OF POST-GRADUATE STUDIES

NGUYEN VAN BAC

A STUDY ON THE VALIDITY OF THE CURRENT FINAL ENGLISH TEST FOR THE 2ND SEMESTER NON-ENGLISH MAJORS AT HANOI UNIVERSITY OF INDUSTRY

(Nghiên cứu về tính xác thực của bài thi tiếng Anh cuối học kỳ thứ hai hiện nay dành cho sinh viên không chuyên tiếngAnh tại trường

Đại học Công Nghiệp Hà Nội)

M.A Minor Programme Thesis Field: Methodology Code: 60 14 10

Supervisor: Pham Thi Hanh, M.A

Hanoi, 2010

TABLE OF CONTENTS

CANDIDATE’S STATEMENT……… i

ACKNOWLEDGEMENT……… ii

ABSTRACT……… iii

TABLE OF CONTENTS ……… iv

LIST OF ABBREVIATIONS ………

vii LIST OF TABLES AND CHARTS ………

viii CHAPTER 1: INTRODUCTION………

1 1.1 RATIONALE………

1 1.2 SCOPE OF STUDY………

2 1.3 AIMS OF STUDY………

2 1.4 METHODS OF STUDY………

3 1.5 RESEARCH QUESTIONS………

3 1.6 DESIGN OF STUDY………

3 CHAPTER 2: LITERATURE REVIEW………

5 2.1 RELATIONSHIP BETWEEN LANGUAGE TESTING AND LANGUAGE TEACHING AND LEARNING………

5 2.2 LANGUAGE TESTING………

6 2.2.1 Purpose of language testing………

6 2.2.2 Types of language testing………

7 2.2.3 The current trends in language testing………

9 2.3 QUALITIES OF A GOOD TEST………

10 2.3.1 Reliability………

10 2.3.2 Validity………

11

4.3 SUGGESTIONS FOR IMPROVING THE QUALITY OF THE

CURRENT FINAL TEST FOR THE SECOND SEMESTER NON

4.3.1 Take the students’ language ability and knowledge in

LIST OF ABBREVIATIONS

1 EC Economic and Computer Science Students Group

2 ESP English for Specific Purposes

3 HaUI Hanoi University of Industry

6 OM Other Major Students Group

7 SLA Second Language Acquisition

8 TESOL Teaching English to Speakers of Other Languages

9 TOEIC Test of English for International Communication

LIST OF TABLES AND CHARTS

Table 1: A framework for language assessments

Table 2: The syllabus for the second semester

Table 3: Factors to consider in writing items and tasks

Table 4: Key points presented in the course book New headway pre-intermediate Table 5: List of test scores selected from the second semester final test

Chart 1: Opinions of students on the time allowance of the final test

Chart 2: Teachers’ comments on time allowance of the test

Chart 3: Appropriateness of the final test in student’s opinion

Chart 4: Teachers’ comment on the test appropriateness

Chart 5: Test items best measure students’ true ability in students’ perception Chart 6: Teachers’ opinions on test items best measuring your students’ true ability Chart 7: Student’s comments on the level of Grammar and Vocabulary test

Chart 8: Teachers’ comments on Grammar and Vocabulary test

Chart 9: Student’s comment on the difficulty level of Reading Comprehension test Chart 10: Teachers’ comment on the difficulty level of Reading commprehension test Chart 11: Student’s comment on the appropriateness of Writing test

Chart 12: Teachers’ comment on the writing test

Chart 13: Student’s comment on Listening commprehension test

Chart 14: Teachers’ comment on the construct of Listening comprehension test

CHAPTER 1: INTRODUCTION

1.1 RATIONALE

Along with the emergence of the globalization, English has been proving its importance in most areas including science, technology, telecommunication, media, culture, international relations In Vietnam, English teaching and learning have drawn a lot of concerns from not only teachers and students, but the whole society

English is a non-major subject for most students of Hanoi university of Industry (HaUI) However, it has received more attention and time than any other basic subjects taught in the training program On the average, students have to learn English in 5 semesters, four of which serve for general English, and in the last semester students will learn English for Specific Purposes (ESP) Compared with other universities in Vietnam, the time for English program at HaUI is one of the longest

To evaluate student’s English level and achievement at Hanoi University of Industry, testing is an essentially important tool Testing and assessment have been considered as the light on both the nature of language proficiency and language learning In other words, tests can produce the assessment of students’ ability of language use For each semester, students have to take three progress tests, and one final achievement test In the final achievement test, listening, speaking, reading, grammar knowledge and writing, each accounts for 20% of the total mark scale

Although there are a lot of tests used as mentioned above in the process of teaching, it is recognized that these tests may still not exactly evaluate the student’s ability to use English Some students have excellent performance in class, but the results of their test are not satisfactory, and vice versa Other teachers at Hanoi University of Industry also hold this view and they often complain that the current final achievement test for the second semester does not reflect the true language competence of their students Some students and teachers share the same point that what is taught in the program is not included in the test; therefore, it seems not to measure students’ achievement of the course and their expected linguistic skills and knowledge Through interaction with other teachers, I reckon that test writers often choose the test items somewhere else, but not based on the course book and the syllabus given at the beginning of the course

One more reason why I choose this topic for my research is that the test evaluation and assessment at Hanoi University of Industry appear not to receive proper attention Being a teacher of English, I have also involved in designing many kinds of test for non English major students at HaUI, but there are no formal discussions, no systematic and comprehensive assessments, and no research on the appropriateness of the tests

With above mentioned reasons, I have decided to choose the research topic: “A study on

the validity of the current final English test for the 2nd semester non-English Majors at Hanoi University of Industry.” It is believed that this study will be helpful for English

teachers in English Faculty of Hanoi University of Industry who often participate in designing the progress tests and final achievement exams

1.2 SCOPE OF STUDY

The scope of this minor thesis is limited to a study on examining the validity of the current final test for second semester non English major students in terms of its validity for the non-English majors at Hanoi University of Industry

Due to the limitations of time, the author cannot send the questionnaires to all non-English students of Hanoi University of Industry However, to achieve a broad view from the teachers and students of Hanoi University of Industry about the final test in terms of its validity, the author tries his best to give the questionnaires to the students of 5 Faculties including Economic Faculty, Chemistry Technology Faculty, Electronic Technology Faculty, Mechanical Technology Faculty, and Electrical Technology Faculty The students questioned are all university students, not covering the college students The author also cannot conduct the survey and interview with all the teachers of English Department; instead, he selects the experienced ones who regularly involve in designing tests for non English major students and those who are currently involving in teaching first year students in their 2nd semester

- to investigate the appropriateness of the current final test for the 2nd semester non English majors in terms of time allowance, difficulty level, test contents

- to find out the teachers’ and students’ comments on the test validity;

- to provide some suggestions for improving the test in terms of its validity

1.4 METHODS OF STUDY

The study employs a combination of methodologies including quanlitative and quantitative

methods to achieve the aims mentioned above

The author himself base on the literature review on theory and principles of language testing, the characteristics of a good test, the test reliability and validity to achieve the overview of language testing The synthesizing of the literature helps the author to make a framework for the study

Questionnaires are sent to teachers and students involving in teaching and learning in the second semester to collect information on their views of the test’s validity

In addition, an informal interview and discussions are also carried out with the teachers of English and their students to gain more information on the appropriateness of the test

1.5 RESEARCH QUESTIONS

The study is conducted to find the answers to the following research questions:

1 Does the final test for non English major students in the 2nd semester give a true picture

of truly the students’ English Competence according to the view of teacher and students?

2 Does the test measure what is purported to measure (i.e its validity)?

3 How can the test be made valid? In what way should the current final test be improved?

1.6 DESIGN OF STUDY

The study consists of five chapters, organized as follows:

Chapter 1- Introduction- provides background to the study, identifies the problems, states the aim, purpose and significance of the study, the scope, the methods, the research questions and the design of the study

Chapter 2 - Literature review- Presents a review of related literature that provides the theoretical background of the testing and evaluation in general and the test validity in

particular This review also provides an overview of other studies related to testing, evaluation, especially the evaluation of tests in terms of its validity

Chapter 3 - The Study- Provides information about the subjects of the study It then describes the data collection instruments and data collection procedure The rationale for choosing such data collection instruments is also provided

Chapter 4 - Findings and Discussions- Analyses and discusses the data collected to reveal the real results and the validity of the final 2nd semester exam for non English major students of HaUI The causes for any problems if any and some implications for effective final achievement tests will be also discovered

Chapter 5 - Conclusion- Summarizes the major findings that are hoped to find the appropriate way to enhance the validity of final achievement tests to non English majors Limitation of the study and suggestions for further research are also given in this chapter

CHAPTER 2: LITERATURE REVIEW

2.1 RELATIONSHIP BETWEEN LANGUAGE TESTING AND LANGUAGE TEACHING AND LEARNING

Shohamy (2000) introduced three dimensions of potential contributions of LT to SLA: (1) defining the construct of language ability; (2) applying LT findings to test SLA hypotheses and (3) providing SLA researchers with quality criteria for tests and tasks Also, he gave out three dimensions of potential contributions of SLA to LT including (1) identifying language components for elicitation and criteria assessment; (2) proposing tasks for assessing language; (3) informing language testers about differences and accommodating these differences

When regarding relationship between language testing and language teaching and learning, Hughes (1989) supposed that it has both negative and positive sides He believed that “too often language tests have a harmful effect on teaching and they fail to measure accurately whatever it is they are intended to measure.”(1989:1) He argues that good teaching may not create good language tests and vice versa

Bachman (1990) backed the notion that the language testing has positive effects on language teaching and learning In his words, “advances in language testing are stimulated

by advances in our understanding of the processes of language acquisition and language teaching.” (1990:3) For the use of language tests in education program, he said that, “the fundamental use of testing in an educational program is to provide information for making decisions, that is, for evaluation” (Bachman, 1990: 54) He supposed that through language testing we can evaluate learners’ achievements in each certain learning period, self-evaluate our ways of teaching or teaching methods, or the language test can provide

us with input into the language teaching process It was also believed that language teaching and language learning help rater and test makers have more information and more input resources for designing and improving achievement tests

In short, language testing and language teaching and learning have a close and interrelated relation Teaching and learning provide a great source of language materials for test and in turn, testing reinforces, improves and encourages the teaching and learning process

“teaching and testing are so closely related that it is virtually impossible to work in either field without being constantly concerned with the other.” (Heaton, 1988:5)

2.2 LANGUAGE TESTING

2.2.1 Purpose of language testing

Shohamy (1985: 6) made a distinction between classroom tests and external tests Classroom tests are written and administered by teachers while external tests are designed and submitted by an external agency The purposes of classroom tests are to find out whether what was taught in the program was also successful acquired; evaluate and improve instruction; obtain information on students’ progress and language knowledge; help organize learning/ teaching materials; provide information for grades; help diagnose students’ strengths and weaknesses in the language and motivate students to learn

External tests, however, evaluate proficiency; decide whether to accept students to a certain program; provide information for administrative decision- special treatment to certain group, assist in selection and grouping; help evaluate the curriculum; serve research purposes and obtain information for grading

Having some ideas similar to Shohamy, but Henning explains the purpose of language testing in a different way According to him, language tests aim to deal with the diagnosis and feedback, screening and selection, placement, program evaluation, providing research criteria, and assessment of attitudes and sociopsychological differences (Henning, 1987,

pp 1-4)

He states that the most common aim of language test is to find out strengths and weaknesses in students’ learning ability In this sense, the use of diagnostic tests provides critical information to the student, teacher as well as administrator that should make the leaning process more efficient

Language tests can also be used to decide whether students should be allowed to participate in a particular program of instruction To make fair selection and decision, the test must be accurate in the sense that they must provide information that is both reliable and valid

Another use of tests is to classify students’ ability to learn languages In this sense, tests are used to identify a particular performance level of the student and to place them at a suitable level of instruction

In addition, tests are usually used to provide information about effectiveness of programs

of instruction In this sense, group mean or average scores are of greater interest than isolated scores of individual students

Language tests score can be used to provide research criteria such as comparisons of methods, and techniques of instruction, text books or audio visual aids Also, tests can be used to assess student’s attitude toward the target language, its people and their culture, which are essential elements for good language learning

2.2.2 TYPES OF LANGUAGE TESTING

Harrison (1991) introduces four types of language tests, including placement, diagnostic, achievement and proficiency Placement tests are designed to classify new students into certain group, so that they can start the course at the same level as the other students in class The placement test is concerned with the student’s present standing, thus it relates to general ability rather than specific points of learning

Diagnostic tests are used for checking student’s progress in learning particular elements of the course The test may be given at the end of a unit in the course book or a lesson designed to teach on particular point The diagnostic test tries to find out that how well the students learnt a particular material, and it’s closely related to particular elements in the course which have just been taught

Achievement tests look back over a longer period of learning than the diagnostic tests The test aims at showing standard which the students have now reach in relation to other students at the same stage Achievement tests covers a much wider range of material than a diagnostic test and relate to long term rather than short term objectives

A proficiency test aims at assessing the student’s ability to apply in actual situations what

he or she has learnt The test is not usually related to any particular course because it is concerned with the student’s current standing in relation to his or her future needs

The following is the summary of types of language test established by Harrison:

Category Content Purpose Considerations Placement General reference

forward to future learning

Grouping speed of results

variety of tests interview Diagnostic Detailed reference back

to class work

Motivation Remedial work

Short term objectives New examples of the materials taught Achievement General reference back

to the course

Certification Comparison with others at the same stage

Decision about sampling Similar material to that taught in new context

Proficiency Specific purposes

Reference forward to particular applications of language acquired

Evidence of ability to use language in practical situations

Definition of operation needs Authenticity Context Strategies for coping

Table 1: A framework for language assessments

(Source: Harrison, 1991: 5) Henning (1987), however, develops different categories for types of language testing He introduces seven types including objective vs subjective tests, direct vs indirect tests, discrete point vs integrative tests, aptitude, achievement and proficiency test, criterion or domain referenced vs norm referenced or standardized tests, speed tests vs power test and other test categories (Henning, 1987, pp 4-9)

The objective vs subjective tests are distinguished on the basis of the manner in which they are scored An objective test may be scored by comparing examinee responses with an established set of acceptance responses or scoring key The example of this kind of test is multiple choice test On the other hand, a subjective test may be scored by opinionated judgment based on insight and expertise of the scorer The example of this type would be

free composition or cloze tests which permit all grammatical acceptable responses to systematic deletions from a context

Direct tests are said to test language performance directly whereas indirect tests indirectly tap true language performance The direct tests are usually in the forms of spoken tests which are the ratings of language use in real communication situations The indirect tests are usually in the forms of written tests such as multiple choice, cloze tests

Discrete point tests, as a variety of diagnostic tests, are designed to measure knowledge or performance in very restricted areas of the target language Integrative tests, on the other hand are used to assess a greater variety of language abilities

Aptitude tests are usually used to measure the suitability of a specific program of instruction or a particular kind of employment Achievement tests are used to measure the extend of what students have already learnt Proficiency tests are the most often global measures of ability in language or other context area

For the criterion-or domain-referenced test, the instructions are designed after the test are created The tests must match teaching objectives perfectly and they are useful when objectives are under constant revision Such kinds of test are useful with small and/or unique group for whom norms are not available The norm- referenced or standardized tests, on the other hand, must have been administered to a larger number of examinee from the target population Acceptable standards of achievement can only be found by reference

to the mean or average score

Speed test is the test in which the items are easy but the time seem to be insufficient In contrast, power test includes difficult items, but the time is sufficient

Henning (1987) also mention some other categories of tests including examinations vs quizzes, questionnaires, single stage and multi stage tests, language skill tests and language feature tests, etc

2.2.3 The current trends in language testing

Shohamy (1985: 5) gives out three trends in language testing Firstly, it’s the transition from discrete point tests to integrative tasks In the past, language test often based on the independent items like putting the correct verb forms, selecting the lexical elements Tests, nowadays, however, mostly aim at testing communicative competence, and the tasks are

much wider, such as writing letters, comprehension of a whole text with specific elements

in the text

The second trend in language testing is the transition from indirect to direct/ authentic tests Until now, the test methods were mainly indirect, that means it has no relation to the real life situation which are more similar to what the test takers will encounter in real language use

The last trend in language testing mentioned by Shohamy (1985) is the transition from knowledge to performance type tests In performance type tests, students or test takers have to apply the knowledge of the language performing certain functions like actually speaking or actually writing

2.3 QUALITIES OF A GOOD TEST

Three most important characteristics of a good test, according to Harrison (1991), are reliability, validity and practicality However, according to Bachman and Palmer (1996), a test's usefulness can be determined by considering the following measurements qualities of the test: reliability, construct validity, authenticity, interactivity, impact, and practicality In this minor MA thesis, I will mention the three qualities of a good test including reliability, validity and practicality with special focus on validity in the next part of the thesis

2.3.1 Reliability

The test is reliable if it consistently provides accurate measures of abilities at all times, with different students and/or different testers According to Harrison (1991: 10), “the reliability of the test is its consistency.” He confirmed that it is very important that the students’ score should be the same or nearly the same whether the test taker takes one test

or another, and the same result the test taker obtain whether the test is marked by one person or another, and a test should measures the same thing all the time “There are three aspects to reliability: the circumstances in which the test is taken, the way in which it is marked and the uniformity of the assessment it makes.” Harrison (1991: 11)

Henning (1987: 74) supposes “Reliability is thus a measure of accuracy, consistency, dependability, or fairness of scores resulting form administration of a particular examination.” He added that if reliability is concerned with accuracy of measurement, reliability may increase when the error of measurement is made to minimize Therefore,

we should take care of the amount of error present in our measurement so that the reliability could be quantified

The term reliability, according to Bachman and Palmer (1996), refers to consistency of measurement Elaborately, they say that a reliable test score is consistent across different characteristics of the testing situation Moreover, if test scores are inconsistent, they provide no information about the ability being measured Because it is impossible to eliminate inconsistencies on the whole, we try to reduce variations in the test's task features

2.3.3 Practicality

The test must be well organized in advance in relation to time, space, classroom management, equipment, cost “Practicality is the relationship between the resources that will be required in design, development, and use of the test and the resources that will be available for these activities” (Bachman and Palmer, 1996:36) They illustrated that this quality is unlike the others because it focuses on how the test is conducted Moreover, Bachman and Palmer (1996) classified the addressed resources into three types: human resources, material resources, and time Based on this definition, practicality can be measured by the availability of the resources required to develop and conduct the test Therefore, our judgment of the language test is whether it is practical or impractical

2.4 VALIDITY

According to Henning (1987), validity has been distinguished into empirical and non empirical kinds The non-empirical validity does not required the collection of data or the use of formulae (e.g content or face validity, response validity) while the empirical kinds

of validity usually involve resource to mathematical formulae for the computation of validity coefficient The common kinds of empirical validity include concurrent and

predictive validity Another kind of validity mentioned by Henning is construct validity

Having some similar ideas with Henning, Bachman (1990) also introduced five main types

of validity including content validity, criterion validity, concurrent validity, predictive validity and construct validity In this minor MA thesis, I will classify the major types of

validity based on the assumptions of Henning (1987) and Bachman (1990)

2.4.1 Content or face validity

Commonly, testing specialists consider content and face validity to be synonyms (Magnusson, 1967) Of course, some others make distinction between them and suppose that face validity, unlike content validity, is often determined impressionistically

Content or face validity is intuitive and logical but usually lacks an empirical basis The name of this kind of validity shows that it is concerned with whether or not the content of the test is sufficiently representative and comprehensive for the test to be a valid measure

of what it is supposes to measure

The test content must be selective For example, the achievement test’s content should be bound to the content of instruction which in turn is constrained by the instructional objectives

According to Bachman (1990), there are two aspects of content validity including content relevance and content coverage The content relevance requires the specification of the behavioral domain in question and the attendant specification of the task or test domain Content coverage is the extent to which the tasks required in the test adequately represent the behavioral domain in question Demonstrating that a test is relevant to and covers a given area of content or ability is therefore a necessary part of validation

2.4.2 Response Validity

Response validity refers to the extent to which examinees respond in the manner expected

by the test developer It mentions the response manner of the test takers and the instruction

of the test For example, if the test takers respond in a difficult and unreflective manner, their obtained score may not represent their actual ability Moreover, if the instruction of the test is unclear and the test format is unfamiliar to the examinees, their response may not reflect their true ability The two cases mentioned above may be said to be lack of response validity

2.4.3 Concurrent validity and predictive validity

“Concurrent validity is a kind of empirical criterion related validity.” (Henning, 1987) The validity is based on the collected data and formulas applied to generate an actual numerical validity coefficient Of course, the validity coefficient derived represents the strength of relationship with some external criterion measure

To validate a test of some particular ability in this way, one administers a recognized, reputable test of the same ability to the same persons concurrently or within a few days of the administration of the test to be validated

Bachman (1990) supposed that concurrent validity can examine differences in test performance among groups of individuals at different levels of language ability, or examine correlations among various measures of a given ability

Predictive validity has close relationship to concurrent validity It is usually reported in the form of a correlation coefficient with some measure of success in the field or subject of interest The predictive validity can tell us how well test scores can predict some future behavior (Bachman 1990: 250)

2.4.4 Construct validity

The construct validity is empirical in nature because it involves the gathering of data and testing of hypotheses However, unlike concurrent and predictive validity, it does not have any one particular validity coefficient associated with it

According to Henning (1987), the purpose of construct validation is to provide evidence that underlying theoretical constructs being measured are themselves valid The construct validation usually begins with a psychological construct that is part of a formal theory

which enables certain predictions about how the construct variable will behave or be influenced under specified conditions Then the construct is tested under the conditions specified, and it is said to be valid if the hypothesized result occur and the hypotheses are supported

Construct validity concerns the extent to which performance on tests is consistent with predictions that we make on the basis of a theory of abilities, or constructs (Bachman 1990: 255) In order to examine the construct validation, it is necessary to exam patterns

of correlations among item scores and test scores, and between characteristics of items and tests and scores on items and tests; analyze and model the processes underlying test performance; study group differences; study changes over time, or investigate the effects

of experimental treatment (Messick 1989)

CHAPTER 3: THE STUDY

3.1 THE SUBJECT AND THE CONTEXT OF ENGLISH TEACHING AND LEARNING AT HAUI

3.1.1 English teaching and learning context at HaUI

English faculty is one of the biggest faculties of Hanoi University of Industry There are more than 150 teachers of English who are divided into three divisions One division is in charge of teaching English for students of English, the other one is in charge of teaching English for secondary and vocational student, and the biggest one teaches English for all college and university non English major students All students of Hanoi University of Industry study English as their foreign language

According to the objectives given in the syllabus , the teaching aims of the English course for the non English students in the second semester are stated as follows:

In general, it helps enhance the knowledge and skills students have studied at the elementary level (the 1 st term), as well as improve General English level of student up to pre-intermediate level

In details, it aims to provide students with knowledge of vocabulary, grammar, pronunciation and develop their listening, speaking, reading and writing skills based on natural and social science topics; give students orientation about the importance of English

in their life and in their future jobs; and build and practice languages learning skills as well

as develop their own thinking and ideas when communicating in English

Grammar: Grammatical points are improved and enhanced through each unit All

principles related to grammatical points in each lesson are practiced effectively through group work and pair work

Vocabulary: In this section, students have chance to improve their own vocabulary

considerably Vocabulary provided mostly is related to the topic of each unit

Skill work: In this part, students improve and develop their listening, speaking, reading

and writing skills These skills are integrated and this integration will help students to uncover their creativeness, which brings about the best learning result

Everyday English (communication focus): Students are equipped with some cultural

knowledge of English speaking countries and communication samples Besides, students can entertain with songs and interesting conversation practice

Writing: Students are able to write some short paragraphs about the topics related to each

unit such as writing about their last holiday, future plans, hometowns, etc

The syllabus of General English course for the second semester non English major students

of HaUI is described in the following table:

1 Unit 1: Getting to know you! 4 4

3 Unit 3: It all went wrong 4 4

4 Unit 4: Let’s go shopping! 4 4

5 Stop and check 1 + Progress test 1 1 1

6 Unit 5: What do you want to do? 4 4

7 Unit 6: Tell me! What’s it like? 4 4

10 Stop and check 2 + Mid-term test 1 1

12 Unit 10: Scared to death 4 4

13 Unit 11: Things that changed the

world

14 Unit 12: Dreams and reality 4 4

15 Stop and check 3 + Progress test 2 1 1

16 Unit 13: Earning a living 4 4

17 Unit 14: Love you and leave you 4 4

18 Stop and check 4 + Revision 2

Table 2: The syllabus for the second semester

Course books being employed by teachers and students of Hanoi University of Industry are the set of New Headway by John and Liz Soars (2000) (elementary, pre-intermediate and intermediate), Talktime by Susan (2004), and TOEIC Analysts by Taylor (2006) As for non English majors in the second semester, the main course book is New Headway Pre-intermediate (2000) by Liz and John Soars Besides, students are recommended to use

another reference book named English Grammar in Use by Murphy, R In the teaching process, teachers also use other materials to present and recycle the basic structures of English to develop students’ proficiency in using these structures in certain contexts The focus is also placed on reinforcing and improving students’ knowledge of vocabulary and students’ ability of communication

The major teaching points of the course book for the second semester are presented in appendix 5

3.1.2 English Testing for non English majors at HaUI

For each semester, students are required to take at least three progress tests and one final achievement test During my teaching at HaUI, I reckon that testing is not the main concern of teachers Testing has not been paid proper attentions and carefully studied in terms of its validity, reliability, format and practicality

Within the scope of this thesis, the study focuses on investigating the validity of the final achievement English test (for the second semester) for non English major students who have been learning English for 120 class hours covering all 14 units of New Headway Pre-intermediate Hereunder is the testing format registered to the second semester non English Majors named Test 2 or the final achievement test

Test 2 with the time allowance is 60 minutes has total score of 100 points and consists of the following parts:

Section A (20 points): Grammar and Vocabulary This section includes 20 multiple choice questions and is marked 20 points

Section B (20 points): Reading comprehension This section contains 2 short reading passages with 10 multiple choice questions

Section C (20 points): Listening In the listening section, students are required to listen to several short conversation or short talk and then answer the questions There are 10 multiple choice questions and 10 true/ false questions with 1 point for each correct answer Section D (20 points): Writing Students have to do 5 sentence building questions by selecting the correct answers, then write a short paragraph about one of the topic given before

Section E (20 points): Speaking In speaking section, students often introduce about themselves and then talk about one of the topic they have been assigned (See appendix 4)

3.1.3 Subject of the study

Subjects of this study are students and teachers from HaUI This is the University which gives out the most time for EFL with 6 periods each week In the university, English

is learned not as a major but as an instrument Students are oriented to take the TOEIC test

in their graduation examination, which helps them a lot in their work in the future

3.1.3.1 Students

150 first year college students are selected from different classes of Department of Computer science, Department of Garment and Fashion design, Department of Chemistry, Department of Economics, and Department of Mechanical Engineering, Department of Electronic Engineering, and Department of Electrical Engineering Of which, students of economics and computer science are considered having better ability in using English and they are required to take the TOEIC Test in their graduation exam At the same time, students of other departments are required to take a B level test when they graduate Their English, as classified by the placement test, is at elementary level and pre-intermediate level Most students studied English between 3 and 7 years at lower and upper secondary school Among them, some learnt other foreign languages rather than English As a result, their English proficiency is considerably varied

Based on the situation that students of Economics and computer science are judged as better users of English and they will takes the TOEIC course and TOEIC Test before graduating, and that students of other majors will take the other course named Talktime and will take B level test as the requirement for their graduation, the author divided the student population into two separated groups The first group includes students of Economics and Computer Science (75 students, hereinafter referred to as EC) and the second one includes other students (75 students hereinafter referred to as OM) participated in this survey

3.1.3.2 Teachers

The English Department is one of the biggest departments of HaUI in terms of its staff number There are more than 140 teachers of English who are in charge of teaching English for almost all students of HaUI including vocational students, college students and university students In this study, 15 teachers of English at HaUI are selected They all

have been teaching English for the first year students and have experienced in test preparation They are well trained and have at least three years of teaching experience Ten

of them have master degrees issued by College of Foreign Language, Vietnam National University and Hanoi University, and the rest have been pursuing a master course in TESOL

Both students and teachers are willing and enthusiastic to take part in this study

3.2 RESEARCH METHODS

This empirical study is carried out with three data collection instruments, namely survey questionnaire, interview and document analysis The overall purpose of these data collection instruments is to investigate the validity of the final achievement test registered

to the second semester non- English major students The author hopes that these collection instruments will help him to find out the judgment of teachers and students about the test validity Moreover, the document analysis is expected to give more data about the test validity

3.2.1 Survey questionnaire

For the validation methods, Huong (2000: 67) introduces three methods of test validation: Internal, External and Judgmental She supposes that the internal methods refer to ways of investigating the internal structures of the test scores to ascertain the adequacy of the test scores as indicators of the latent traits, or sample of behaviors or both External method investigate the relationship between test scores and other measures external to the test Judgmental methods refers to methods which are based on the judgments of either expert

or ordinary people about the test content and tasks

Thus the questionnaires are the key methods employed to collect the data It aims at collecting information indirectly from students and teachers Regarding the benefits of survey questionnaire, Richards and Lockhart (1994:10) emphasizes that “survey questionnaires are useful way of gathering information about affective dimensions of teaching and learning, such as beliefs, attitudes, motivation and preferences and enable a teacher to collect a large a mount of information relatively quickly.”

The student survey questionnaire is chosen to carry out this survey for it has many profits

In the first place, it can reach a huge amount of people in a short time According to Nunan

(1993: 143), “the time required to collect data is less compared to interviews.” It will take

us only a week to hand out and collect data from survey questionnaire, and one which to analyze the data Secondly, it is the quite easy and simple to summarize and report collected data since all the informants will be polled the same questions The third advantage of student survey questionnaire is that students are given a chance to express their thought without embarrassment as their names are kept confidential Last but not least, it is inexpensive using survey questionnaires

Description of survey questionnaire

Before constructing questionnaire itemss, the author studied carefully the objective of the study so that each item is directly referenced against one or more of the research objectives He also piloted the survey questionnaire to see its strength and weakness in collecting data, so that he can make change for appropriateness based on the comment of his colleagues and students

* Survey questionnaire for students

The author designs 14 questions in the survey questionnaire for students, in which questions 1 to 4 collect students’ comments on the whole test The next question collect students’ comment on the grammar and vocabulary section of the test Questions 6, 7 aims

at gathering students’ opinions of reading section while the next three questions ask about the appropriateness of writing section and the last question is about the speaking section In this study, the data collected from students’ questionnaires will be presented in separated table equivalent to each section In short, these questions were designed to find out how students evaluate the validity of the current final English test for the second semester non English major students in terms of its construct, content and time allowance

* Survey questionnaire for teachers

There are 14 questions in the survey questionnaire for teachers It is basically similar to the one for students; however, some questions are different Teachers are also polled to reveal their assessment of the validity of the test regarding the construct, time allowance, the content of the test and the methods to improve the quality of the test, especially in terms of its validity

To help students and teachers understand and be able to decide what and how to respond in

a relevant way to a certain question, clear instructions are given at the beginning of survey

session Additionally, the survey questionnaires are translated into Vietnamese, the learner’s and teacher’s L1; and the Vietnamese version is handed out to avoid their misunderstandings

3.2.2 Interview

An inevitable downside of survey questionnaire, which slightly affects the quality of collected data, is the lack of space for unpredicted responses In order to lessen this imperfection, informal interview used in this study contain more blank for informants to express their own ideas Referring to benefits of interview, Nunan (1993: 150) express that

“it gives the interviewer a great deal of flexibility The interview can follow up a respondent’s answers to obtain more information and clarify vague statements.”

Description of interview

The informal interview is aimed at getting more information from students and teachers as they could not expose when answering the questions in survey questionnaire The study uses the semi-structure type of interview (see appendix 3 for interview questions)

* Interview with students

15 students from 5 classes will be selected randomly to attend the interview which is focus

on the judgments about the time allowance, the content and construct of the test

* Interview with teachers

10 teachers will be selected at random to take part in the interview which is also focus on the content, construct and the time allowance Of course, they are asked about how to enhance the quality of the test in terms of its validity

290 students’ marks of final achievement test for the second semester non English major students are collected for the document analysis

Collection of student’s results is aimed at discovering explicitly the range of student’s marks Through the mark statistics, the author can assess the difficulty level of the test and the distribution of students’ test scores

3.3 DATA COLLECTION PROCEDURE

To collect the information about the test validity of the final achievement test for non English major student at HaUI, first of all, 150 survey questionnaires were handed out to the first year students of different majors as mentioned before At the same time, 15 teachers of English were asked to complete 14 survey questionnaires which were designed for teachers only

After survey questionnaires were collected, interviews were carried out with 15 students and 10 teachers chosen at random to find out more detailed data, which were recorded on a paper

Next, 290 students’ mark data were collected randomly for the document’s analysis

When data collection is accomplished, the data analysis will be initiated The results of survey questionnaire and interview will be analyzed to investigate the validity of the final achievement test for the second semester non English major student of HaUI

CHAPTER 4: FINDINGS AND DISCUSSIONS

4.1 DATA ANALYSIS

In this section, the data collected from the survey questionnaires, and the interview with teachers and students will be analyzed and discussed Moreover, the scores analysis will also be done to reveal the validity of the final achievement test for the second semester non English major students

4.1.1 Data analysis of students and teachers survey questionnaires and interviews

One hundred and fifty questionnaires were handed out to the students of Hanoi University

of Industry who already took the final achievement tests for the second semester Students were chosen from different faculties including Faculty of Computer science, Faculties of Garment and Fashion design, Faculties of Chemistry, Faculties of Economics, and Faculties of Mechanical Engineering, Faculties of Electronic Engineering, and Department

of Electrical Engineering

The questionnaires for teacher were completed by 15 teachers who were in charge of teaching English for students in their second semester Along with the questionnaire for teachers, I also have some informal to those who participated in teaching students at the second semester and those who are in charge of composing test or selecting test items

As mentioned before, the questionnaires were used to collect students’ and teachers’ evaluation about the test in terms of time allowance, content, and construct, so that the author can partly discover the validity of the test

Time allowance of the final test

55

58

2 1 0

10 20 30 40 50 60

Enough time for test

Not enough time

Too much time

EC OM

Chart 1: Opinions of students on the time allowance of the final test

As shown in the chart 1, 77.3% of students of other majors (OM) suppose that the time allowance for the final test is not enough When being asked about the time allowance, most of them would like to have more time to finish their answers On the other hand, only

24 percent of students of economics and computer science (EC) say they do not have enough time to do the test

In contrast, the number of students find enough time for the test among EC students is 55 out of 75 while that of OM students is just 18/75 Only few students (2 EC and 1 OM) suppose that the time for the final test is too much It seems that the test is too easy to EC students but too difficult to OM students

47%

40%

13%

Enough Not enough Too much

Chart 2: Teachers’ comments on time allowance of the test

As shown from the chart above, 47 % of teachers determined that the time allowance of the test is enough while there are 40 % of them suppose that students do not have enough time

to complete the test or the test needs more time so that the test construct can be varied and teacher can evaluate students’ achievements more thoroughly Only 13% agree that students have too much time to do the test

Chart 3: Appropriateness of the final test in student’s opinion

When asking about whether or not the test can measure what students have been taught, 85.3 percent of students in EC group agree that the test is appropriate or can measure what they have been taught while only 14.7% of them disagree In OM group, more than 57 percent say “yes” and nearly 43 percent say “no”

Chart 4: Teachers’ comment on the test appropriateness

6

9

0 1 2 3 4 5 6 7 8 9 10

Concerning the appropriateness of the test, 60% of teachers (9 teachers) participating in this survey approve that the test content can measure what they have taught during the semester However 40% of teachers (6 teachers) believe that the content of the test does not cover what they have taught, and that is the reason why the test results of students are not satisfied