Luận văn thạc sĩ VNU ULIS non english major students’ perceptions and attitudes towards the use of the TOEIC listening and reading test as a graduation requirement

INTRODUCTION

Background of the study

Accompanying the globalization of the English language, a global commercial testing industry has been established, specially merchandising goods known as standardized English language proficiency tests As Templer (2004) argues, these tests have increasingly controlled access into privileged educational and occupational paths in the professional and corporate world, and accordingly affected many local language policies Heated discussions and concerns have centered on the impacts of standardized tests on societies and on languages other than English, and how national and local authorities manage entry to language education and language rights (Tollefson, 2002)

In Vietnam, among standardized English language proficiency tests, the Test of English for International Communication (TOEIC) introduced in 2001 has gradually assumed a greater gate-keeping role Although it may not be as a well- known certificate as TOEFL (Test Of English as a Foreign Language) or IELTS (International English Language Testing System), TOEIC has become a popular English proficiency test not only in Vietnam but also in some other Asian countries like Japan, Korea and Taiwan Being a tool to measure workplace English proficiency for non-native English speakers, TOEIC certificates are required for job application in many business corporations in our country such as Viettel, Vietnam Airlines, LG Electronics, Panasonic, and even in stated-owned financial institutions

The importance of English as a lingua franca in the context of the fast globalization was early acknowledged by the Vietnamese government, and as a result, English education has been prioritized at all levels of the Vietnamese education system At tertiary levels for example, undergraduates generally took 200 hours of English at university over four academic years (Ton & Pham, 2010) Yet, this instruction was delivered in a manner which did not allow the vast majority of students to achieve sufficient levels of competence to pass the certifications required for recruitment As Ha (2007) indicated, the communicative competence in English of most Vietnamese employees was still very limited, and many freshly graduated students failed to find jobs in foreign companies because of the English language requirements

Vietnamese decision-making bodies were aware that major changes should be made to upgrade students‟ English proficiency and get them prepared better for global competition at the workplace Accordingly, the Circular No 05/2012/TT- BGDĐT was issued by the Ministry of Education and Training (MOET), effective as from April 2012, indicating that non-English major graduates need to have English proficiency equivalent to B1 (CEFR), 450 TOEIC, 450 TOEFL ITP, or 4.5 IELTS As a result, the majority of HE institutions in Vietnam gradually have set a requirement for their students to achieve a satisfactory score on one of the recommended English language proficiency tests before graduation

The statistics show that approximately 120 HE institutions in Vietnam, over

30 universities in Hanoi alone, are using TOEIC scores normally ranging from 450 to 550 as an exit requirement for their students For example, the Decision No

148/QĐ-ĐHBK-ĐTĐH dated 5 September 2017 by the Rector of Hanoi University of Science and Technology states that non-English majors need present the TOEIC certificate of minimum 500 points or equivalent certificates before being assigned a graduation thesis While this university and some HE institutions such as People's Security Academy and University of Transport Technology cooperate with the only

ETS-authorized organization, IIG Vietnam, to set TOEIC-taking examinations for their own students; others request their students to take the tests at IIG Vietnam headquarter by themselves Obviously, the TOEIC test is now one privileged high- stakes standardized test in Vietnam.

Rationale of the study

The use of the high-stakes standardized test to determine graduation eligibility has a direct impact on students who take the test to meet the exit requirement, and it has important educational and economic implications for students and society at large Language learners or students whose education and language learning are directly influenced by the implementation of the language graduation requirement policy are perhaps the most important stakeholders in this testing situation However, their views on the policy implementation are rarely considered by university authorities

Previous research has suggested that learners‟ attitudes towards and perception of a test and its use can affect their motivation and performance on the test (Bachman & Palmer, 1996) It has been about seven years since the TOEIC test was officially used as an exit English test at some HE institutions in Vietnam, but there is little research examining the appropriateness and effectiveness of the policy, despite its increasing popularity This study is an overall investigation into the test- takers‟ own opinions on the English requirement policy for graduation and the use of the TOEIC certification as an exit requirement at tertiary education levels Given that the test takers‟ perceptions can have wide-ranging consequences, opinions about the use of the TOEIC tests as an exit test may lead to further investigation to ensure the tests‟ validity.

Aim of the study and research questions

The study aims mainly to investigate non-English major students‟ perceptions of and attitudes towards the use of TOEIC Listening and Reading test scores to determine their graduation eligibility in some HE institutions However, it is necessary to examine their perceptions of the English requirement policy for graduation in general first, and then their opinions about the appropriateness and effectiveness of the TOEIC as an exit test could reveal whether the test use is in line with the intended purposes of the policy In order to achieve this aim, the following objectives were addressed in the study:

- To identify how non-English major students perceive the English requirement policy for graduation

- To investigate the opinions of the students who take the TOEIC Listening and Reading test for graduation about the appropriateness and effectiveness of this exit requirement

With these objectives in mind, the researcher conducted the study to answer these research questions:

1 What are non-English major students’ perceptions of the English requirement policy for graduation?

2 What are their perceptions of the appropriateness and effectiveness of the TOEIC as an exit test?

Methodology and scope of the study

This study was carried out at a prestigious TOEIC preparation center in Hanoi, where thousands of students from different universities study to get the target TOEIC scores for graduation The quantitative data were collected by means of survey questionnaires from 197 non-English major students (n = 197), who were sophomores, juniors, or seniors, of both genders, varied majors and English proficiency The follow-up semi-structured interviews of eight participants were conducted to obtain the qualitative data Then the quantitative data were analyzed using SPSS, and the qualitative results were analyzed with the aid of NVivo to develop themes and codes The triangulation of quantitative data and qualitative themes also increased the reliability of the research findings.

Significance of the study

As this research is an endeavor to emphasize the importance of listening to the ideas from students – high-stakes standardized test-takers, there are several significant contributions of this study First, the findings of this study might contribute to raise other stakeholders‟ awareness that test takers‟ voices can provide information about high-stakes standardized English language proficiency tests from different perspectives Second, this research gives the participants chances to share their thoughts and experiences of taking the TOEIC tests as a graduation requirement Given that the test takers‟ perceptions can have wide-ranging consequences, their opinions about the use of the TOEIC tests as an exit test also encourage a further investigation to ensure the tests‟ validity and reliability Finally, investigations into these issues could provide the MOET policymakers and university authorities with important insights into the appropriateness of the English graduation requirements or cut-off scores.

Structure of the thesis

The thesis contains five chapters including this introduction Chapter 2 reviews the theories of students‟ perceptions of and attitudes towards high-stakes standardized tests and some previous studies on the use of the TOEIC test as an exit requirement in Vietnam It also identifies the research gap that this study should fill in After that, chapter 3- the methodology presents the way this study was conducted such as the research design, participants, data collection and analysis procedures Next, chapter 4 presents the results and discussion of this study Finally, chapter 5 reports the conclusion of the study and provides information about the limitations of this study and recommendations for future studies as well.

LITERATURE REVIEW

High-stakes tests

High-stakes tests usually refer to large-scale standardized tests or public examinations, whose results are used for making important educational and professional decisions about individuals, such as admissions, graduation, employment, or promotion (Madaus, 1988) High-stakes testing policies “represent a motivational policy” (Ryan & Brown, 2005, p 358) which applies performance contingent rewards and punishments to standardized test scores Performance contingent rewards refer to those given for satisfying or surpassing a certain requirement or standard, whereas performance-contingent punishments are those given for failing it (Deci & Moller, 2005)

Under high-stakes testing policies, schools, teachers, and students are the key stakeholders affected For schools, the received amount of subsidies or grants could be determined by students‟ test results or performance ratings For teachers, their teaching might be evaluated based on their students‟ test scores For students, if they perform up to certain standards, they are rewarded through praise or through concrete rewards, such as money; in contrast, if they perform badly or fail the test, they could be punished by being held back in school or being unable to graduate (Kohn, 2000) From this perspective, high-stakes testing can create a context where the stakeholders (i.e., schools, teachers, and students) direct their attention to test

Reliability and Validity of standardized tests

In discussing the same candidate writing the same test at a different time, Hughes (1989) notes, “the more similar the scores would have been, the more reliable the test is said to be” (p 29) While this view of reliability issues being the result of random measurement error is quite common (Lewis, 1999), Bachman

(1990) uses a broader definition encompassing not only “how much of an individual‟s test performance is due to measurement error”, but also how much is due “to factors other than the language ability we want to measure” (p 160)

Bachman (1990) discusses three factors that may affect test reliability The first are Test Method Facets, which are broken down into the five categories of testing environment, test rubric (organization), input (format of and nature of), expected response (format of, nature of, and restrictions on), and the relationship between input and expected response (reciprocal, non-reciprocal, adaptive) The second are Personal Attributes and include aspects such as age, gender, cognitive style and, background The third are Random Factors and can include aspects such as tiredness and a candidate's emotional condition, as well as random differences in the testing environment

In defining validity, Hughes (1989) states, “a test is said to be valid if it measures accurately what it is intended to measure” (p 22), while Bachman (1990) on validity notes, “In examining validity, we look beyond the reliability of the test scores themselves, and consider the relationships between test performance and other types of performance in other contexts.‟‟ (p 236) Bachman further quotes the standards of educational and psychological testing as in part saying, “validity always refers to the degree to which the evidence supports the inferences that are made from the scores” (1990:237)

Bachman goes on to point out that reliability and validity are not independent entities, but are complementary in that reliability is a necessary condition for validity (1990:160) Thus, a certain grammar test may be highly reliable, yet not considered a valid test of spoken English, while a poorly assessed spoken test may have low reliability but may otherwise be considered valid for its purpose.

Different reactions to high-stakes standardized testing

The use of high-stakes standardized testing and its impact on educational contexts remain controversial Policymakers intend to use high-stakes standardized tests to make desired educational changes They argue that “the promise of rewards or the threat of sanctions is needed to ensure change” (National Research Council,

1999, p 35), which assumes that the behaviors that the stakeholders adopt associated with high scores will be reinforced, whereas those with poor test performance will be diminished From this perspective, high-stakes standardized tests have been treated as powerful and “effective tools for controlling educational systems and prescribing the behavior of those who are affected by their results”

Many agree on the usefulness of the use of external contingencies to control behaviors Shanker (1993, as cited in Ryan & Brown, 2005) argues that when the achievement outcomes, such as test scores, are significantly attached to the consequences, students will have “the incentive to work hard and achieve because they know something important…is at stake” (p 9) Finn (1991) also holds that high-stakes standardized testing can effectively change student‟s learning behavior when contingent rewards and sanctions exist behind the test results Thomas (2005) argues that some students under high-stakes testing policies will be more motivated to exert more effort to improve their test scores in the future Stecher (2002) also concludes that students with high-stakes testing are motivated to put in more effort, know better about their own abilities and what to study, and align their own effort with rewards

Some research studies also demonstrate the potential positive effects of high- stakes standardized testing on students For example, in their qualitative study focusing on the impact of a high-stakes test on 102 low-achieving students in Chicago public schools, Roderick and Engel (2001) found that lower-performing students could hold positive attitudes towards the high-stakes testing policies if the goal is perceived achievable They further suggest that these students would be motivated to study harder if they are provided with “incentives… through goals that provide an opportunity for feedback, a tangible reward, and a way to construct meaning regarding learning” (p 219)

The power and controlling nature of high-stakes standardized testing have been subject to criticism for its potential undesirable consequences (Ryan & Brown,

2005) The study conducted by Noble and Smith (1994) shows that when high stakes is placed behind test performance, teachers tend to focus on the activities that could boost test scores, such as practicing test-taking skills or reviewing past examination papers, and their students are thus directed to learn toward the test rather than to enhance long-term knowledge growth

Amrein and Berliner (2002) took a comprehensive look at the consequences of high-stakes standardized tests by conducting a state-by-state analysis in the U.S

A total of states were included and four separate standardized state tests (i.e., the American College Testing Program (ACT), Scholastic Aptitude Test (SAT), National Assessment of Educational Progress (NAEP), and Advanced Placement (AP) were examined The analyses revealed that the state‟s high-stakes testing policy failed to enhance transfer Amrein and Berliner (2002) found that after the policy was implemented, the students from two-thirds of states performed worse on the ACT; in other words, these students did not perform better on the outside tests, although increased scores were shown on the state‟s high-stakes test Some other studies (Stecher, 2002; Ryan & Brown, 2005) also raise some other concerns other negative impacts of high-stakes testing on students, such as decreasing intrinsic motivation, stimulating test anxiety and frustration, being more competitive, and using more inappropriate test-taking strategies.

Setting standards of high-stakes testing

Thomas (2005) indicated that five of the more popular criteria used in the U.S to set standards for high-stakes tests are: (a) the ideal student, (b) the attainable level, (c) other students, (d) each student‟s own record, and (e) some combination of (a)–(d) (p 59)

The criterion of “the ideal student” is the standard of a person who is ideally fitted to display all worker-virtues at a high level of efficiency after entering the workforce and able to attain an optimal degree of self-fulfillment Business executives are most impressed by these features of high level of attainment when selecting their employees However, to apply such a standard, education leaders are obligated to explain and specify the numbers of test items and the level of difficulty in each test, which can increase the level of difficulty in practice As Thomas singled out, the ideal level of this standard will be too high if only a small portion of learners earn acceptable marks, and will be too low if an excessively large proportion of students receive extremely high ratings

The attainable level is the criterion that education leaders (such as policy makers, administrators, test-makers, or teachers) believe that most students can answer correctly if their students have studied diligently The assumption of this standard implied that all participants can master all of the learning objectives For the purpose that all students or nearly all students are to pass, the standard tends to be set at a level that the least adept can reach This low expectation may “result in the average and above-average students not being challenged to exert their best effort, because they lack the incentive to learn far beyond the limits of such an expectation” (p 60) Failing a test can “motivate some learners to greater effort to score higher in the future” (Thomas, 2005, p 61) However, if the attainable level is mandated to such a high standard that few learners reach it, the large portion of the students who fall below the pass line may hold negative views about their abilities and “view themselves as failures.” Some students, who already have been working diligently and still fail, may become discouraged, give up and drop out

Additionally, having a high proportion of students fail to pass the tests after applying this criterion manifests that “the instruction has been very poor, or that the methods of evaluating student progress have been faulty, or that the achievement standards have been unreasonably strict” (p 60)

The criterion of “other students”, according to Thomas‟s categorical definition, compared students‟ performances to those of their classmates, grade- mates, or age-mates The school administration can avoid being accused of setting an unreasonably high or low attainment level However, the decisions for the cut-off score (which is subjectively decided by opinion) to distinguish students‟ passing or failing will be challenging to policy makers, administrators, test-makers and teachers, which is subjectively decided (p 61)

The criterion of “each student‟s own record” drew on the judgment of passing or failing a test by focusing on the progress that each student has made for a given period of time This individual-progress approach has been highly eulogized for its promotion of the goal of developing a person‟s potential of being what he or she is capable of However, Thomas explained that the drawbacks of this type of criterion-setting are (a) developing a false sense of a person‟s actual ability if only comparing against his or her past record; (b) failing to provide information about how a learner performs compared with other age-mates or global counterparts of the same age and (c) suffering rude shock in the real world after a learner realizes that the high grade received in school for his or her moderate improvement was unrealistic and devastating

The Hybrid Approach to combining the features of the ideally-educated- person, attainable-level, and student-comparisons criteria is the fifth popular achievement standard proposed by Thomas The combination of above-mentioned perspectives is a viable alternative in terms of setting achievement standards The whole learning process is regarded as a system arranged in hierarchical stages combining all these elements The student-comparisons aspect can be viewed as the starting stage for setting their criteria of learning and testing at their own learning pace, specifically to the learning disadvantaged student group that should pass the required tests during any given year in order to avoid the failure label The attainable-level element is found in the stage of progress as proposed each year by the school authorities so that eventually every learner can attain the standard set by their school in a certain given period of time The ideally-educated-person element is obviously represented by the ideal alternate aim to have virtually all students proficient in knowledge and practical skills in the related field after they graduate from schools

In summary, Thomas (2005) denoted that people‟s beliefs about “sufficient competency” and “fair treatment” can lead to their disagreement with the test-score levels or cut-off points by which to determine the designations success or failure of a test (p 76) Disagreement in proper standards of tests is bound to continue

Expecting complete accord regarding standards-setting is challenging and unrealistic.

Preparing students for high-stakes standardized testing

Crocker (2005) defined appropriate test preparation as “instruction that is geared not only toward study of content from the domain of knowledge sampled by the test, but also toward practicing the skills that will allow students to demonstrate their knowledge on various types of assessment exercises” (p 161) He explained that teaching for assessment occurs when students are taught “the broader content domain represented by the curricular standards, not simply to that subset of content sampled by the items on a single test form” He used the term “teaching “for” the test” to distinguish the negative connotation of “teaching to the test” When dealing with students‟ urgent needs in high-stakes English exit examinations, test preparation is the primary concern Four essential elements of teaching for the test include: (a) a challenging core curriculum, (b) comprehensive instruction in that curriculum, (c) developing students‟ test-taking skills, and (d) adherence to ethical guidelines regarding preparation of students for assessment (Crocker, 2005)

Approaches to test preparation offered by Smith and Delisi (2001) were useful to classroom instructors in differentiating curriculum alignment from measurement-driven instruction The four stages of test preparation for instructors were as follows: “(a) teach without paying attention to the standardized test and hope that the students‟ abilities will show through on the assessment; (b) spend most of the time in instruction as instructors normally do, but spend some time going over item formats to be found on the assessment so that students will be familiar with these formats; (c) analyze the content of the assessment, make certain that instructors cover the content in the regular instructional program, then work on item format and test taking skills as well; (d) analyze the content of the assessment and restructure the instructional program around that content exclusively” (Crocker,

2005, pp 90-91) The first approach illustrates teaching without test preparation and without curricular alignment The second approach illustrates instruction without curricular alignment, but with some attention to test-taking skills The third approach describes a reasonable balance of instruction with curricular alignment and instruction in test taking skills, especially there is concerted effort to teach subject matter knowledge and test taking skills that will have broad utility to the students beyond this immediate examination situation Two important premises of undertaking the third approach are that (a) the assessment represents a good sample of the core curriculum, and (b) the core curriculum is worthy and important The fourth approach crosses alignment the line toward measurement-driven instruction

Crocker (2005) used various sources to develop his classroom strategies of proper test preparation (Campanile, 1981) Some of these strategies included: (a) demonstration of a positive attitude toward test preparation with students and parents, (b) building concentration endurance in test-like conditions without becoming fatigued or distracted, (c) practicing various test item formats with instructions and answer sheets as those on the tests, (d) timing practice tests, (e) modeling good problem-solving strategies (how to approach the test items, to determine the meaning of the question, to provide various ways of posing the same question, and to arrive at the correct answer), (e) practicing working through tests with various difficulty levels, (f) diagnosing any response patterns that needed to be corrected before testing through homework grading or class work, (g) building students‟ test-taking vocabulary, (h) explaining how score rubrics are used to award points for performance assessment items, providing examples of responses generating full, partial, or no credit, and helping students evaluate their responses

Crocker (2005) also suggested four criteria for determining if classroom instruction practice is ethical These four criteria are as follows: (a) Academic ethics: test preparation should be guided by the ethical cannons of the education profession, dealing with cheating, misrepresentation, and respect for intellectual property or work of others (Popham, 1991); (b) Validity: Test preparation should improve validity of test scores by allowing only students who have knowledge or partial knowledge of content being tested in an exercise to display that knowledge; (c) Transferability: Test preparation should provide student with skills that have applicability to a broad range of testing situations (Popham, 1992); (d) Educational value: test preparation that leads to improvement in student scores should simultaneously increase student mastery of the content domain tested (Popham, 1991).

Perceptions and Attitudes towards high-stakes standardized testing

According to Campbell (1967), perception is a process where one will form an impression about someone or something Based on the definition, perception is constructed as a result of individual observation towards certain things or events occurring around them which will produce certain perception The perception then affected their attitude towards certain matters or objects of attitudes

Lindsay and Norman (1977) stated that perception is the process by which organisms interpret and organize sensation to produce a meaningful experience of the world The researchers describe the definition into two categories, which are sensation and perception Sensation refers to the immediate, relatively unprocessed result of stimulation of sensory receptors in the eyes, ears, nose, tongue, or skin Sensations refer to the human sensory systems which function as the receptor of information regards certain matters or object observed by an individual It is the stimulation process where the input is transferred directly to the human brain On the other hand, perception refers to the one‟s ultimate experience of the world and typically involves further processing of sensory input Perception involved more process of thinking as a result of the information received from the sensory systems as regards certain things or events It is the output process where the judgments or beliefs were produced by an individual and it influenced the way they think and feel

Gardner (1985, p 9) stated that attitude is an evaluative reaction to some referent or attitude objects, inferred on the basis of the individual‟s beliefs or opinions about the referent Ajzen and Fishbein (2005, p 3) defined that attitude is a disposition to respond favorably or unfavorably to an object, person, institution, or event The definitions of attitudes has been varied as a result of continues studies done by researchers Even though the researchers have their own definition of attitudes, they still share the facts that attitudes guide and direct an individual‟s behavior Attitude is more to how human perform their thinking or their reaction toward certain things or matters Meanwhile, perception did not involve action as it is more to the way of human thinking Nevertheless, perception and attitudes are related as perception is the root of thinking which invoke certain judgments or beliefs that influence human attitude

6.2 Importance of test-takers’ perceptions and attitudes towards high-stakes standardized testing

Shohamy (2001a) argued that listening to the test-takers‟ voices about tests will lead to a better understanding about tests, especially for high-stakes tests which have powerful effects not only in students‟ educational lives, but also in their social lives That is, while high-stakes test takers are often considered passive participants in the testing process, their opinions provide information from a different perspective that might not be addressed by other stakeholders She also emphasized the importance of knowing students‟ perceptions towards tests as it can lead into information about the importance, the use, and the meaning of the tests to the students‟ lives In addition, Murray et al (2012) pointed out test-takers‟ attitudes toward test may have significant implications to test validity, which leads to evaluation of the appropriateness, meaningfulness, and usefulness of high-stakes tests According to Tsai & Tsou (2009), test-takers‟ perceptions are significant because they provide evidence of tests‟ effectiveness and one of the aspects in construct validity

Previous research has also suggested that learners‟ attitudes towards and perceptions of a test and its use can affect their motivation and performance on the test (Bachman & Palmer, 1996) Empirical research conducted by Brown and Hirshfeld (2008) demonstrated that meaningful relationships exist between students‟ conceptions of assessment and their academic achievement In higher education, students‟ perceptions of assessment are pivotal because assessment has a critical impact on quality of learning (Entwistle, 1991) Students will be motivated to prepare for the examination if they perceive it as pivotal; that is, the test-takers‟ perceptions of the examination determine the level of the effort they are willing to take (Wolf & Smith, 1995) In other words, how much effort will be put in the examination is based on the test-taker‟s perception, not merely judged by the stakes to which the examination may be attached.

TOEIC as a high-stakes standardized test in Vietnam

TOEIC (Test of English for International Communication) is the worldwide- standardized test of workplace English proficiency for non-native English speakers

With its widest score range, TOEIC claims it can measure all English levels (from beginning to advanced) which is proper for all working positions, from basic jobs to management thus this is the preferred English language assessment tool to measure the progress of leaners and help them to achieve their set-goals in his careers The TOEIC test was developed by Educational Testing Service (ETS), the United States of America in 1979 at the request of the Ministry of International Trade and Industry in Japan

TOEIC consists of two types of tests First is the TOEIC Listening and Reading Test, which consists of different comprehension assessment activities The comprehension is divided into two equal parts of 990 score in total One of these comprehension tests consists of listening comprehension and the other is reading comprehension The candidates obtain score on a scale of 5 to 495, which later adds to a total score of 990 The second type is the newer TOEIC Speaking and Writing Test, which consists of various parameters to measure the English skills of a candidate It includes pronunciation, vocabulary, grammar, fluency, overall coherence and structure, which make up for a score of 400 in total Candidates get separate scores for speaking and writing They also get an option to appear for the speaking test only, leaving the writing test It approximately takes 20 minutes to complete this test; however, the writing test takes 60 minutes Further, each of these tests comprises of 200 marks, which are then added to a total of 400

7.2 Use of the TOEIC test as a graduation requirement in Vietnam’s HE institutions

TOEIC Listening and Reading Test was introduced to Vietnam in 2001 by IIG Vietnam, the only organization authorized by ETS to provide this test TOEIC Speaking and Writing test was also provided five years later; however, it has not been as popular as the previous type so far In fact, when it is said about TOEIC in Vietnam, it normally refers to TOEIC Listening & Reading test (see Appendix 3 for a sample test) Also, for the purpose of the current study, TOEIC is short for TOEIC Listening & Reading (LR) test

First, TOEIC is used as a basis for recruitment and promotion in competitive labour markets such as information technology, airline industry and international trade Since 2010, the Ministry of Education and Training (MOET) in Vietnam has encouraged HE institutions to set English proficiency thresholds for undergraduates and to implement an English requirement policy for graduation The objective is to raise students‟ English language proficiency (ELP) and to better prepare students to cope with global competition and meet the ELP requirements of the workplace

Two years later, the MOET has advocated the use of standardized English proficiency tests as exit requirements by issuing the Circular No 05/2012/TT- BGDĐT, indicating that non-English major graduates need to have English proficiency equivalent to B1 (CEFR), 450 TOEIC, 450 TOEFL ITP, or 4.5 IELTS

As a result, the majority of HE institutions in Vietnam gradually have set a requirement for their students to achieve a satisfactory score on one of the recommended English language proficiency tests before graduation

The statistics show that approximately 120 HE institutions nationwide, over

30 universities in Hanoi alone, are using TOEIC scores to determine their students‟ graduation eligibility The cut-off scores set by most universities, such as National University of Civil Engineering, Hanoi Law University, Vietnam Maritime University and Thai Nguyen University, are 450 points Some universities require higher TOEIC scores for graduation For example, the Decision No 148/QĐ- ĐHBK-ĐTĐH dated 5 September 2017 by the Rector of Hanoi University of Science and Technology states that non-English majors need present the TOEIC certificate of minimum 500 points or equivalent certificates before being assigned a graduation thesis While this university and some HE institutions such as People's Security Academy and University of Transport Technology cooperate with the only ETS-authorized organization, IIG Vietnam, to organize TOEIC-taking examinations for their own students; others request their students to take the tests at IIG Vietnam headquarter by themselves These test-takers, however, get the same valid score certificate regardless of either way they take the tests.

Previous studies on the use of TOEIC as an exit requirement in Vietnam

There have been a couple of studies related to the use of TOEIC as a graduation requirement in HE institutions in Vietnam The first study entitled “The

TOEIC Test as an Exit Requirement in Universities and Colleges in Danang City, Vietnam: Challenges and Impacts” was conducted by Nhan (2013) Conducting this research at the very beginning of introducing the TOEIC certification exit requirement, the researcher found that unsolved challenges had hinted at the unfeasibility of implementing the policy at that moment The limited qualified human resources together with the lack of necessary materials and facilities and the time-constraint curriculum allocated to the teaching and learning of English at HE institutions make it unfeasible for students to achieve the required TOEIC score

Besides, a number of students with low levels of English or from poorer backgrounds may be disadvantaged in taking supplementary TOEIC preparation courses Then, it is more practical if HE institutions choose a locally based English test to decide graduation eligibility of currently final-year undergraduates

However, Nhan (2013) argued that in the long run, it must be confirmed that the use of certain English certification, including TOEIC certificates, as an exit requirement should remain an aim It is because companies have been more interested in using an internationally recognized standard or the TOEIC standards for recruitment; and Vietnam‟s HE institutions themselves need to be competitive among new foreign-invested international universities In that case, students need to be informed right at the start of their tertiary lives so that they allocate enough time and attention to improve English skills and practice for the English exit tests

English programs and teacher training should also be modified in a way that could encourage learners‟ production of the target language

The second study entitled “Impact of using TOEIC as an exit requirement at a public university in Vietnam” was carried out by Tran (2016) The result of the study showed that using TOEIC scores as a graduation requirement exerted strong impact on the English language education at a public university The first impact is on English language curriculum; accordingly, the teachers reformed the EFL curriculum and created the TOEIC-oriented program to prepare their students for exit exams The second impact is on classroom teaching and learning; that is, the teachers focus on teaching test-taking strategies and pay less attention to interactive and communicative activities.

Summary

Based on the elaboration in this chapter, it is clear that listening to test- takers‟ voices about tests will lead to a better understanding about high-stakes standardized tests, as test-takers are able to point out issues that might not be noticed from the view of other stakeholders In the current study, the TOEIC test required by some Vietnam‟s HE institutions is viewed as high-stakes because its result determines granting or denying of the academic degree This policy has been gradually implemented by numerous universities for seven years, and the time is long enough so that its effectiveness can be evaluated comprehensively The limited studies on the TOEIC exit standard in Vietnam, as mentioned above, either have focused only on academic factors which affect teaching and learning for the TOEIC exit test or have not yet given a definite answer to the appropriateness of the policy from the test-taker‟s perspective because one of them was carried out at the pilot implementation stage How crucial students perceive the English exit standard policy and the TOEIC as a graduation requirement remains uncertain, which creates the research gap that the present study should fill in.

METHODOLOGY

Research design

This survey research had a mixed methods design, combining both quantitative and qualitative procedures As Creswell (2003) indicated, “with the development and perceived legitimacy of both qualitative and quantitative research in the social and human sciences, mixed methods research, employing the data collection associated with both forms of data, is expanding” (p 208)

In addition, mixed methods of quantitative and qualitative procedures were conducted in order to increase the depth and breadth of the study, which assisted the researcher to see the research matters from different angles, so that a deeper understanding of the topic could be obtained The triangulation of data that were obtained through different methods also increased the reliability of the findings (Allwright & Bailey, 1991)

Among a variety of mixed methods designs, Explanatory Sequential Strategy in which a qualitative data collection builds directly on the result of the initial quantitative data collection is chosen (Creswell, 2014) Since Explanatory Sequential Strategy allows a thorough understanding of the quantitative results by explaining them with qualitative data, it is believed to provide the researcher with a satisfying explanation for participants‟ common responses In the present research, a survey questionnaire was used as a primary instrument of quantitative data collection in order to find out how non-English majors perceive the use of TOEIC scores as a graduation requirement; and then the qualitative part employed the method of interview to better understand the reasons for some of the survey

Research site

This study was conducted at one of the oldest and most prestigious TOEIC preparation centers in Hanoi There TOEIC preparation classes are divided into four levels, including basic grammar class, PreTOEIC class for the score target of 350, TOEIC A class for the score target of 450+, and TOEIC B class for the score target of 600+ This center was chosen as the research site because thousands of students from different universities throughout Hanoi had participated in its evening classes in order to achieve the TOEIC target scores for graduation requirement Also, the researcher has worked as a part-time English teacher at this center for five years and taught many of the participants of the present study Conducting research in this familiar site brings the researcher considerable advantages in terms of access to the research site, persuading learners to participate, and establishing rapport with participants Such advantages, according to Marshall and Rossman (2011), can contribute the quality of the research data and research findings.

Participants

For the purpose of this study, based on the center‟s learner database, the researcher selected participants who met certain criteria First, they were non- English majors from HE institutions where the TOEIC exit requirement policy had been adopted Second, these students had taken the official TOEIC tests at least once for the past six months and had valid TOEIC score reports by the time of data collection If they had taken the official TOEIC tests a longer time before, they might not have remembered their testing experiences exactly In all, 205 eligible students were invited to participate in an email survey; however, a few students did not reply to the invitation, resulting in a final number of 197 participants They were in academic years at university from the second to the fourth, of both genders, and varied in English proficiency according to their TOEIC scores The researcher would also find out whether these features, as shown at Table 1, probably affected participants‟ perceptions of the research issues

Table 1: Demographic information of participants Items Number (n = 197) Percentage (%)

Processing the survey data, the researcher did not find any illogicality among answer choices of every single questionnaire that is doubtful to be clarified

Therefore, researcher decided to select the interviewees first on a voluntary basis

Of the 197 questionnaire respondents, 36 students volunteered to take part in a follow-up interview by providing their contact information in reply emails Due to the researcher‟s time constraint and because of the similar results from several of these questionnaires, eight finally selected students represented different majors and universities, both genders, and various TOEIC scores, including those who do and do not support the use of the TOEIC as a requirement for graduation However, the representation was not equal across the criteria but was partly dependent on the number of volunteers and the ratio of the responses.

Data collection methods

Questionnaires can be distributed in large numbers and to varied groups of respondents It is more economical to use questionnaires, compared with doing interviews, when the number of subjects is large for instance (Bryman, 2008;

Walliman, 2011) Moreover, the respondents usually have an option about whether to reveal their personal information or not, thus avoiding the risk of exposing their identities Responses in a questionnaire are also easier to quantify and process with the aid of certain software, making it possible for the researcher to see and present results in a quick and conspicuous way In the present study, the researcher emailed the soft-version questionnaires in the attachment to all the participants, which was not only more economical but also more convenient for them to complete the survey than the fact that the hard copies were delivered directly to them

A variety of items may appear in a questionnaire, ranging from closed questions to open questions, from dichotomous to multi-option selection items, and Likertscale items with various numbers of points along the continuum (Brown,

2000) The closed questions or selection items yield data that are easier to quantify, compare, and classify, since the number of choices is limited and predetermined (Creswell, 2008a), whereas the open questions reserve more space for respondents to express themselves „in their own terms‟ (Brown, 2000; Punch, 2009) The advantage of using open questions, then, is that the researcher may obtain depth in responses, and the responses may include the ideas or categories that predetermined classifications may have overlooked However, it is also possible that the responses obtained are not as detailed as expected by the researcher

When the responses are detailed, they are more difficult to quantify, compare, and classify After these advantages and disadvantages being taken into consideration, the data collection from questionnaires of this study was conducted according to the basic steps as follows:

Designing the questionnaires: Besides demographic information items of participants in the first part, the researcher designed a 13-statement questionnaire in Vietnamese with five-point Likert scale items, ranging from totally disagree to disagree, neutral, agree, and totally agree (see Appendix 1) Table 2 gives detailed explanations for the question statements

Table 2: Structure of survey questionnaire Research Question 1: Perceptions of the English requirement policy for graduation

Purpose of investigation Question statements

Necessity of the policy 1 It is necessary for universities to require their students to pass a certain score in an ELP test before graduation

Goal of the policy 2 Passing an exit English standard ensures that students have minimum ELP for their future work

Policy‟s impact on English learning motivation

3 I study English harder when my university adopts the exit English requirement policy

4 I would still be motivated to study English at university even if there were no English graduation requirements

Policy‟s impact on ELP 5 I would still be motivated to prepare to take test-taking motivation standardized English language proficiency tests even if there were no English graduation requirements

Research Question 2: Perceptions of the appropriateness and effectiveness of the TOEIC as an exit test

Purpose of investigation Question statements

Impact of TOEIC preparation on ELP

6 Preparing for the TOEIC test helps me improve my ELP

Universities preparing their students for TOEIC exit tests

7 English learning curriculum in my university is sufficient for me to take the TOEIC exit test

8 My university often organizes TOEIC preparation classes for students before exit English exams

Appropriateness of cut- off scores

9 Required TOEIC cut-off score is achievable without difficulty

Validity of TOEIC scores 10 TOEIC scores accurately reflect my ELP

Benefit of having a TOEIC certification

11 If I have a high-score TOEIC certificate, I will feel more confident to apply for a job after graduation

12 TOEIC speaking and writing tests should be required for graduation

13 If there were no English exit requirements, I would still be motivated to prepare for and take the TOEIC test

Piloting the questionnaires: The questionnaires were given directly to 10 students of a TOEIC preparation class at the center, who were not included as participants of the study The researcher was present while students filled in the questionnaires to provide explanations and clarifications should they be required

The pilot stage revealed no problems with the comprehension of question statements in the questionnaire and assisted in improving the validation of the methods and findings in this study (Creswell, 2008a)

Conducting the survey: The official questionnaires in the attachment were emailed to 205 eligible participants, including an appealing invitation to the survey in the text The researcher reminded the students who attached incomplete questionnaires or did not reply after a week of filling out the survey Finally, 197 valid questionnaires in the form of soft copies (96.1 %) were collected during twelve days They were all numbered from 1 to 197 and saved in a folder for convenient access by the researcher whenever necessary

The interview is a good way to obtain subjects‟ opinions and/or feelings about a certain topic, event or action in a highly personal and detailed level (McDonough & McDonough, 1997; Punch, 2009) The conversational nature of an interview enables the interviewer to adjust the questions, shift the order of questions, omit some questions, or probe more after certain questions right on the spot according to the flow of the interview and the interviewee‟s responses, especially in the less structured forms of interviews (Roulston, 2010) The interviewees can also clarify their points and express themselves more extensively than when completing a questionnaire Moreover, a face-to-face interview provides the interviewer with valuable opportunities to “read between and beyond an interviewee‟s words”: not just what they say, but how they say it, their intonations, facial expressions, or their gestures (Walliman, 2011) By paying proper attention to these signals, an interviewer may learn more than simply what an interviewee is telling them; they may also help determine whether to probe more into certain parts of the interview to see what the interviewee truly means or actually wants to say In this sense, interviews provide richer and more profound responses than any kinds of questionnaires

For a semi-structured type of interviews employed in this research, the interviewer does not have to follow a number of set questions in a rigid order

Instead, a number of core questions or themes are suggested, and are to be covered during the interviews However, the interviewer may provide explanation, elaborate on a specific question, or ask additional questions to help the interviewee to understand fully and answer the questions adequately Based on the response to a certain question, the interviewer may decide to investigate deeper into a matter by asking probe questions (Bogdan & Biklen, 2007), seeking more information or more detailed response from the interviewee on a point of interest The order of the core questions is also not predetermined, but it depends on the flow of the interview This kind of interviews guarantees the coverage of the main questions the interviewer is seeking answers for, thus ensuring that the data needed are collected, and can be used for comparison and categorization The interview process of the present study consisted of the following basic steps

Designing the interview: A list of interview questions in Vietnamese was prepared, based on prominent features discovered from questionnaire analysis (see Appendix

2) In response to the first research question, the interview questions further explored the factors influencing students‟ perceptions of the English graduation requirement policy, such as policy goals and policy impact on students‟ ELP and English learning Moreover, asking the test-takers about their experiences and opinions of preparing for and taking the TOEIC exit tests, such as test design, test purposes and score validity, provided an explanation for the second research question

Conducting the interviews: The interviews were one-on-one and audio recorded in the center‟s meeting room, but the researcher also took notes on relevant points for further elaboration during the interviews The researcher conducted the interviews in 7 steps as suggested by McNamara (1999): (1) choose a setting with little distraction; (2) explain the purpose of the interview, (3) address terms of confidentiality, (4) explain the format of the interview, (5) indicate how long the interview usually takes, (6) tell the interviewees how to get in touch with the interviewer later if they want to, and (7) ask if the interviewee had any question before the interview.

Data analysis procedures

The data from valid questionnaires were processed through SPSS

Descriptive statistics on the percentages of the participants who agree and disagree with question statements, their mean score and standard deviation were summarized so that characteristics of the respondents were estimated, which could advance the knowledge of educational policy makers, university administrators, and the general public about the participants‟ attitudes toward the English requirement policy for graduation and the use of the TOEIC as an exit test In addition, cross-tab analyses based on Chi Square test explored possible correlation between the students‟ ELP measured by their TOEIC scores and their perceptions of the research issues

Specifically, for the first research question, P-value revealed whether there was a significant difference between higher- and lower-proficiency groups in their opinions about the necessity of the exit English requirement policy Concerning the second research question, P-value examined whether the students of higher and lower ELP view the impact of TOEIC preparation on ELP, the appropriateness of cut-off scores, added productive skills testing, and TOEIC test-taking motivation differently

The audio recordings of the interviews were transcribed by the researcher, and then the transcripts were used as data for analysis - a practice encouraged by Kvale & Brinkmann (2009) However, only selected excerpts from the interviews that were to be quoted were translated into English Each participant was provided with a copy of his/her transcript for checking information he/she had provided, and they had the rights to delete any given information they felt uncomfortable with from the recordings

First, the researcher read the transcripts in their entirety several times to obtain a holistic impression of the responses and to determine a preliminary list of analytical categories that had the potential to provide answers to the research questions For example, one of the categories was policy goals, intended to examine the students‟ viewpoint on the goals of the English exit requirement policy Further analysis was conducted to identify different variants under each analytical category

For example, under the analytical category, policy‟s impact on ELP, three variants emerged from the data: positive, negative, and no impact Each variant was defined and described in the coding scheme following the procedures for analyzing semi- structured interviews proposed by Schmidt (2004) The coding scheme was subsequently revised to incorporate new analytical categories or variants and to remove those that were redundant or irrelevant The researcher then coded the entire dataset using the qualitative data analysis software NVivo to make the processes of coding and identifying the relationships among themes more convenient.

RESULTS AND DISCUSSION

Research Question 1: What are students‟ perceptions of the English requirement

The students were asked to respond to five statements about their viewpoints regarding the necessity of the exit ELP requirement policy, the goal of the policy, and the policy‟s impact on their English learning The participants‟ responses ranging among five Likert scales are grouped mainly into agreement or disagreement (see Table 3) Accordingly, the majority (80.2%) supported the policy and thought that universities should require students to achieve a certain level of English language proficiency before graduation The reason for their support was that it would ensure the minimum ELP for their future work (71.5%) Also, most of the students admitted that they studied English harder thanks to the adoption of the exit English requirement policy (91.3%), but surprisingly, nearly 80 percent of the students indicated that they would still be motivated to study English even if the policy were not in place, implying that the policy is not the only motivator for many students However, only 75 out of 197 respondents (about 38.1%) indicated that they would still be motivated to prepare and take standardized ELP tests, regardless of the presence of the policy This result suggests that the students are aware of the importance of learning English, but they are not really interested in being assessed by a test unless it is compulsory

Table 3: Students’ opinions about the exit English requirement policy

It is necessary for universities to require their students to pass a certain score in an ELP test before graduation

Passing an exit English standard ensures that students have minimum ELP for their future work 71.5 22.1

I study English harder when my university adopts the exit English requirement policy 91.3 8.7

I would still be motivated to study English at university even if there were no English graduation requirements

I would still be motivated to prepare to take standardized English language proficiency tests even if there were no English graduation requirements

To explore whether the students‟ opinions about the exit English requirement policy are related to their English language proficiency, the survey respondents were split into two proficiency groups, higher and lower (falling above and below respondents‟ mean TOEIC score of 523) It should be noted that these were relative proficiency groups only for the convenience of the analysis and should not be interpreted as representing different proficiency levels based on the TOEIC scores of a more representative sample The mean difference in the total TOEIC scores between the higher- and lower-proficiency groups was 156.9 total score points, and the difference was statistically significant

Table 4: Students’ opinions about the exit English requirement policy by their ELP

Opinions about the exit English requirement policy

Cross-tabulations were performed to determine if English language proficiency could predict students‟ viewpoints for or against the English graduation requirement policy Table 4 shows that no significant difference was found between the higher- and lower-proficiency groups in their opinions about the necessity of the policy (χ 2

(1) = 0.76, p >0.05) That is, the students have a positive attitude toward the English requirement policy for graduation regardless of their English language proficiency measured by their TOEIC scores

The interview data extended the survey results and helped illuminate potential factors influencing students‟ perceptions of the English graduation requirement policy The students‟ responses centered on three major themes: policy goals, policy‟s impact on ELP, and policy‟s impact on English learning motivation

Table 5 indicates the number of students who commented on each of the subcategories within each main category

Table 5: Interviewees’ perceptions of the exit English requirement policy Main category Sub-category Students (n = 8)

To ensure students have minimum

To help improve students‟ ELP 6

To motivate students to study English 7

To prepare students for further study 2

To prepare students for future employment 7

Policy impact of English learning motivation

The students made numerous comments on what they thought the exit English requirement policy was designed for Collectively, the data showed that the students had a fair understanding of the policy goals and were generally in support of its implementation Four students considered that the policy aimed to ensure that graduates have the minimum level of English proficiency for communication, while six interviewees thought that the policy could help students improve their ELP

Seven students believed that the policy was put in place to motivate students and to prepare them for future employment; two students thought that the policy could help prepare students for their future studies

When asked about the policy‟s impact on English learning, the interviewees largely commented positively about its influence on their ELP “I think it‟s a necessary requirement, because when we go on the job market, English is always required So it encourages students to learn English sooner in order to achieve English proficiency for future employment,” commented Student 6 (higher proficiency, sophomore) In contrast, several students felt that the policy had limited or no impact on their ELP or English learning motivation A few pointed out that the policy‟s impact was short term in nature: “I‟m a final-year student now, so the requirement pushed me to study English for a while Honestly, I am a lazy student

After I passed the required TOEIC score for graduation, I lost my motivation to study English,” said Student 8 (lower proficiency, senior) Another factor that influenced the students‟ views about the policy pertained to their perception that an appropriate cut-off score for graduation was lacking

Student 1 (higher proficiency, senior) reported:

I don‟t think the policy is meaningful nor has impact on students‟ learning motivation because the requirement is so low In my university, the requirement is

450 on the TOEIC As you know, even if you don‟t prepare for the test and you just guess randomly, you might be able to get 200 or 300 points As far as I know, many companies presently expect minimum 600-point TOEIC certificates from candidates If the exit requirement were higher, I think it would help students

TOEIC certificate not only for graduation but also for job application, to kill two birds with one stone

Similar concerns about the inadequate cut-off scores were raised in the responses of three students who generally supported the policy but were suspicious about the positive washback that the policy could induce For instance, Student 3 (lower proficiency, junior) stated:

Well, many of my schoolmates agree that a higher graduation requirement would really motivate us to study English Since the requirement in my university is so low, most students think that they don‟t need to work hard and they‟ll be able to meet the requirement anyway So, although I think it‟s a good policy, it doesn‟t really help motivate students

Interestingly, Student 3, whose most recent TOEIC score was 460 and was one of the least proficient students in the entire interviewee group, seemed confident that most students at his university, which was a university of law, would not have much difficulty meeting the requirement In fact, several universities might set a cut-off score of 450 or even lower on the TOEIC to avoid having a small passing rate However, the students‟ comments above reveal that an inadequate cut score could undermine the goals of the policy and adversely cause students to lose the incentive to work hard, knowing that the threat of failing was minimal Moreover, as Student 2 (higher proficiency, junior) mentioned, “When employers‟ expectations are much higher than universities‟ exit requirements, passing the cut- off scores is not really meaningful.”

Research Question 2: What are test-takers‟ perceptions of the appropriateness

Given that students have multiple standardized ELP tests to choose from for meeting the exit requirements, it is logical to assume that students who choose to take the TOEIC tests would perceive the use of the TOEIC as an exit test more favorably To determine whether this assumption holds, the following analysis included responses of all participants (n7) who might take the TOEIC tests for

Table 6 shows the frequency and percentage of students who agreed or disagreed with each statement related to their perceptions about the use of the TOEIC test The majority of the students (85.6%) reported that preparing for the TOEIC had a positive impact on their ELP Regarding universities‟ preparing their students for English exit tests, the minority of students (37.1 %) stated that their English curriculum was conductive to helping them pass TOEIC exit tests It means that most of them might not learn the English language programs which would be tested before graduation

Table 6: Survey respondents’ perceptions of the TOEIC as an exit test

Preparing for the TOEIC helps me improve my ELP 85.6 14.4

English learning curriculum in my university is sufficient for me to take the TOEIC exit test 37.1 62.9

My university often organizes TOEIC preparation classes for students before exit English exams 57.5 42.5 Required TOEIC cut-off score is achievable without difficulty 68.2 22.8

TOEIC scores accurately reflect my ELP 63.7 36.3

If I have a high-score TOEIC certificate, I will feel more confident to apply for a job after graduation 78.5 21.5 TOEIC speaking and writing tests should be required for graduation 52.6 47.4

If there were no English exit requirement, I would still be motivated to prepare for and take the TOEIC 64.5 35.5

Surprisingly, only about half of the students (57.5%) reported that their universities often organized TOEIC preparation classes for students before exit English exams The others probably had to prepare themselves for this high-stakes test, and that might be the reason why they flocked into specialized TOEIC preparation centers These results revealed that few universities got their students prepared for the exit TOEIC tests, and that the exit TOEIC requirement had limited impact on the English language education in some HE institutions, inconsistent with the previous study by Tran (2016) Regarding whether the cut-off score is easily achievable, nearly 70 percent of respondents, however, confirmed to pass the required standard without difficulty

Students‟ views about the validity of the test scores were also somewhat mixed A slight majority of respondents (63.7%) indicated that the test scores accurately reflected their level of ELP, and the other half disagreed The respondents‟ differing views on score validity may have been influenced by the exit requirement policy that required only TOEIC listening and reading test scores

Students who did not show confidence in the validity of the scores may feel that, as a measure of receptive (listening and reading) and not productive (speaking and writing) skills, the TOEIC LR test could not fully reveal their language competency

The result could also reflect an issue of how the statement was phrased Had the students been asked about whether the scores demonstrated their English listening and reading abilities in the workplace, the results could have been different

However, as many as eighty percent of students still agreed that they would feel more confident to apply for a job after graduation if they had a high-score TOEIC

LR certificate This implies that it is not easy to achieve impressive TOEIC LR scores, and this standardized certificate has its own value

Students‟ perceptions regarding the requirement of TOEIC speaking and writing tests for graduation were divided, with slightly more than half (52.6%) admitting that productive skills were important and should be required It was unsurprising to note that the students‟ mean TOEIC LR scores differed significantly between the two groups of students who were in favor or not in favor of the requirement of productive skills The results indicated that students with better listening and reading skills as measured by the TOEIC test had a significantly stronger preference for the requirement of an adequate level of speaking and writing proficiency before graduation (see Table 7) Students with lower receptive skills are less likely to pass the graduation requirement, and thus they might have a fear of adding another obstacle if productive skills are also required

When asked about their test-taking motivation, almost 65% of the students responded that they would be motivated to prepare to take the TOEIC tests even if they were not required to pass an English proficiency test before graduation This high level of test-taking motivation demonstrated that the students believe in the utility of the TOEIC test scores for purposes other than passing the exit requirement and confirmed the popularity of the TOEIC in comparison with other standardized English language tests

The second part of Research Question 2 addressed whether the students of higher and lower ELP view the appropriateness and effectiveness of the TOEIC as an exit test differently Four separate cross-tab analyses were performed to determine if the perception differences exist (see Table 7) The analysis showed that the higher- and lower-proficiency students did not differ in their views on the use of the TOEIC to improve their ELP, χ 2 (1) = 0.62, p = 0.43 In general, the majority were positive about its impact on language learning An insignificant difference was also found for students‟ perceptions on the validity of the TOEIC scores, χ 2 (1) 1.53, p = 0.21

Table 7: Survey respondents’ perceptions of the use of the TOEIC as an exit test by their ELP

TOEIC helps improve my ELP

TOEIC Speaking & Writing should be required

I would take TOEIC without requirement

With regard to the requirement of the TOEIC speaking and writing test scores for graduation, results of the cross-tab analysis indicated a significant difference between two proficiency groups, χ 2 (1) = 17.43, p < 0.001 The odds ratio effect size indicated that the higher-proficiency group was 3.4 times more supportive of requiring speaking and writing tests compared to the lower- proficiency one A significant difference between proficiency groups was also found for test-taking motivation, χ 2 (1) = 15.79, p < 0.001 The odds ratio effect size indicated that higher-proficiency students were 3.5 times more likely to prepare to take the TOEIC regardless of the presence of the policy, compared to the lower- proficiency ones

The interviewees were asked to discuss their experiences and opinions about preparing for and taking the TOEIC tests and the use of the TOEIC as an exit test

To help answer the research question, the discussion focused on students‟ opinions about the use of the TOEIC as an exit test Students‟ perceptions involved three major themes: test design, test purposes (other than graduation requirement), and score validity Table 8 summarizes the students‟ responses

When responding to the question about the utility of the TOEIC as an exit test, the student interviewees generally commented positively, specifying that the test has good design features Specifically, six students believed that the test content is authentic and the TOEIC is a good measure of workplace English Others stated that the listening test reflects real-world English language use because it includes various native accents A few students reported that the TOEIC is a much better and preferred test for graduation requirements Three students felt that the reading test is too long and that they had difficulty finishing the test in time

Table 8: Interviewees’ perceptions of the TOEIC as an exit requirement Main category Sub-category Number of students

Score validity Scores reflect actual ELP 5

Scores do not reflect actual ELP 2 The second major theme pertains to the students‟ purposes for taking the test, besides meeting the graduation requirement The interviewees reported that obtaining a satisfactory score on the TOEIC before graduation could simultaneously serve multiple purposes, such as qualifying for certain jobs, converting into credit points (for example, 600 points of TOEIC equivalent to 10 marks for all English credits), or simply assessing one‟s ELP One sample quote follows (Student 7, higher proficiency, senior):

TOEIC is now a very popular standardized test in Vietnam, so I decided to try it for my graduation requirement first Then I had a TOEIC certificate of 600 points and above, the marks of all of my English credits were revised to A+ Also, since I‟m about to graduate, the test certificate is very helpful when I apply for jobs

Four students added that the TOEIC test assesses not only workplace English but also language use in real-world contexts These students were sensitive to what is assessed in the test and considered that the test could also measure their general ELP Student 5 explained in detail (lower proficiency, sophomore):

CONCLUSION

Recapitulation

This study examined TOEIC test takers‟ perceptions of and attitudes for the English requirement policy for graduation and the use of the TOEIC as an exit test in some HE institutions in Vietnam The survey results and the interview data collectively showed that the majority of the students were in favor of the policy and were positive about the use of TOEIC scores to determine students‟ graduation eligibility, regardless of their levels of ELP measured by TOEIC scores The TOEIC test scores were also perceived favorably by the respondents as having relatively high levels of reliability and validity; however, the cut-off scores set by some universities might be too low to bring about positive washback

The survey respondents generally believed that preparing to take the TOEIC tests was helpful for improving their ELP and acknowledged that there could be added value in requiring students to take the TOEIC Speaking and Writing tests

Especially, higher-proficiency group measured by TOEIC listening and reading scores was much more supportive of requiring productive skill tests compared to the low-proficiency one; however, the interviewees requested that additional teaching and learning resources should be provided to help students improve their speaking and writing proficiency in order to succeed in all the four English language skills

Students‟ perceptions of the purposes of the TOEIC tests are in line with the intended uses of the TOEIC for preparing test takers to gain a competitive advantage in the job market The results of the study have provided an important piece of empirical evidence in support of the use of the TOEIC tests in Vietnam‟s

HE context, consistent with the conclusion by Nhan (2013) that “in the long run, the been more interested in using an internationally recognized standard for recruitment” Also, it confirms the effectiveness of this requirement, which fills the gap of her research when she pointed that “the effectiveness of the policy in Danang City is yet to be seen” at the pilot stage.

Implications of the study

Based on the research finding that an inadequate cut-off score could adversely cause students to lose the incentive to work hard, it is highly recommended that MOET policy makers and university decision makers periodically should review the cut-off scores to ensure that the requirements are appropriate for their students and the intended purpose of the exam In addition, with the growing trend of using the TOEIC as a university gate-keeping test, more research that investigates this high-stakes standardized test use across different contexts is called for In light of the study results, future research will benefit from exploring individual learner factors (e.g., motivation, relationship between attitudes and gain scores), educational background (e.g., academic majors), and social background (e.g., relationship between TOEIC scores and job performance) Also, that Vietnamese graduates are proficient in the four English skills may be an indispensable requisite in the future, so further research should explore the social, educational, and economic impact of requiring the TOEIC Speaking and Writing tests and issues that could arise as a result of such a requirement

In high-stakes testing situations such as university exit tests, there is also a need to inform students about the test design, intended use, and score interpretations When students are well informed of the test practices, they can better establish a link between their learning goals and the assessment tasks, and positive washback can be promoted Moreover, policymakers should keep in mind that students‟ perceptions about test use can play an important mediating role in policy implementation As the study results revealed, students‟ beliefs may not always be consistent with the intentions underlying the ELP testing policy required of them Future research should continue to examine test-taker perceptions of the impact of exit tests on English learning, focusing on individual students, their learning goals, and their understanding of test scores Reasons for possible inconsistencies between the intended policy objectives and student perceptions should be identified, discussed, and adequately addressed to successfully achieve the intended purposes of an ELP testing policy.

Limitations of the study

Similar to any research study, the present study has some limitations The first limitation lies in its scale where there were only 197 student participants

Although they came from different universities, their views may not be representative of all non-English majors from hundreds of Vietnam‟s HE institutions which have employed the TOEIC exit standard requirement, because it was a case study at a center in Hanoi Secondly, when conducting research on the perceptions of a high-stakes standardized test, it may have been better if the opinions of other stakeholders such as teachers and university administrators had been simultaneously considered so that the result comparison could help paint an overall picture of English graduation requirements at tertiary levels Finally, the appropriateness and effectiveness of the use of TOEIC as an exit condition could have evaluated more thoroughly if some of these participants had been graduates and joined the job market, or even they had been working part-time in an international environment at the research time

1 Ajzen, I & Fishbein, M (2005) The Influence of Attitudes on Behavior In

Albarracín, D., Johnson, B.T and Zanna, M.P (Eds.), The Handbook of Attitudes Erlbaum, Mahwah, 173-221

2 Allwright, D & Bailey, K M (1991) Focus on the Language Classroom:

An introduction to classroom research for language teachers Cambridge:

3 Amrein, A.L & Berliner, D.C (2002, March 28) High-stakes testing, uncertainty, and student learning Education Policy Analysis Archives, 10

(18) Retrieved March 13, 2011, from http://epaa.asu.edu/epaa/v10n18/

4 Bachman, L F (1990) Fundamental Considerations in Language Testing

5 Bachman, L F., & Palmer, A S (1996) Language testing in practice

Oxford, England: Oxford University Press

6 Bogdan, R C & Biklen, S K (2007) Qualitative Research for Education:

An Introduction to Theories and Methods (5th ed.) Boston, MA: Pearson

7 Brown, J D (2000) Using Surveys in Language Programs Cambridge:

8 Brown, G T L., & Hirschfeld, G H F (2008) Students' conceptions of assessment: Links to outcomes Assessment in Education: Principles, Policy

9 Bryman, A (2008) Social Research Methods (3rd ed.) Oxford: Oxford

10 Campanile, P (1981, March) Evening up the score Instructor, 58-59

11 Campbell, D T (1967) Stereotypes and the perception of group differences American Psychologist, 22(10), 817-829

12 Creswell, J W (2003) Advanced Mixed Methods Research Designs (Ch 8)

In Tashakkori, A & Teddlie, C (Eds.) Handbook of Mixed Methods in

Social and Behavioral Research (pp.209-240) Thousand Oaks, CA: Sage

13 Creswell, J W (2008a) Educational Research: Planning, Conducting, and

Evaluating Quantitative and Qualitative Research (3rd ed.) Upper Saddle River, NJ: Pearson

14 Creswell, J W (2014) Research Design: Qualitative, Quantitative, and

Mixed Methods Approachs (4th ed.) The US: Sage Publications

15 Crocker, L (2005) Teaching For the Test: How and Why Test Preparation Is

Appropriate In R P Phelps (Ed.), Defending standardized testing (pp 159–

174) Mahwah, NJ, USA: Lawrence Erlbaum

16 Deci, E L., & Moller, A C (2005) The concept of competence: A starting place for understanding intrinsic motivation and self-determined extrinsic motivation In A E Elliot & C Dweck (Eds.), Handbook of competence (pp

17 DHBKHN (2017) Quyet dịnh so 148 /QĐ-ĐHBK-ĐTĐH ngay 5 thang 9 nam 2017 quy dinh chuan tieng Anh dau ra doi voi sinh vien dại hoc he chinh quy Khoa 62 [Decision No 148/ QĐ-ĐHBK-ĐTĐH dated 5 September 2017 defines the exit English requirement for regularly trained students of the 62 nd academic course]

18 Entwistle, N.J., & Entwistle, A (1991) Contrasting forms of understanding for degree examinations: The student experience and its implications, Higher

19 ETS (2017) Who we are Retrieved June 1 st , 2017, from http://www.ets.org/about/who/

20 Finn, C E J (1991) We must take charge: Our schools and our future New

21 Gardner, R C (1985) Social psychology and second language learning: The role of attitudes and motivation London: E Arnold

22 Ha, T T D (2007) Nang cao hieu qua giang day tieng Anh o bac dai hoc nhin tu goc do giao vien chu nhiem [Raising English Language Teaching standard at tertiary level from teachers‟ perspectives] In V T P A (Ed.), Giang day tieng Anh bac dai hoc tai Viet Nam: Van de va giai phap [Teaching English at tertiary level in Vietnam: Problems and solutions] (pp

9-17) Ho Chi Minh City: Nha xuat ban Dai hoc Quoc gia Thanh pho Ho Chi Minh [Hochiminh National University Publisher]

23 He, L., & Shi, L (2008) ESL students‟ perceptions and experiences of standardized English writing tests Assessing Writing, 13(2), 130-149 doi:10.1016/j.asw.2008.08.001

24 Hughes, A (1989) Testing for Language Teachers Cambridge: CUP

25 Kohn, A (2000) Burnt at the high stakes Journal of Teacher Education, 51,

26 Kvale, S (1996) Inter Views: An introduction to qualitative research interviewing Thousand Oaks, CA: Sage

27 Kvale, S & Brinkmann, S (2009) Interviews – Learning the Craft of

Qualitative Research Interviewing Thousand Oaks, CA: Sage

28 Lewis, R J (1999) 'Reliability and Validity: Meaning and Measurement'

Paper presented at Society for Academic Emergency Medicine, Boston, Ma

Available: www.ambpeds.org/ReliabilityandValidity.pdf (10 September

29 Lindsay, P H., & Norman, D A (1977) Human information processing: An introduction to psychology New York: Academic Press

30 Madaus, G E (1988) The influence of testing on the curriculum In L N

Tanner (Ed.), Critical issues in curriculum: The eighty-seventh yearbook of the National Society for the Study of Education (pp 83-121) Chicago:

31 Marshall, C., & Rossman, G B (2011) Designing Qualitative Research (5th ed.) Thousand Oaks, CA: Sage Publications

32 McNamara, C (1999) General Guidelines for Conducting Interviews,

Authenticity Consulting, LLC, Retrieved from: http://www.managementhelp.org/evaluatn/interview.htm

33 Messick, S (1998) Test validity: A matter of consequence Social Indicators

34 MOET (2012) Thong tu so 05/2012/TT-BGDĐT cua Bo GDĐT quy dinh trinh do tieng Anh cho sinh vien tot nghiep dai hoc khoi nganh khong chuyen ngu co hieu luc thi hanh tu thang 4-2012 [Circular No 05/2012/TT-BGDĐT was issued by the Ministry of Education and Training (MOET) regulating the English proficiency for non-English major graduates, effective as from April, 2012]

35 Murray, J C., Riazi, A., & Cross, J L (2012) Test candidates‟ attitudes and their relationship to demographic and experiential variables: The case of overseas trained teachers in NSW, Australia Language Testing, 29(4), 577-

36 National Research Council (1999) High stakes: Testing for tracking, promotion, and graduation Washington, DC: National Academy Press

37 Nhan, T (2013) The TOEIC test as an exit requirement in universities and colleges in Danang City, Vietnam: Challenges and impacts International Journal of Innovative Interdisciplinary Research, 2(1), 33–50

38 Noble, A J., & Smith, M L (1994) Old and new beliefs about measurement driven reform Educational Policy, 8, 111-136

39 Popham, W J (1991) Appropriateness of teachers‟ test-preparation practices Educational Measurement: Issues and Practice, 10, 12-15

40 Popham, W J (1992) A tale of two test-specification strategies Educational

41 Punch, K F (2009) Introduction to Research Methods in Education

42 Roderick, M & Engel, M (2001) The grasshopper and the ant: Motivational responses of low-achieving students to high-stakes testing Educational Evaluation and Policy Analysis, 23 (3), 197-227

43 Roulston, K (2010) Reflective Interviewing: A Guide to Theory and

Practice Thousand Oaks, CA: Sage

44 Ryan, R M & Brown, K W (2005) Legislating competence: The motivational impact of high stakes testing as an educational reform In A E

Elliot & C Dweck (Eds.), Handbook of competence (pp 354-374) New York: Guilford Press

45 Schmidt, C (2004) The analysis of semi-structured interviews In U Flick,

E v Kardor, & I Steinke (Eds.), A companion to qualitative research (pp

46 Shanker, A (1993, May 23) Where we stand: Goals 2000 New York Times, p A7 Shohamy, E., Donitsa-Schmidt, S., & Ferman, I (1996) Test impact revisited: Washback effect over time Language Testing, 13 (3), 298-317

47 Shohamy, E (2001a) The power of tests: Critical perspective on the uses of language tests Essex, London: Pearson Education

48 Smith, J K., Smith, L F., & DeLisi, R (2001) Natural classroom assessment Thousand Oaks, CA: Corwin

49 Stecher, B M (2002) Consequences of large-scale, highs takes testing on school and classroom practice In L.S Hamilton, B.M Stecher, & S.P Klein (Eds.), Making sense of test-based accountability in education (pp 79-100)

50 Tasi, Y., & Tsou, C.-H (2009) A standardized English language proficiency test as the graduation benchmark: Student perspectives on its application in higher education Assessment in Education: Principles, Policy & Practice, 16(3), 319–330

51 Templer, B (2004) High-stakes testing at high fees: Notes and queries on the international English proficiency assessment market Journal for Critical

52 Thomas, R M (2005) High stakes testing: Coping with collateral damage

Mahwah, New Jersey: Lawrence Erlbaum Associates, Inc

53 Tollefson, J W (2002) Language policies in education: critical issues New

54 Ton, N N H., & Pham, H H (2010) Vietnamese teachers’ and students’ perceptions of Global English Language Education in Asia, 1, 48-61

55 Tran, T D (2016) Impact of using TOEIC as an exit requirement at a public university in Vietnam Bristish Council New Directions in English

Language Assessment: Standardized Testing and Proficiency Scales

56 Walliman, N (2011) Your Research Project: Designing and Planning Your

Work (3rd ed.) Thousand Oaks, CA: Sage

57 Wolf, L F., & Smith, J K (1995) The consequence of consequence:

Motivation, anxiety, and test performance Applied Measurement in Education, 8(3), 227-242

I am Le Thi Nu, a graduate student at University of Languages and International Studies - Vietnam National University, Hanoi I am currently conducting research to investigate non-English major students’ perceptions of the use of the TOEIC LC as an exit English test at universities I would be grateful for your time completing this questionnaire, and I assure you of the confidentiality of your responses

Thank you so much for your valuable support!

Please fill in the blank or tick () an appropriate option

6 Testing site □ University □ IIG Vietnam

Please state whether you agree or disagree with the following statements by putting a tick () in the appropriate column

(1) Totally disagree, (2) Disagree, (3) Neutral, (4) Agree, (5) Totally agree

No Statement Scale of agreement

A Perceptions of the English requirement policy for graduation

It is necessary for universities to require their students to pass a certain score in an ELP test before graduation

2 Passing an exit English standard ensures that students have minimum ELP for their future work

3 I study English harder when my university adopts the exit English requirement policy

I would still be motivated to study English in university even if there were no English graduation requirements

I would still be motivated to prepare to take standardized English language proficiency tests even if there were no English graduation requirements

B Appropriateness and Effectiveness of the use of TOEIC as an exit English test

6 Preparing for the TOEIC helps me improve my ELP

7 English learning curriculum in my university is sufficient for me to take English exit test

8 My university often organizes TOEIC preparation classes for students before exit English exams

9 Required TOEIC cut-off score is achievable without difficulty

10 TOEIC scores accurately reflect my ELP

11 If I have a high-score TOEIC certificate, I will feel more confident to apply for a job after graduation

12 TOEIC speaking and writing tests should be required for graduation

If there were no English exit requirements, I would still be motivated to prepare for and take the TOEIC tests

BẢNG CÂU HỎI KHẢO SÁT

Về việc áp dụng chuẩn đầu ra TOEIC tại các trường Đại học

Tên tôi là Lê Thị Nụ, học viên cao học tại trường Đại học Ngoại ngữ - Đại học quốc gia Hà Nội Hiện nay tôi đang tiến hành khảo sát ý kiến của các bạn sinh viên không chuyên tiếng Anh về việc áp dụng chuẩn đầu ra TOEIC tại các trường Đại học Các bạn vui lòng bớt chút thời gian giúp tôi trả lời trung thực các câu hỏi dưới đây Tôi xin cam đoan bảo mật toàn bộ thông tin khảo sát và chỉ sử dụng vào mục đích nghiên cứu

Phần 1: Thông tin chung về đối tượng khảo sát

Vui lòng điền thông tin vào ô trống hoặc đánh dấu () vào lựa chọn phù hợp

3 Sinh viên năm thứ mấy?

6 Hình thức thi TOEIC chuẩn đầu ra

□ Trường Đại học phối hợp với IIG Việt Nam tổ chức thi tại trường

□ Thi tại trụ sở IIG Việt Nam

7 Điểm TOEIC đầu ra theo yêu cầu của trường

8 Số lần đã thi TOEIC chính chức

9 Điểm TOEIC cao nhất đạt được

Phần 2: Nội dung khảo sát

Vui lòng cho biết mức độ đồng ý của bạn về các phát biểu dưới đây bằng cách đánh dấu () vào ô số phù hợp:

(1) Rất phản đối, (2) Phản đối, (3) Không có ý kiến, (4) Đồng ý, (5) Rất đồng ý

STT Ý kiến Mức độ đồng ý

A Sự cần thiết phải áp chuẩn đầu ra tiếng Anh cho sinh viên

Các trường Đại học cần yêu cầu sinh viên đạt điểm chuẩn nhất định trong một kỳ thi đánh giá năng lực tiếng Anh trước khi tốt nghiệp

2 Đạt chuẩn tiếng Anh đầu ra ở Đại học sẽ đảm bảo cho sinh viên có năng lực tiếng Anh tối thiểu trước khi đi làm

3 Việc áp dụng chuẩn đầu tiếng Anh đại học sẽ thúc đẩy sinh viên học tiếng Anh chăm chỉ hơn

Tôi vẫn tích cực học tiếng Anh ở Đại học ngay cả khi trường của tôi không yêu cầu chuẩn đầu ra tiếng Anh

Tôi sẽ vẫn tham gia một kỳ thi đánh giá năng lực tiếng Anh tiêu chuẩn quốc tế cho dù trường Đại học của tôi không yêu cầu chuẩn đầu ra tiếng Anh

B Sự phù hợp và Hiệu quả của việc áp dụng chuẩn đầu ra TOEIC

6 Quá trình luyện thi TOEIC giúp tôi cải thiện khả năng tiếng Anh của bản thân

7 Chương trình học tiếng Anh chính khóa tại Đại học đủ để tôi thi lấy chứng chỉ TOEIC đầu ra

8 Trường Đại học của tôi thường tổ chức các lớp luyện thi TOEIC cho sinh viên trước khi thi đầu ra

9 Tôi thấy không khó để thi đạt điểm chuẩn TOEIC đầu ra nhà trường yêu cầu

10 Điểm TOEIC đạt được đánh giá đúng khả năng tiếng Anh của tôi

Nếu tôi sở hữu một chứng chỉ TOEIC điểm cao tôi sẽ cảm thấy tự tin hơn khi xin việc sau khi tốt nghiệp

Nên bổ sung thêm 2 kỹ năng nói và viết TOEIC vào yêu cầu đầu ra tiếng Anh cho sinh viên thay vì chỉ kiểm tra 2 kỹ năng nghe và đọc hiểu như hiện nay

13 Tôi vẫn thi lấy chứng chỉ TOEIC nếu trường Đại học của tôi không yêu cầu chuẩn đầu ra TOEIC

1 What types of English certificates are accepted as a graduation requirement for your major at your university?

2 Why is the exit English requirement policy necessary/ unnecessary?

3 To what extent does the policy have effect on your ELP?

4 How are you motivated to learn English by the policy?

5 What do you know about the TOEIC test before attending TOEIC preparation courses at this center?

6 In case you have several options of standardized English language proficiency tests for graduation, why do you choose the TOEIC test?

7 Why do you think the cut-off score is appropriate/ inappropriate?

8 Could you give me some comments on the TOEIC test design?

9 In addition to the graduation condition, are there any other purposes for your taking the TOEIC test?

10 Why do you think your TOEIC score exactly/inexactly reflects your ELP?

BẢNG CÂU HỎI PHỎNG VẤN

1 Những chứng chỉ tiếng Anh nào được trường đại học của bạn chấp nhận đạt chuẩn tiếng Anh đầu ra?

2 Tại sao việc áp dụng chuẩn tiếng Anh đầu ra ở đại học lại cần thiết/ không cần thiết?

3 Việc áp chuẩn đầu ra tiếng Anh ảnh hưởng như nào đến khả năng tiếng Anh của bạn?

4 Việc phải đạt đầu ra tiếng Anh ảnh hưởng như nào đến động lực học tiếng

5 Bạn biết gì về TOEIC trước khi tham gia luyện thi tại trung tâm?

6 Trong trường hợp bạn được quyền lựa chọn thi các chứng chỉ tiếng Anh quốc tế khác làm điều kiện tốt nghiệp, tại sao bạn lại chọn thi TOEIC?

7 Tại sao bạn nghĩ mức điểm chuẩn TOEIC nhà trường yêu cầu là phù hợp/ chưa phù hợp?

8 Bạn đánh giá chung về bài kiểm tra TOEIC như thế nào?

9 Ngoài mục đích để ra trường, bạn còn sử dụng chứng chỉ TOEIC đề làm gì nữa không?

10 Tại sao bạn nghĩ điểm TOEIC đạt được đánh giá đúng (hoặc không đúng) khả năng tiếng Anh của bản thân?

APPENDIX 3: SAMPLE TEST FOR TOEIC L/R

Part 2: Question and Response (30 questions)