Measuring Washback Effect on Learning English Using Student Response System

http://dx.doi.org/10.16945/201719109 ≷ ≸ Measuring Washback Effect on Learning English Using Student Response System* 윤태진 (성신여자대학교) Introduction The paper considers an effect of washback that the TOEIC test has on English learning by undergraduate students The effect was studied by implementing a web-based module of solving a subset of TOEIC questions and by evaluating students' performance at multiple time points during a semester The TOEIC was developed in the 1970s by Chauncey Group International, a subsidiary of Educational Testing Service (ETS), in response to a request by the Japanese government for an English language proficiency test developed specifically for the workplace Through collaboration with a Japanese team, the Chauncey Group designed a listening and reading comprehension test to be used by corporate clients, and in 1979, the TOEIC was first administered in Japan to 2,710 test-takers In 2006, ETS (Educational Testing Service), the largest private testing organization based in Princeton, revised the format of the TOEIC and introduced speaking and writing modules With all four components, the test makers describe the TOEIC program as a measure of the 'everyday English * 이 논문은 2017년도 성신여자대학교 학술연구조성비 지원에 의하여 연구되었음 skills of people working in an international environment' when 'scores indicate how well people can communicate in English with others in business, commerce and industry' (ETS, 2008) As the TOEIC program has come to meet the needs of various stakeholders for a wide range of purposes, the popularity and influence of the test has grown considerably over the last 30 years In 2010, ETS administered more than six million TOEIC tests worldwide, establishing the program "as the largest and most widely used English-language assessment for the work place" (Betaneli, 2011) In Korea, 1.9 million examinees took the test in 2007, outnumbering Japanese test-takers for the first time, and in 2008, Choi (2008) described the TOEIC as the most widely used English test in Korea, making up 47.1% of the entire EFL testing market for adult test-takers In Korea, numerous hagwons (private institution) offer TOEIC test-preparation courses for business people, students and graduates seeking to increase their employability In addition, TOEIC test-preparation or incorporation of TOEIC test score is often a part of a university's curriculum Numerous colleges and universities also include the TOEIC or TOEIC preparation classes for reasons such as advertising that the education provided takes globalization into account, a tool for furthering comprehension of English, or a method to improve student employability Despite the scope and popularity of the TOEIC, research into the test has received relatively little attention from researchers Obtaining high scores on EFL tests greatly influences graduation requirements, scholarship allocation, fee reductions, entry into courses and job placements The TOEIC, in particular, has become a powerful device used by universities and companies to fulfill a range of gate keeping purposes, and consequently, it attracts millions of test-takers every year It is likely that focusing on test taking strategies is a familiar method of study for many students, and a strategy that continues from high school into the university setting Cho (2010) confirms this in her study of Korean high school students preparing for the KCSAT (Korean College Scholastic Aptitude Test) Adapting a survey based on Green’s (2007) modification of the SILL (Strategy Inventory for Language Learning), Cho (2010) found that the item to which respondents gave the highest score was, ‘I tried to learn test-taking skills for the KCSAT’ Thus, it is likely that preparing TOEIC tests may have an influence on learning as well as teaching, as is the case with the KCSAT test Washback refers to the influence that tests exert on teaching and learning (Alderson & Wall, 1993; Bailey, 1996; Cheng & Curtis, 2004; Booth, 2012; Loumbourdi, 2013) Pearson (1988: 98) states that "[p]ublic examinations influence the attitudes, behaviors, and motivation of teachers, learners and parents, and because examinations often come at the end of a course, this influence is seen working in a backward direction." The term 'washback' can be understood in the context of the backward direction of the effects that the test at the end of a course has on the course of learning In order for tests to act as agents of positive reform, a 'match' or an overlap between the content and the format of the examination, and the content and format of the curriculum is encouraged (Cheng & Curtis, 2004: 4) Madaus (1988) pointed out that 'it is testing, not the "official" stated curriculum, that is increasingly determining what is taught, how it is taught, what is learned, and how it is learned.' (p 83) Ebel (1979: 23) claims that it is common practice for students to work harder when they know that they are approaching exams than when they not However, relatively little research has documented washback from the perspective of learners (Bailey, 1999; Tsagari, 2007) Hayes and Read (2004) reports on research in New Zealand on the washback effects of preparation courses for the IELTS The study involves intensive classroom observation of two IELTS courses over a 4-week period The results show clear differences between the two courses One was strongly focused on familiarizing students with the test and practicing test tasks, while the other covered a wider range of academic study tasks The research highlights both the potential and the limitations of this kind of study in the investigation of washback However, the class observation is limited in directly quantifying what students actually learned regarding the test The ability to know what they have understood is crucial to good teaching or training The benefits of active learning are widely acclaimed in higher education (Martyn, 2007) Chickering and Gamson, early proponents of active learning, designated 'encourage active learning' as one of seven principles of good practice in higher education (Chickering and Gamson, 1987) A relatively new technology such as the clicker system offers one approach to employ active learning in the classroom They are more formally designated as student response systems (SRS), audience response systems (ARS), or personal response systems (PRS) Clickers help instructors (i) actively engage students during the entire class period, (ii) gauge their level of understanding of the material being presented, and (iii) provide prompt feedback to student questions With clickers, students have an input device that lets them express their views in complete anonymity, and the cumulative view of the class appears on a screen In this regard, the technological advances may be a useful tool for instructors for TOEIC or TOEFL to give feedback to their students, unlike the widely spread perception like the following excerpt: “And it was during that same research that I began to realize that the crucial issue is not to ask whether washback exists, but to understand why it has what effects it does have I will never forget one of the teachers I observed replying to the question: "Is it possible to teach TOEFL communicatively?" by saying: "I never thought of that." Which I interpreted as meaning that he had not given much thought as to what might be the most appropriate way to teach toward such an important test And when I interviewed a group of teachers about what they thought about teaching toward TOEFL, I was surprised to learn that two things they liked most about teaching TOEFL (there were, of course, many things they did not like) was that they did not have to plan lessons, and they did not have to mark homework Two of the most important things teachers is prepare their lessons and give students feedback, and yet when teaching toward TOEFL some teachers at least not feel that this is necessary.” (Alderson, 2004; p.x) It seems that iClicker is the most widely used clickers system It can be purchased at around $40 or rented at about $15 per year in North America Other clicker systems are also available for a fee or for free Top Hat costs about $20 a year for subscription Socrative and Ping Pong have been freely distributed in Korea These student response systems allow instructors to engage their students with educational activities on tablets, laptops and smartphones However, Drawbacks to the use of the clicker system include: the inlcass use of a clicker system refrains instructors from asking more than 20 questions during class Moreover, the instructors are not able to focus on questions which many students are having difficulty in solving A solution to overcome the limited number of inclass questions is combining the inclass use of the clicker system with the use of Google Docs Forms out of the classes Google Docs Forms is good for synchronous and asynchronous audience response and polling The learning curve is low, which made it an alternative to physical Clickers Instructors can send the students a link to the form they created in Google Forms Students can then use their laptops or smartphones to access the form (with no need to create an account or log into a site) As the students submit their answers, the instructor can view their responses in a chart visualization, created by Google Docs Also, the responses are tracked in a spreadsheet inside Google Docs Given the widespread use of the TOEIC in South Korea and relative lack of research on washback from the learner's perspective, this paper will examine what effects the preparation of TOEIC has on student's learning of the English language The study investigated whether active learning resulted in higher learning outcomes Learning outcomes were measured by taking the score on the comprehensive final exam at the end of the semester Experiment 2.1 Participants The study was conducted in the spring semester of 2014 The study took place at a university located in the Chungchung Province in the center of Korea Participants in this study included 41 undergraduates, who are mostly in the second year and who enrolled in the practical English grammar class The participants met twice a week for the class; one for 100 minutes (with a 10 minute break) and the other for 50 minutes for the course lecture In addition to the lecture, weekly assignments of about 200 questions from a TOEIC preparatory book were required to be completed online before the first class of the week In order to determine if any statistically significant differences existed in terms of the TOEIC score, the students were required to take two mock TOEIC exams, one at the beginning and the other at the end of the semester Moreover, in order to determine how well the students retained or reviewed the online questions, they were given midterm exams and final exams which composed of a subset of the same questions as those they answered online using the Google Forms 2.2 Data Collection The current TOEIC is made up of three tests: the L/R test, a speaking and a writing test Both the speaking and writing tests are optional ETS promotes TOEIC as a measure of ‘a person’s ability to communicate in English in the context of daily life and the global workplace environment using key expressions and common, everyday vocabulary’ (Powers, 2010, p 1.2) The scores are norm-referenced, and reported on separate scales of to 495 with a maximum combined score of 990 (Woodford 1982: 3) Unlike criterion referenced tests, which rate the specific tasks a test-taker is capable of doing regardless of the performance of others, norm referenced tests, such as TOEIC, provide a numerical score showing students relations to one another (Thomson, 2012: 3) For the current study, instruction of the courses has been made with a focus on improving scores mainly in Part V (Incomplete sentences) and Part VI (Text Completion) As mentioned above, the benefits of active learning are widely acclaimed in higher education (Martyn, 2007) In order to operationalize active learning for the class, an online tool and a widely-used TOEIC preparatory book are used As a way of motivating students, the instructor told the students who participated in the study that ETS did not make questions for TOEIC but chose them from a test bank, so solving similar questions from past TOEIC tests would help them improve their TOEIC score The instructor added that even though ETS did not provide the questions from the past TOEIC tests, the TOEIC study books adopted in the class had been famous for having such similar questions The instructor divided into 13 parts the grammar and vocabulary parts in the adopted TOEIC Reading book, and all the questions in the 13 parts were requested to be solved throughout the semester The instructor tried to motivate the students by maintaining that if they would solve and understand all the questions that they learned during the semester, their TOEIC score would definitely increase due to the similarity of the question types in the book and in TOEIC, and also by promising that the midterm and final exams for the course would be drawn from the questions that they already solved as weekly assignments The online tool is Google Forms Using Google Forms enabled the researcher to give weekly assignments to students and to collect data from the assignments for the analysis Figure is an example of online questions that students are required to solve before the first class of the week that are made available online using Google Forms During the lecture, the instructor chose only those questions that less than half of the students could answer correctly Those objectively difficult questions could be identified by inspecting the 'Summary Reponses' of the weekly assignments that the students handed in using the Google Forms An example of the visualization of Summary Reponses is presented in Figure With the help of the Summary Responses, instructors can choose what to put an emphasis on and what not to during the class instruction For example, the first question whose correct response is 61% may make students less interested in the question Because there are more students who know how to solve these kinds of questions than those who not, it would be a good type of question that students can collaborate with each other The second question, which only 41% of the students got correct, may be challenging and demand for the explanation of the instructor With the aid of Google Forms, the class could only focus on questions at least half of the class had problems without wasting time explaining even such questions which most students already knew how to solve In explaining questions, the instructor helped students identify discrete points and key patterns Also, the instructor alloted 50 minutes for team-based student presentation That is, the enrolled students made a team of 3, and each three-membered team was required to make a demonstration of explaining difficult questions to other classmates A team of students knew what questions to prepare for the class presentation at least one week ahead based on the collected results available in Google Forms 2.3 Analysis Tsagari (2007) emphasizes the difficulty of providing evidence of whether students have learned better or more (as a result of a particular test) through stakeholder perceptions alone To overcome, quantitative analyses of data are made Evaluation of the study results focused on student learning outcomes The outcomes are the four test scores that the students have taken during the semester: (1) Mid-term, (2) Final exam, (3) the first mock TOEIC, and (4) the second mock TOEIC The outcomes were obtained early March (first TOEIC), mid-April (Mid-term), early May (second TOEIC) and early June (Final Exam) In order to find out whether the final exam scores were indeed higher than those of the midterm on average, a dependent t-test was made On average, participants earned significantly higher scores from the final exam (M=21.10, SE=1.22) than from the midterm exam (M=18.76, SE=1.04), t(38)=3.815, p