A pilot study on pet and aptis tests for english non – major students at hoa sen university

Journal of Science and Technology, Vol.37, 2019 A PILOT STUDY ON PET AND APTIS TESTS FORENGLISH NONMAJORSTUDENTS AT HOA SEN UNIVERSITY QUACH THI TO NU, VUONG NGỌC TIEN, HONG NGUYEN THANH TAM Hoa Sen University; tonuquachthi@gmail.com, tien.vuongngoc@hoasen.edu.vn, tam.hongnguyenthanh@hoasen.edu.vn Abstract This study aims to investigate the appropriateness of PET and Aptis for students whose major is not English (called non-majors) at Hoa Sen University (abbreviated to HSU), to decide on the more suitability of either tests and to give some recommendations for teaching English to the non-majors Both quantitative and qualitative research methods were employed in this study, which was mainly based on the Test Results of 39 non-majors Questionnaires were used to collect extra-information on those tests The research findings suggest that PET and Aptis should be suitable to HSU non-majors, that Aptis could be more suitable than PET, and that the currently applied curriculum should be changed a bit to make the non-majors familiar with composing Email responses and Interactive writing in PET and Aptis Key words Aptis, non-majors, PET INTRODUCTION The English program at English Zone (abbreviated to EZ), HSU, has been designed to equip the nonmajors with the four basic language skills in English, namely reading, writing, listening and speaking, to succeed academically and to be able to communicate fluently in the demanding working environment In order to graduate, the non-majors can submit one of those three certificates: TOEIC 550, TOEFL iBT 64, or IELTS 5.0 These requirements are relevant to the Vietnamese Six-Level Language Competence Framework (VLLCF) 2020 Accordingly, Vietnamese graduates have to complete 3/6 in VLLCF (equivalent to B1 of CEFR) Students can choose a certificate of their own interest and orientation HSU has been cooperating with IIG to provide a free TOEIC exam to students who joined the whole English program at EZ It can be inferred that TOEIC is not the only exit certificate for the non-majors However, most of them choose it because EZ students can take a free exam once and non-EZ students not have to pay a lot of money for this two-skilled test After years of organizing TOEIC exams to the non-majors, we have found out some drawbacks: Firstly, TOEIC no longer matches the English program objectives as the currently-used TOEIC test does not test all the four basic language skills in English Secondly, TOEIC is irrelevant to CEFR and VLLCF used for building EZ curriculum, choosing course books, teaching and assessing TOEIC, therefore, cannot be integrated into the curriculum’s testing and assessment Thirdly, most of the non-majors not take the TOEIC exam right after finishing the English Program but wait until their graduation time, which results in knowledge loss and skill weakening Fourthly, EZ has to provide a TOEIC preparation course as the TOEIC exam is far different from EZ’s English program Last but not least, only 58% of the nonmajors get the required scores This is a waste of HSU investment and students’ time Hence, we need to find a substitute test (1) which is relevant to the training aims, HSU’s English program, (2) which tests all four skills, (3) which is relevant to CEFR and the VLCF, (4) which can be integrated into the teaching program to assess students during and after the program, (5) which can increase the percentage of successful English learners and guarantee the teaching quality, and (6) which is affordable Those new conditions lead us to consider PET and Aptis to assess the non-majors competence after completing the EZ English program These new certificates are provided by prestigious organizations (Cambridge Examinations or British Council) whose assessment systems can be compared to CEFR and MoET In comparison to TOEIC, their prestige, confidentiality, validity and reliability of language assessment are equivalent Upon providing new test, HSU keeps recognizing TOEIC 550, TOEFL iBT 64 and IELTS 5.0 as English graduation benchmark There is a need for us to find out whether these two tests actually meet our hypothesis © 2019 Industrial University of Ho Chi Minh City 160 A PILOT STUDY ON PET AND APTIS TESTS FOR ENGLISH NON-MAJOR STUDENTS AT HOA SEN UNIVERSITY LITERATURE REVIEW 2.1 What is PET A Cambridge English: Preliminary (PET) qualification shows that you have mastered the basics of English and now have practical language skills for everyday use Results and score scale Cambridge English: Preliminary (PET) results are reported on the Cambridge English Scale[1].You will receive a separate score for each of the four basic language skills and use of English, giving you a clear understanding of your performance These five scores are averaged to give you an overall result for the exam You will also be given a grade and Common European Framework of Reference for Languages (CEFR) level Cambridge English: Preliminary is targeted at CEFR Level B1, but also provides reliable assessment at the level above B1 (Level B2) and the level below (Level A2).The candidate’s overall score is averaged from the individual scores for each of the four skills[2] 2.2 What is Aptis Aptis is a business-to-business product whose clients are organizations and institutions using Aptis to benchmark the English language levels of their employees, potential employees, students or teachers (British Council, English Language Assessment Research, 2014, p.7)[3] The history and aim of Aptis development Based on the main or general version of Aptis launched in August 2012 and Aptis development project conducted by an English Language Assessment Research Expert Group at the British Council, Aptis is designed and developed in order to reflect best practice in the area of language testing and fit with the British Council’s ambitions in the area of assessment literacy The British Council’s ambitions are to aim at offering world-class advice and consultancy to many governments, institutions, education ministries, recruitment agencies, and corporation across the world The language teaching and assessment of the Aptis test is mainly linked to the referenced document − the Common European Framework of Reference for Languages (abbreviated to CEFR) [4] Consequently, the testing English levels from A1-C on CEFR, Aptis is an English test for adults (16 years old or older), which can be used to assess ability in all the four language skills.(Tucker, 2014: 11) [5] Scoring The report score of each candidate will be reported either on a numerical scale (0-50) or reported as a CEFR level The numerical scale score includes the overall scores of the four language skills except grammar and vocabulary while the CEFR skill profile will be in form of bar chart along with the CEFR skill descriptors The descriptors of a candidate’s ability which comprise the CEFR indicate the progression to mastery and are presented on a rising six-level scale (A1, A2, B1, B2, C1, and C2) [6] Aptis Cut Score Table [7] should be referenced for clarification SCOPE AND PROCEDURE OF RESEARCH 3.1 Research questions The study done for and reported in this paper aims to explore the suitability of HSU non-majors with either PET or Aptis tests So the study addresses the following research questions: What are HSU non-majors’ perceptions to Aptis? What are HSU non-majors’ perceptions to PET? What are HSU non-majors’ performances on the two tests? 3.2 Participants Polit and Hungler (1999:37)[8] refer to the population as an aggregate or totality of all the objects, subjects or members that conform to a set of specifications In this study, the population was 40 upperintermediate HSU non-majors © 2019 Industrial University of Ho Chi Minh City A PILOT STUDY ON PET AND APTIS TESTS FOR ENGLISH NON-MAJOR STUDENTS AT HOA SEN UNIVERSITY 161 Elmes et al (2012:56)[9] stated that random selection is related to sampling Therefore, it is most related the external validity of the results In this research, random selection has been employed through these processes First, more than 200 EGC4 students aged from 19 to 21 and coming from different majors are informed to register for the new tests as an extra benefits different from TOEIC Second, only 80 students getting 5.6 to 10 score (in which their average score for each skill must be at least 5) in EGC4 are qualified They are put into apreparation class equipping them with knowledge and strategies for both tests Nearly half of them have been learning English for six to ten years, and up to 37% of the students have a prolonging experience with English when they have been exposed to English since the age of six Nearly 50% of the students started at EZ at the Intermediate levels (EGC1 or EGC2) It can be inferred that their Entry levels were relatively high Third, after the mock test (a mixed test of PET and APTIS question types) at the end of the preparation time, the shortlisted 40 students are divided into two equivalent groups of 20 for PET and 20 for Aptis Each group comprises average, fair and good scores The students not know which real test they will take until days before the exam 3.3 Instruments EGC scores The final exam scores of EGC4 students in semester 15.A2 at HSU were used to select participants whose scores range from 5.5 to 8.5 Mock Test The mock test, which consists of four parts, is a mix of the two tests: PET and Aptis [10, 11, 12, 13] The two-folded purpose is to familiarize the students with both of the tests’ question types and to divide them into two equivalent groups for the real tests later Part is a 30-question Listening pack divided into five sessions focusing on ticking the correct picture, MCQs, gap fill, YES/NO, short conversations Part is a 18-question Grammar and Vocabulary focusing on matching definition and MCQs Part is a 28-question Reading pack divided into four parts focusing on putting the story events in the correct order, matching the heading, matching the people with the correct interest, gap fill Part 4is a three-part Writing pack focusing on sentence completion, stating feelings on a club, formal and informal emails Pre-Test Questionnaire (See Appendix B.) The students are first requested to fill in some background information including their full name, gender, age, major, years of learning English, and English levels taken at EZ Then they are asked to give their opinions on the five following parts Part is on test-taking skills: the students evaluate their own knowledge and familiarity to PET and APTIS question types Part is on test psychology: the students express their feelings, aims, seriousness, motivation and confidence on the test Part is on the Mock test evaluation: the students evaluate the clarity of the test instruction, their time management skill, their familiarity with test question types, and the speed and sound quality of the test audio Part is on the relevance of EGC program to the Mock Test: the students identify the overlapping question types between EGC program and PET/APTIS test Part is on the review session: the students evaluate the usefulness of the review days on their test-taking skills and knowledge Post-Test Questionnaire (See Appendix C) The students are requested to fill in some background information including their full name, gender, age, major, years of learning English, and English levels taken at EZ Then they are asked to give their opinions on the six following parts Part is on the overview feelings on the exam: the students evaluate their feelings on doing the test on computers, exam instructions, their familiarity with question types, the time allotted, their results on paper versions, the review time, and the review session content Part is on the Reading Test: the students evaluate their Reading skill, Reading question types, and reading obstacles Part is on the Listening Test: the students evaluate their listening skill, listening question types, and listening obstacles Part is on the Writing Test: the students evaluate their Writing skill, Writing question types, and writing obstacles Part is on the Speaking Test: the students evaluate their Speaking © 2019 Industrial University of Ho Chi Minh City 162 A PILOT STUDY ON PET AND APTIS TESTS FOR ENGLISH NON-MAJOR STUDENTS AT HOA SEN UNIVERSITY skill, Speaking question types, and Speaking obstacles Part is extra information on their attendance frequency in the review session, their preference on those tests, their self-study review time, and their further suggestions for the teachers and university PET/Aptis results Since the two real tests are confidential, we cannot get them from the test providers So in this part, we just describe the general format of the two tests PET test [14] is made up of three papers: Reading and Writing (1 hour 30 minutes), Listening (36 minutes, including minutes' transfer time), Speaking (10-12 minutes per pair of candidates) Aptis test [15]is described as follows: Core (25 minutes), Reading (30 minutes), Listening (25-50 minutes), Writing (50 minutes), Speaking (12 minutes) 3.4 Data analysis In this study, SPSS was used to analyze the collected data Independent sample T-test analysis was used to find whether there were any differences among EGC4 average scores of each skill, between PET and Aptis mock test and real test results RESULT OF RESEARCH Mock test results comparison Figure Mock test results of PET and Aptis groups Figure 1shows that individual scores in two groups PET and Aptis ranged from 40 to 90 Aptis group’s average score is 65.16 while that number for PET is 65.93 It can be concluded that two groups have equivalent band scores, peaking at 70 Independent samples T-test Table 1.Mock test results comparison between PET/Aptis Group Statistics Listening Reading Writing T-test for Equality of Means Aptis N 19 Mean 72,807 Std Deviation 12,8266 Std Error Mean 2,9426 PET 20 72,333 13,8116 3,0884 Aptis 19 70,677 17,6407 4,0471 PET 20 69,821 13,2862 2,9709 Aptis 19 67,434 11,7073 2,6858 © 2019 Industrial University of Ho Chi Minh City T Df Sig (2tailed) Mean Difference Std Error Difference ,111 37 ,912 ,474 4,274 ,172 37 ,865 ,855 4,984 ,434 36 ,667 1,645 3,792 A PILOT STUDY ON PET AND APTIS TESTS FOR ENGLISH NON-MAJOR STUDENTS AT HOA SEN UNIVERSITY Grammar Mock Test PET 19 65,789 11,6682 2,6769 Aptis 19 49,708 20,7435 4,7589 PET 20 52,222 21,2048 4,7415 Aptis 19 65,156 11,8153 2,7106 PET 19 65,932 10,5451 2,4192 163 -,374 37 ,710 -2,515 6,722 -,214 36 ,832 -,776 3,633 An independent sample t-test was run to see if there is a significant difference between two groups in individual scores It can be seen from table that sig level of the Mock Test and all individual skills are bigger than 0.05 (p>0.05), resulting in no differences in individual and overall scores between two groups In other words, two groups performed similarly to each other in both overall and individual scores EGC4 Scores Comparison EGC4 AVERAGE SCORES EGC4 AVERAGE SCORES Figure Average scores of EGC4 between two groups PET and Aptis As can be seen from Figure 2, the students in two groups have average scores ranging from 5.5 to 8.5 Aptis group’s average score is 67.32, and PET’s is 67.55 The figure also shows that two groups have similar average scores peaking from 6.5 to 7.5 - Independent samples T-test Table 2.The comparison of EGC4 average scores between PET and Aptis Group Statistics Listening Reading Writing Speaking Vocabulary Grammar EGC4 AVERAGE SCORES Aptis PET Aptis PET Aptis PET Aptis PET Aptis PET Aptis PET Aptis PET N 19 20 19 20 19 20 19 19 19 20 19 20 19 20 T-test for Equality of Means Mean 71,579 72,000 65,000 65,375 62,368 74,750 73,070 72,456 58,947 59,500 60,526 58,5 67,315 Std Deviation 14,9120 15,5935 17,3606 13,4084 16,7803 10,8185 11,0730 11,4040 20,2470 15,3811 17,4717 20,5899 6,91257 Std Error Mean 3,4211 3,4868 3,9828 2,9982 3,8497 2,4191 2,5403 2,6163 4,6449 3,4393 4,0083 4,6040 1,5858 67,55 8,13358 1,8187 T df Sig (2tailed) Mean Difference Std Error Difference -,086 37 ,932 -,421 4,891 -,076 37 ,940 -,375 4,952 -2,723 31 ,011 -12,382 4,547 ,168 36 ,867 ,614 3,647 -,096 37 ,924 -,553 5,739 ,331 37 ,743 2,026 6,131 -,097 37 ,924 -,234 2,423 © 2019 Industrial University of Ho Chi Minh City 164 A PILOT STUDY ON PET AND APTIS TESTS FOR ENGLISH NON-MAJOR STUDENTS AT HOA SEN UNIVERSITY Table shows the comparison of EGC4 average scores and detailed scores between Aptis and PET groups through T-tests The sig level values of average scores and detailed scores comprising of Listening, Reading, Speaking, Vocabulary, Grammar are all greater than 0,05 (p>0.05), it can be concluded that there is no significant difference between two groups In other words, two groups have equivalent average scores and detailed scores As for Writing scores, with t-test results are t=2,723, df=31, p=0,011 As p