Đang tải... (xem toàn văn)
This study investigated the influence of topics and raters on the speaking scores of 20 sophomores at FELTE, ULIS, VNU, as well as whether or not there exist differences between the two raters in the scoring process. The speaking topics were derived from an English test preparation course that these students are currently studying, and its format was the same as part 3 of the IELTS speaking test. A survey and interviews were utilized to collect the data, which were then analysed by pairedsamples TTest and contentbased analysis respectively. The analysis of the result revealed that no significant differences were detected between the two raters’ scores of the candidates. However, the change in topics might influence the scores of the candidates. In addition, interviews with raters indicated that there exist huge differences between the two raters in their rating process, yet surprisingly, these differences did not seem to exert any significant impact on the scores.
VIETNAM NATIONAL UNIVERSITY, HANOI UNIVERSITY OF LANGUAGES AND INTERNATIONAL STUDIES FACULTY OF ENGLISH LANGUAGE TEACHER EDUCATION GRADUATION PAPER AN INVESTIGATION INTO STUDENTS’ VARIATION OF SPEAKING SCORES WITH DIFFERENT RATERS AND TOPICS A CASE STUDY: SECOND-YEAR ENGLISHMAJORED STUDENTS AT A LANGUAGE UNIVERSITY Supervisor: Dương Thu Mai Student: Ngô Phương Nga Course: QH2014.F1.E6 HÀ NỘI – 2018 ĐẠI HỌC QUỐC GIA HÀ NỘI TRƯỜNG ĐẠI HỌC NGOẠI NGỮ KHOA SƯ PHẠM TIẾNG ANH KHÓA LUẬN TỐT NGHIỆP NGHIÊN CỨU VỀ SỰ DAO ĐỘNG ĐIỂM BÀI THI NÓI CỦA HỌC SINH VỚI CÁC CHỦ ĐỀ KHÁC NHAU VÀ NGƯỜI CHẤM KHÁC NHAU: MỘT NGHIÊN CỨU VỀ SINH VIÊN NĂM THỨ HAI CHUYÊN NGÀNH TIẾNG ANH TẠI MỘT TRƯỜNG ĐẠI HỌC ĐÀO TẠO NGOẠI NGỮ Giáo viên hướng dẫn: Dương Thu Mai Sinh viên: Ngơ Phương Nga Khóa: QH2014.F1.E6 HÀ NỘI – 2018 ACCEPTANCE PAGE I hereby state that I: Ngo Phuong Nga, QH2014.F1.E6SP, being a candidate for the degree of Bachelor of Arts English Language Teacher Education accept the requirements of the College relating to the retention and use of Bachelor’s Graduation Paper deposited in the library In terms of these conditions, I agree that the origin of my paper deposited in the library should be accessible for the purposes of study and research, in accordance with the normal conditions established by the librarian for the care, loan or reproduction of the paper Signature ……………………………… May 2018 ACKNOWLEGEMENTS Firstly, I would like to express my sincere gratitude to my supervisor, Ms.Duong Thu Mai, Ph.D., for her patience, profound knowledge, as well as her whole-hearted assistance in my researching and writing time I could not imagine having a better supervisor for my thesis than her Also, I would like to grasp this opportunity to send my thanks to the two teachers at FELTE, ULIS, VNU for their enthusiastic participation in my study, and for the incentive they offered throughout the realization of this paper Besides my supervisor and teachers, I would like to send my thanks to the sophomores at FELTE, ULIS, VNU for their zeal in participating in the research Last but not least, my deepest sincere thanks goes to my friend and my family, especially my grandmother for her tremendous support and encouragement Without them, this research could not be fulfilled i ABSTRACT This study investigated the influence of topics and raters on the speaking scores of 20 sophomores at FELTE, ULIS, VNU, as well as whether or not there exist differences between the two raters in the scoring process The speaking topics were derived from an English test preparation course that these students are currently studying, and its format was the same as part of the IELTS speaking test A survey and interviews were utilized to collect the data, which were then analysed by paired-samples T-Test and content-based analysis respectively The analysis of the result revealed that no significant differences were detected between the two raters‟ scores of the candidates However, the change in topics might influence the scores of the candidates In addition, interviews with raters indicated that there exist huge differences between the two raters in their rating process, yet surprisingly, these differences did not seem to exert any significant impact on the scores ii TABLE OF CONTENTS ACKNOWLEGEMENTS i ABSTRACT ii TABLE OF CONTENTS iii LIST OF ABBREVIATIONS vi LIST OF FIGURES vii LIST OF TABLES viii PART A: INTRODUCTION 1 Rationale 2 Statement of research problem & questions 3 Scope of the research 4 Significance Organization of the study PART B: DEVELOPMENT CHAPTER I LITERATURE REVIEW Performance-based assessment 1.1 Definitions of performance-based assessment 1.2 Characteristics of performance-based assessment Assessing the speaking skills 2.1 Spoken versus written language 2.2 Model of oral assessment 10 2.3 Oral assessment process 13 Topics in oral assessment 18 3.1 Definition of topics 18 3.2 The importance of topics in oral assessment 18 iii 3.3 Related studies on topic influence in oral assessment 19 Raters in oral assessment 20 4.1 The raters factor in second language performance-based assessment 20 4.2 Definition of rater reliability in language assessment 21 4.3 Factors that affect rating operation 22 4.4 Rater effects 24 4.5 Related studies on inter-rater reliability and rater effects in assessment 27 Chapter summary 28 CHAPTER II METHODOLOGY 30 Research questions 30 Research participants and the selection of participants 31 Data collection instruments 33 Data collection procedure 34 Data analysis procedure 36 Chapter summary 40 CHAPTER III FINDINGS AND DISCUSSION 41 Research question 1: The students‟ speaking competence 41 Research question 2: The variation of the students‟ scores under the influence of different topics 44 Research question 3: The variation of the students‟ scores under the influence of different raters 46 Research Question 4: Differences between raters in the rating operation 51 Chapter summary 53 PART C: CONCLUSION 55 Summary of the findings and discussion 56 Implications 57 iv Limitation of the study 57 Suggestions for further research 58 REFERENCES 59 APPENDICES 63 APPENDIX 1: Questions for the speaking test 63 APPENDIX 2: Speaking scores of the test-takers 64 APPENDIX 3: INTERVIEW FORM 77 v LIST OF ABBREVIATIONS CEFR The Common European Framework of Reference for Languages IELTS International English Language Testing System FELTE Faculty of English Language Teacher Education ULIS University of Languages and International Studies VNU Vietnam National University, Hanoi vi LIST OF FIGURES Figure A conceptual framework for performance testing (Milanovic & Saville, 1996) 10 Figure An expanded model of speaking test performance (Fulcher, 2003) 12 Figure A framework for describing the construct definition for a test of second language speaking (Fulcher, 2003) 16 Figure Traditional fixed response assessment and assessment involving judgement (McNamara, 1996) 21 Figure IELTS and the CEFR ("Common European Framework", n.d.) 38 Figure Descriptive statistics for the scores of topic by rater 41 Figure Descriptive statistics for the scores of topic by rater 41 Figure Descriptive statistics for the scores of topic by rater 42 vii 6.75 , verb tense) nce) FC LR GA P 6-7 6-7 6-7 6-7 (clear sound but 6.75 mispronoun ce sometimes 6.75 FC LR GA P 6-7 6-7 6-7 6-7 (mispronou nce) 5.75 6.25 6.25 FC LR GA P 5-6 5-6 FC LR GA P 6 65 6.0 FC LR GA P 6 (misused 6.25 collocation, selfcorrection) 10 11 12 5.75 5.5 7.5 6.0 5.75 7.75 FC LR GA P 5-6 5-6 (hesitance) (misused (frequent (mispronou collocation) errors) nce) FC LR GA P 7-8 7-8 7-8 (fluent, (varied) (clear coherent) 13 6.0 6.0 sounds) FC LR GA P 6 6 66 (hesitance) (WF) (mispronou nce) 14 6.5 6.75 FC LR GA P 6-7 6-7 6-7 6-7 FC LR GA P 6-7 7 FC LR GA P 7 6-7 FC LR GA P 7-8 7 FC LR GA P 6-7 6-7 6-7 FC LR GA P 7 FC LR GA P 7 Repetition 15 6.75 7.0 16 6.75 7.0 17 7.0 7.25 18 6.75 6.75 19 6.5 6.5 20 6.5 6.5 67 MARKING SHEET – RATER No Topic 1: Topic 2: Education Health NOTE Quite fluent with ideas being generally clear (though not well developed due to over emphasis on vocab) 7.0 7.0 Use complex vocabulary but tend to be over-formal/bookish, some word choices are awkward Generally control of grammar despite occasional mistakes Clear pronunciation with varied features Quite good fluency (pauses are content related), ideas come through well and naturally 7.0 7.0 Use both basic and complex structures, Grammar errors (Lack be, verb tense, S-V) Very flexible vocabulary 5.5 5.5 Fluency is inconsistent, ideas come through not quite well at times 68 Vocabulary is adequate for the topic, frequent word form and collocation mistakes Some grammar errors (Verb tenses) Clear vocabulary though inaccurate at some words Pronunciation is clear but inaccurate at some words (daily) Noticeable pauses, some responses not answer the 5.0 5.0 questions, Vocabulary and grammar are barely adequate for the topics Limited pronunciation features, Problems with ending sound, Fluency is inconsistent but ideas come though clearly Vocabulary has both common and less common words, Grammar control is inconsistent (good with verb tense but 6.0 6.0 make S-V disagreement) Mixed control of pronunciation correct her own mistakes though there are some other errors which did not affect understanding (figure out) 69 Noticeable pauses but ideas still come through Adequate vocabulary 5.5 5.5 Need better control of grammar, mainly use basic structures Mixed control of pronuncation, problems with long vowels, ending sound Noticeable pauses but ideas come through 5.5 5.5 Try to use complex vocabulary but with less accuracy Generally good control of grammar Clear pronunciation, problems with Sh sound Quite fluent and clear, rare pauses, ideas come through well 6.5 6.5 Vocabulary is adequate, though not varied Good control of grammar Clear pronunciation but often Lack ending sound, Good fluency, ideas come through well, 6.0 6.0 Adequate vocabulary though not varied (has problems with collocation, word form) Need better control of grammar 70 Mixed control of pronunciation, problem with Sh sound, Not quite fluent but ideas come through well 10 6.5 6.0 Vocabulary is sufficient Grammar is controlled well Clear vocabulary 71 No Topic 1: Topic 2: Education Health NOTE Ideas not come through well, very low fluency with 11 4 noticeable pauses Very poor control of grammar and vocabulary Limited range of pronunciation features Progressively fluent despite some pauses and repetition, ideas are coherent Use both common and some less common vocabulary, tend to 12 6 be less accurate with less common vocabulary Need better control of grammar (verb tense, verb forms) Good pronunciation, easy to understand but need more consistent control Ideas come through at times, but fluency is low with 13 4.5 4.5 noticeable pauses Barely adequate vocabulary Poor control of grammar 73 Pronunciation features are limited Quite fluent with few pauses (mainly related to idea not vocab), answers are well coherent Manage to paraphrase, use less common vocabulary wit some 14 6.5 7.0 awareness Generally good control of grammar but with some inaccuracies (S-V disagreement), use both basic and complex structures Clear and effective pronunciation Speak clearly and fluently Ideas are well coherent Generally good control of grammar 15 7 Manage to use less common vocabulary with awareness despite occasional inaccuracies Clear and effective pronunciation 16 7.0 7.0 Speak clearly and ideas come through well Vocabulary: sufficient for topics, attempt some less common 74 words with mixed success Good use of verb tense, make some minor slips Pronunciation is generally clear Very fluent and ideas are clear Vocabulary is flexible but make occasional mistakes (word 17 7.5 7.5 form) Varied grammatical structures and generally good control of grammar despite occasional mistakes (S-V agreement) Clear and effective pronunciation Not quite fluent, ideas come through well 18 7.0 7.0 Flexible vocabulary for the topics Need better use of verb tense, grammar (S-V agreement) Clear pronunciation Quite fluent and ideas come through well 19 8.0 8.0 Flexible vocabulary despite occasional errors (collocation, word form) Use both common and less common structures; Need more 75 consistent control of grammar Quite fluent and ideas come through well 20 6.5 6.5 Vocabulary includes both common and less common words but tend to be less accurate in using complex ones Need better control of verb tense and grammar in general 76 APPENDIX 3: INTERVIEW FORM Interview Questions These are some questions about your rating procedure Please give your answer sincerely because that is the only way to ensure the significance and success of this research Rater 1: please skip question 4, 5, 6, Rater 2: please skip question Thank you very much for your cooperation I would also hope you two have a happy, healthy and successful new year! Love, Phuong Nga Your name is …? ………………………………………………………………………………… Question 1: When you grade a performance, you use holistic scoring (give students a single and overall assessment score for the paper as a whole) or analytic scoring (grade students on separate criteria)? * o Holistic scoring o Analytic scoring Question 2: How you grade a performance? Can you specify all steps? * ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… Question 3: (For rater 1) Do you feel tired when you are required to grade many performances at the same time? If yes, you think that the scores of some last performances might suffer from your tiredness or boredom? ………………………………………………………………………………… 77 ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… Question 4: (For rater 2) How many sessions did you spend grading all performances? (your answer can be day, mornings, or days, etc) ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… Question 5: (For rater 2) How many performances did you grade per each session? ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… Question 6: (For rater 2) Do you feel tired when you are required to grade many performances in total? If yes, you think that the scores of some last performances might suffer from your tiredness or boredom? ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… Question 7: There are four criteria in the marking rubric: Fluency and coherence, lexical resource, grammatical range and accuracy, and pronunciation Can you rank these four criteria in order of importance? with is the most important criterion and is the least important one * Fluency and coherence Lexical resource 78 Grammatical range and accuracy Pronunciation Question 8: There are many key indicators in fluency and coherence criterion Can you rank these indicators in order of importance? with is the most important criterion and is the least important one * Speech rate Speech continuity Logical sequencing of sentences Clear marking of stages in talking The use of cohesive devices (e.g connectors, pronouns and conjunctions) within and between sentences 10 Question 9: There are many key indicators in lexical resource criterion Can you rank these indicators in order of importance? with is the most important criterion and is the least important one * The variety of words used The adequacy and appropriacy of the words used The ability to circumlocute (get round a vocabulary gap by using other words) with or without noticeable hesitation 11 Question 10: There are many key indicators in Grammatical Range and Accuracy criterion Can you rank these indicators in order of importance? with is the most important criterion and is the least important one * 79 The length and complexity of the spoken sentences The appropriate use of subordinate clauses The range of sentence structures The number of grammatical errors in a given amount of speech The communicative effect of error 12 Question 11: There are many key indicators in pronunciation criterion Can you rank these indicators in order of importance? with is the most important criterion and is the least important one * The amount of strain caused to the listener The amount of unintelligible speech The noticeability of L1 (i.e first language) influence 13 Question 12: Are you a lenient or severe rater in your opinion? Why? * ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… ………………………………………………………………………………… 80 ...ĐẠI HỌC QUỐC GIA HÀ NỘI TRƯỜNG ĐẠI HỌC NGOẠI NGỮ KHOA SƯ PHẠM TIẾNG ANH KHÓA LUẬN TỐT NGHIỆP NGHIÊN CỨU VỀ SỰ DAO ĐỘNG ĐIỂM BÀI THI NÓI CỦA HỌC SINH VỚI CÁC CHỦ ĐỀ KHÁC NHAU VÀ NGƯỜI CHẤM KHÁC NHAU: ... VÀ NGƯỜI CHẤM KHÁC NHAU: MỘT NGHIÊN CỨU VỀ SINH VIÊN NĂM THỨ HAI CHUYÊN NGÀNH TIẾNG ANH TẠI MỘT TRƯỜNG ĐẠI HỌC ĐÀO TẠO NGOẠI NGỮ Giáo viên hướng dẫn: Dương Thu Mai Sinh viên: Ngơ Phương Nga Khóa:... speaking was found in this section, followed by the description of the IELTS rating scale which was later used for the data collection process in this research The third part of this chapter revolved