Serine_Poghosyan_Thesis_An Examination of Content Validity of Englsih Entrance Exam (1)

Department of English Programs A Thesis Submitted in Partial Fulfillment of the Requirements for the Degree of Master of Arts in Teaching English as a Foreign Language (TEFL) Title An Examination of the Content Validity of the Unified School Leaving and University Entrance Examination in English of 2009 Thesis Advisors: Catherine Buon, PhD Hossein Farhady, PhD Thesis Reader: Lilianna Edilian By: Serine Poghosyan Yerevan, Armenia 2010 ii To my parents for their unsurpassed love and care iii ACKNOWLEDGEMENTS There is nothing in the world we can accomplish all by ourselves, and we are all interdependent on one another for anything that we As I am approaching the end of this particular bittersweet journey, I embrace the humility of knowing that I am dependent on everyone in my world as I know it I would like to acknowledge the contributions of the particular individuals who have helped to direct me into the world of cognition and have guided me from the conception of this thesis to the completion of its final pages Particular gratitude is extended to Dr Hossein Farhady, who started this journey to my thesis as my mentor giving me the idea of the thesis and providing me with invaluable guidance throughout the whole research process even after his resignation and departure to the USA This thesis would not have been possible but for his persistent and unyielding requirements as an instructor and my first supervisor It is an honor for me to express my profound gratitude to Dr Catherine Buon, the associate dean of the Department of English Programs and my present supervisor Dr Buon took up the official supervision of my research work immediately after she discovered the situation I was in and created an atmosphere of effective collaboration and cooperation with a reasonable portion of pressure, push and ample encouragement I offer my deepest gratitude, love and appreciation to Lilianna Edilyan, my second reader, for her humane treatment, unwavering belief in my potential and the ongoing guidance, support and patience during the whole period of my studies and as a reader of my thesis reading and proofreading my work persistently giving me relevant detailed feedback and directions iv I would like to thank the teachers I had the honor to work with as a first year student at the AUA I express my special gratitude to Dr Sivakumar Sivasubramaniam who taught me the first class I ever took at the TEFL program and within a year changed my values as a teacher instilling in me a more humanistic approach to language teaching I am thankful to him for the first steps I made towards critical thinking and constructivist paradigm of language teaching I would like to express my sincere gratitude to Dr Robert Agajeenian who helped me to facilitate my studies at AUA I deeply appreciate and am thankful to him for his support and guidance I am deeply grateful to Rubina Gasparyan for her helpful and interesting classes, for her patience and readiness to guide us in class and after hours In fact, she was the first to lead us to the endless wonders of assessment and testing and it is due to her approval that I took up this particular topic It is a pleasure to thank Shaku and Melissa Brown who gave me expedient ideas and insights into teaching English in the contemporary English class They challenged me to reach beyond what I had purported to achieve as an EFL teacher My only academic regret is that there wasn’t an extended opportunity to be all these great teachers’ student I am indebted to many of my colleagues and friends, who have not only encouraged and supported me with their belief in me but also helped me out with participants for my study I owe my loving thanks to Mariam Poghosyan, my sister and the second rater of the writing section of my study She sacrificed a few days of her well-deserved holiday carefully checking the compositions encompassed in the study v I bow my thanks to my dearest family, loved ones, and friends who have been unceasingly patient and understanding in the hope that one day this would all be worthwhile, please know that I deeply appreciate and treasure your sacrifices Finally, I dedicate this work and extend my eternal love and appreciation to my parents Zvart Poghosyan and Hmayak Poghosyan whose continuous support, undaunted faith in my abilities, and desire for me to fulfill myself as an individual, made it possible for me to attain my goal I’ll love you forever vi Table of Contents DEDICATION……………………………………………….……… …………… .III ACKNOWLEDGMENT…….…………………………………………………….………IV TABLE OF CONTENTS ………………………… …………….………….………… VII LIST OF ABBREVIATIONS ………………………………………………… …….…IX LIST OF TABLES …………………………………………………………………… X LIST OF FIGURES …………………………………………………………….………….X ABSTRACT…………………………………………………………………………….….XI Chapter 1: Introduction……………………………………… ……… ………………….1 1.1 Background of the Study……………………………………………………………… 1.2 Objectives of the Study 1.3 Significance of the Study 1.4 Research Paper Outline Chapter 2: Review of the related literature……………………………………… … … 2.1 Types and Purpose of Language Tests 2.2 Test Measurement Qualities…………………………………………………………….8 2.2.1 Language Test Reliability………………………………………………………… 2.2.2 Language Test Validity…………………………………………………………… 2.3 Construct Validity ………………… 11 2.4 Criterion Validity 17 2.5 Content Validity .18 2.5.1 Evaluation of Test Content 21 2.6 The Effect of Coaching/Tutoring on Validity 25 2.7 The Relationship of Alignment between Curriculum, Assessment, and Instruction to Content Validity 30 2.8 Statement of Purpose .32 Chapter 3: Methodology……………………………………………….…… ………… 33 3.1 Data and Participants .33 3.2 Instrumentation .34 3.3 Procedure…………………………………………………………………………… 36 3.4 Data Analysis …………………………………………………………………………42 Chapter 4: Result and Discussion…………………………………………………… … 44 4.1 Results 44 4.1.1 The Reliability Analysis of Tests and 44 4.1.2 The Validity Analysis…………………………………………………………… 45 4.1.3 The Analysis of the Production Section ………………………………………….61 4.2 Analysis of Qualitative Data .64 4.3 Discussion…………………………………………………………………………… 64 vii 4.4 Summary of the Findings 69 Chapter 5: Conclusion…………………………………………………………………… 71 5.1 Conclusion .71 5.2 Applications and Implications .73 5.3 Limitations and Delimitations of the Study 73 5.4 Suggestions for Further Research 75 References………………………………… ………………………………………….……77 Appendices…………………………………………………………….……….…… …….83 Appendix Test Specifications Based on English Textbook Series……………… 83 Appendix 1.1 Test 1- Achievement Test Designed for the Present Study… 89 Appendix 1.2 Table of Contents…………………………………………… 99 Appendix A Brief Presentation of the High School Curriculum Objectives of the English Language set by the MES………… ………………………….… 106 Appendix Table of Contents of the English Textbooks (Sample)… … … … 108 viii List of Abbreviations EFL English as a Foreign Language ETS Educational Testing Service MCQ Multiple Choice Questions MDS Multidimensional Scaling MES Ministry of Education and Science of Armenia RA Republic of Armenia SLA Second Language Acquisition SME Subject-matter Experts TLU Target Language Use USL & UEEE Unified School-leaving and University Entrance Exams in English ix List of Tables Table A…………………….……………………………………………………………….39 Table 1…………………….……………………………………………………………… 46 Table 2…………………….……………………………………………………………… 46 Table 3…………………………………………………………………………………… .48 Table 4………………………….………………………………………………………… 49 Table 4.1…………………………………………………………………………………51 Table 5………………………….………………………………………………………… 54 Table 6………………………….………………………………………………………… 55 Table 7………………………….………………………………………………………… 62 Table 8………………………….………………………………………………………… 63 List of Figures Figure 1…………………….……………………………………………………………….45 Figure 2…………………………………………………………………………………… 46 x Abstract The purpose of the present study was to investigate and determine whether and to what extent the content of the Unified School Leaving and University Entrance Examination in English (USL & UEEE) of 2009 was consistent with the curriculum (or teaching /learning objectives) of the English language program in secondary schools of Armenia The instructional materials (the textbooks for Grades to 10) approved of by the Ministry of Education and Science of Armenia and the High School teaching/learning objectives of the English Language set in the curriculum were analyzed to serve as a basis for the research Two tests (the USL & UEEE and a test designed exclusively for this study on the basis of the instructional materials) were administered to sixty-two school-leavers randomly selected from different secondary schools of Yerevan and the preparatory course of RussianArmenian (Slavonic) University The analyses of the results obtained (i.e the total scores on each test) revealed a strong association between the two tests However, the analysis of the relationship between the components of the two tests showed a weak or moderate degree of relationship in most cases, and most of the comparisons of the components between the tests displayed statistically significant differences The analysis of the production section revealed a low correlation between the scores on compositions and those on the tests suggesting that the test scores were not representative of the writing ability xi Word-building Participles L: Listen to the dialogue and answer the questions Gerunds Writing: Write a passage about your special interests Read the text “Wonderland” and the comprehension exercise Be good at; be fond of, be interested in, be keen on, be crazy about, be indifferent to, be popular with, have a dislike for  Likes and dislikes e.g I don’t mind, I like, I enjoy, I’m fond of, I’m crazy about… I can’t stand/bear, I hate… The use of the one P: Listen and repeat Read the text “The Art of Gardening” and retell it Read the text “Three Men in a boat” by J.K Jerome and the exercises 10 11 DP: Young people’s lifestyle today Prepositions Punctuation Read the passage from an article “The sporting Spirit” by G Orwell and the comprehension Phrasal verbs: put down (off, on, up, aside); Turn off (on, up, down); cheer up, calm down, give up, pick up; Ph V with get, go, give, ring, take, come, write, wake DP: Going to win  Feeling upset (sports and  Cheering up games) people e.g Risk (n, v), take a risk, risky, else, advice, special, specialize, taste, tasty, tasteless, tasteful, spare, mind (n, v); Tiny, breathe, breath, move, movement, movies, moving, needle, surgeon, earthquake, destroy Hobby and Interests Hobby, wide-spread, rare/ unusual, handiwork, modelmaking, knitting, embroidery, needle work, sewing, cooking, fishing, hiking, gardening, philately, playing the piano, dancing, painting, photography Collections Collect, a collection of ~, items of collection, album, arrange, rearrange, exchange, enrich Nation, national, nationality, surprise (v, n), surprising, nature, natural, by nature, forbid (forbade, forbidden) Attract, to attract smb.’s attention, support, overcome, give in, unless, attitude 115 exercise How disappointing! I’m so unlucky Bad luck Never mind Come on Cheer up Don’t worry It doesn’t matter Ouch! Oh! Yippee! Oh dear! Sports and Games To go in for sports, to take up some sport, to give up sport, champion, to be champion in, all-round champ., many-time champ., championship, compete, competitive, competition, cup, to end in draw, game, outdoor games, indoor games, referee, gym, record, to beat a record, score, with the score of, to keep the score of, to open the score, to even the score, to score a goal, stadium, sports (adj.), sportsman, train, trainer, team, victory, win, to win a match/a game of chess, to win the victory, to win the prize, winner 116 1a 1b Grammar focus Listening / Pronunciation No listening tasks or exercises are found Unit Subunits English Textbook Writing/reading Speaking Expressions, speech features Read the text “Travelling to Britain” and answer the questions D: Read the dialogue “Philip meets the Browns” and act it out Read the text “A Wall of Water” and the comprehension exercises D: Read the dialogue “Sea or Air?” and act it out Vocabulary/ Topical vocabulary To travel, travels, journey, tour, to tour round, trip, to go on a trip To look forward to, at last, at least To sail, sailing, sailor, to sink, on board, to pull, to break up, to break out, to turn over, mess, be in a mess, to differ, to differ from, different, to pick up; Lifeboat, harbor, deck, crew, declare, goods, storm, duties, channel, Customs, hovercraft, route 117 1c Revision: Modal Verbs: can, may, must & their substitutes in the future and in the past D: Read the dialogue “Going through the customs” and act it out 1d Writing: Write a short story: “Next year I am going to travel to …” 2a Read the text “The United Kingdom” and answer the questions Going through the customs May I see your bag, please? Would you come this way, please? Have you got anything to declare? Island, isle, islander, to occupy, to be washed by, separate (v, adj.), channel, canal, numerous, shore, bank, coast D: Read the dialogue “The Map of Britain” and act it out Use the map and add whatever you know about the UK 2b Read the text “The Prince of Wales” and the comprehension exercises 2c 2d To choose, choice, to be pleased with, to promise (v, n), loyal, royal Revision: Comparative adjectives; Comparative adverbs; Writing: Write a short story: “My Native Land is rather small” Look at the statistics and point out the similarities and differences between the given countries A lot (very little) in common; fairly similar; a great deal; nothing like…; There isn’t much difference; 118 3a Read the text “Pictures of London; Where the tourists go” and answer the questions visit (v, n), to go on a visit, to pay a visit to, someone, visitor, battle, famous, be famous for, something, sights, go sightseeing, climb (up, down), guard (v, n), be on guard, far, a long way; to leave (for), to enjoy, to enjoy oneself Parking space, to park, to worry (about), be worried about, belong to, to rob, robbery, robber, at the bottom, at the top, stick, suburb, raven D: Read the dialogue “Ronny’s First Day” and act it out 3b Read the text “The View was fantastic” and answer the questions 3c 3d Revision: Mixed tenses; Relative Pronouns: who, which, that, whose Writing: Write a short story: “A visit to Yerevan.” Attracting attention and asking the way Excuse me Yes? Pardon me Look here! Well? I say… What can I for you? Could you tell me …? (how to get to, the way to, where the nearest … is) 119 4a Read the text “Welcome to Wales” and answer the questions D: Read the dialogue “Wales isn’t in England” and act it out Read the text “The Legend of King Arthur” and the comprehension exercises 4b Talk about Wales and the Welsh Area, industry, industrial, strike, be/go on strike, to protect, countryside, to beat, to realize, to keep one’s promise, to keep silence, to keep a secret, to keep free, pride, be proud of Power, powerful, danger, dangerous, safe, safety, accept, except, peace, at peace, in peace, peaceful, to appear, news, to follow, to touch, to ride, imaginary, to fetch Life, live (v, n), alive, living, die, death, dead, dying Talking about the story 4c Revision: Mixed tenses; Adjective vs Adverb Latest vs last; farther vs further; nearest vs next 4d Writing: Write a short story: “My favorite historical (legendary) figure” Ways of expressing Requests Will you (please)…? Would you…? Could you…? May I ask you to …? May I trouble you for…? 120 5a Read the text “Scotland” and answer the questions To divide (into), divide smth in half, lonely, rough, to catch, grow, to surround, surrounding, surroundings, to contain, to hold; Lovely, splendid, to exhibit, exhibition, a good deal, To lose, loss, hope (v, n), diver, setting, unique, independence, celebrity, loch D: Read the dialogue, answer the questions and act it out 5b 5c Read the text “Bonnie Prince Charles” and the comprehension exercises Passive Voice; Participles (Present & Past) 5d Writing: Write a short story: “Let’s visit Scotland” 6a Read the text “Holidays and Customs” and answer the questions D: Read the dialogue and act it out Read the text “Two presents” and the comprehension exercises 6b 6c Making Suggestions Let’s…; Why don’t we…? What about going…? Why not…? To imagine, imagination, to bore, boring, to excite, exciting, special, to guess, funny, to play tricks on, to blow up, to collect, collection Eve, save, tear, expression, to express, to burn, to admire, admiration, step (n, v) Revision: Reported speech 121 6d Writing: Write a short description: “I sent a card to my mother, brother, friend on…” Congratulations and Wishes 7a Read the text “Going to school in Britain” and answer the questions Talk about school uniform 7b Read the text “A Letter to a Penfriend” and the comprehension exercises Birthday Happy birthday! Many happy returns of the day! Holidays Happy/merry Christmas! Happy New Year! Happy Easter! Best Wishes for … Day Passing an exam Well done! Taking an exam Good luck! The Best of luck! Comprehensive school, to wear, break, assembly, to pray, prayer, activity, uniform, choir Subject, timetable, computer buff, headmaster, form master, caretaker, band; Staff room, to kick, to tell on, fair, to own up Read the passages “The Football” and the comprehension exercises 7c Reflexive Pronouns; Reciprocal Pronoun: each other Relative Pronouns: who, which, whose 122 7d Writing: Write a short story: “My School” 8a Read the text “Some days are like that” and answer the questions 8b Read the passages “Light and Dark” and the comprehension exercises D: Read the dialogue “It’s out of the question” and act it out Do the comprehension exercises 8c Expressing anger, irritation, surprise It’s a nightmare Leave me alone! It’s none of your business! It’s the limit! It’s awful! What a shame! Shame on you! What a cheek! How annoying! You don’t say so! Is that so! Dear me! Indeed? Really? To manage, manage fine, average, busy, busy (of places), a busy day, be busy with, storey, multi-storey, get oneself ready, get smth ready, to tidy, towards Charge, in charge, to miss, to remember, memory Traffic warden, nearly, pitch dark, to reach, to vacuum, boring, rushhour, vacuum cleaner, dish-washer, to be over, show, firm, tin-opener, hair-dryer, cooker Revision: The Past Perfect tense Word-building: words ending in -er 123 8d Writing: Make up a dialogue on a telephone conversation Read the text “Road Safety” and answer the questions; correct the wrong sentences; put the sentences together 9a 9b 9c 9d Taking a message Read the text “In Hospital” and put the sentences together Message, to give someone a message, to take a message I’m afraid; be/get well Accident, crash (n, v), to hurt, to injure, injury, to damage, cause, care, careful, careless, to obey, traffic Hit (n, v), emergency, service, operator, call (n, v), to dial, ambulance, to put smb through, to happen to to lend, to borrow, to watch out Revision: Past perfect; Past continuous Writing: Look at the pictures and the notes and write a short story Helping an injured person; Emergency; Warning and Caution 124 Grammar focus Writing /reading 1a Expressions, speech features Vocabulary/ Topical vocabulary Revision + Check up 1b 1c Speaking Read the text “My New School of thought” (Part 1) and the comprehension exercises Revision: Adj vs adv.; Subject verb agreement, the definite article “the”; Past Simple vs Present Simple; Present Perfect; Modal Auxiliaries: can, can’t, have to, don’t have to, may, must, must not, should, ought to Verbs followed by adjectives: smell, taste, look, sound Talking Points based on the text and topic under discussion Unit Subunits English Textbook Ancient, distribute, probably, astonish, compel, accept, approve, damage, to satisfy, Prepositions: into, out of, behind, in front of, up, down, inside, outside, between, among, near, far from, through, over, in the middle of, with 125 2a Greeting: Hello Hi How are you? Agreeing: Certainly, Sure With pleasure Parting: See you! Cheerio So long Invitation: Come in, please I’d like to ask you over for dinner Gratitude: Thanks a lot It’s no trouble Thanks for everything You’re welcome Introducing people: Let me introduce … How you do? Nice to meet you Expressing enthusiasm: Great, Well done, Magnificent Fantastic Terrific Expressing regret: What a pity Expressing anger: Nonsense Shame on you How could you? Apologizing: Sorry No problem Requesting: May I …? Could you please (say it again)? Warning: Look out! Be careful 2b Asking and answering Special Questions Questions with: who, what & which Read the text “My New School of thought” (Part 2) and the comprehension exercises Weary, disgust, thrash, amazed, sympathy, brutal, punish, acquaintance Questions with a preposition at the end 3a Shop-assistant, tailor, librarian, park keeper, conductor, cobbler, barber, dress-maker, 126 3b 3c Stative verbs; 4a Word building: -ment Passive Voice Read the text “The Selfish Giant” (part 1) and the comprehension exercises to wander, selfish, blossom, delicate, hail, delight, notice board, embarrassed, thirsty, hungry, confused, courteous, scared, doubtful, frightened, sleepy, mad, joyful, Preps: wait for, angry with, graduate from, ready for, listen to, kind to, agree with, afraid of lie-lay-lain, lay-laid-laid; Read the text “The Selfish Giant” (part 2) and the comprehension exercises grow, knock down, rest, laugh, melt, sight, casement, pass by, huge, be awake, to cease Read the text “How King Alfred Learnt to Read” and the comprehension exercises Read the text “English Mealtimes and English Food” and the comprehension exercises Proud (ly), patient (ly) , different, clever, valuable, useful, interesting, encourage, to calm, to master, rule, legend Food: fried (meat, fish, eggs), chipped, broth, cabbage, porridge, boiled, sausage, chops, beer, ham, a piece of pie, hot chocolate Berries: strawberries, raspberries, gooseberries, currants Vegetables: onion, carrot, tomato, garlic, potatoes, been, aubergine, egg-plant Fruit: peach, grapes, pear, melon, watermelon, cherries, plum, banana Past Perfect 4b 5a 5b 6a Word building: -able, -ion, -ing, -ness, ful Let’s … Do (one’s homework, an assignment, the housework, the typing, the ironing, the laundry) Make (breakfast, lunch, dinner, the bed, dessert, plans, arrangements, a telephone call, an effort) To run out of smth 127 6b 6c 7a Revision: Simple Past vs Past Continuous; Sentence Construction; Passive Voice -ing after preps: after, before, while, Read the text “King Alfred and the Cakes” and the comprehension exercises to raise, to win, to burn, to bow, to save, to amuse, to dream, to tear, to watch, alone Read the text “How We Kept Mother’s Day” and the comprehension exercises to celebrate, to pack, to arrange, to stay, to hire, to decide, to fetch, to realize, to turn out Reported Speech 7b 8a A/an vs some 8b Possessive Pronouns Reflexive Pronouns 9a Less vs fewer Read the text “The Mystery of Thor Bridge” and the comprehension exercises Have a cold (stomach-ach, toothache, headache, a sore throat), Department store, supermarket, baker’s shop/bakery, chemist’s, butcher’s, grocer’s greengrocer’s, dairy, fishmonger’s, jeweler’s shop, fur hop, footwear, leather wear, gift shop, bookshop, stationery, perfumery, household goods, confectionary, florist’s fancy goods, toy shop, furniture, shoe polish, gloves, glue, razor, insect killer, window cleaning liquid, skis Violent temper, look after, connected with, estate, suspect, governess, villain, liar, cruel, reputation, creature, bullet, harm, innocent, fly, Parts of body: appendix, bone, heart, brain, kidney, liver, lung, muscle, nerve, skin, stomach, vein, neck 128 9b 10a 10 10b 11a 11 11b Clauses of time and condition Read the text “An Explosion at Night” and the comprehension exercises Read the text “Halloween is a Holiday for Children in America” and the comprehension exercises Read the text “Back to Earth” and the comprehension exercises Read the text “The Pool of Tears” (extract from “Alice in Wonderland”) and the comprehension exercises I would (I’d) like to do… I wonder if… I wonder (who)… Suddenly, immediately, angrily, quiet, difficult, terrible, dark, crime, patient, recover, whisper, stream, tremble Beside, besides, except, hard, hardly Celebrate, holy, All Saints’ Day, pumpkin, jack-o-lantern, lantern, monster, trick, treat, adult, candy, Talkative, hard-working, lazy, moody, easy-going, sociable, optimistic, reliable, shy, untidy, reserved, impatient Exclaim, manage, desperate, go on, strike, trot, curiosity, queer, a great deal, wonder Supplementary Reading (Topic: Animals) Read the text “Brown Wolf” and the comprehension exercises Disappear, spring, lick, bark, return, drag, bring up 129 ... for the entrance exams of RAU The English exam of RAU slightly differs from that of the Unified School Leaving and University Entrance Examination in English (USL and UEEE) It consists of three... validity by this, since the examination of the generalizability of score interpretation is a significant part of construct validity too Interactiveness is the extent and type of involvement of. .. for validity Reliability can be seen as one type of validity evidence Validity was often established through correlations of a test with other tests Validity is argued on the basis of a number of