Assessing writing (2009)

Assessing Writing THE CAMBRIDGE LANGUAGE ASSESSMENT SERIES Series editors: J Charles Alderson and Lyle F Bachman In this series: Assessing Vocabulary by John Read Assessing Reading by J Charles Alderson Assessing Languages for Speci®c Purposes by Dan Douglas Assessing Listening by Gary Buck Assessing Writing by Sara Cushing Weigle Assessing Writing Sara Cushing Weigle CAMBRIDGE UNIVERSITY PRESS Cambridge, New York, Melbourne, Madrid, Cape Town, Singapore, São Paulo, Delhi, Dubai, Tokyo Cambridge University Press The Edinburgh Building, Cambridge CB2 8RU, UK www.cambridge.org Information on this title: www.cambridge.org/9780521784467 © Cambridge University Press 2002 This publication is in copyright Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press First published 2002 6th printing 2009 Printed in the United Kingdom at the University Press, Cambridge A catalogue record for this publication is available from the British Library ISBN 978-0-521-78446-7 Paperback ISBN 978-0-521-78027-8 Hardback Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate Information regarding prices, travel timetables and other factual information given in this work are correct at the time of first printing but Cambridge University Press does not guarantee the accuracy of such information thereafter To my family: Clarke, Tommy, and James Contents Series Editors' Preface Acknowledgements Introduction page x xii The nature of writing ability 14 Basic considerations in assessing writing 39 Research in large-scale writing assessment 58 Designing writing assessment tasks 77 Scoring procedures for writing assessment 108 Illustrative tests of writing 140 Beyond the timed impromptu test: Classroom writing assessment 172 Portfolio assessment 197 10 The future of writing assessment Bibliography Index 230 245 261 ix 254 ASSESSING WRITING Land, R E and Whitley, C (1989) Evaluating second language essays in regular composition classes: Towards a pluralistic U.S In D M Johnson and D H Roen (eds.), Richness in writing: Empowering ESL students (pp 284±293) New York: Longman Leki, I (1992) Understanding ESL writers NH: Heinemann Educational Books Leki, I and Carson, J (1997) Completely different worlds: EAP and the writing experiences of ESL students in university courses TESOL Quarterly 31 (1), 39±70 Lewkowicz, J (1997) Investigating authenticity in language testing Unpublished doctoral dissertation, University of Lancaster Lloyd-Jones, R (1977) Primary trait scoring In C R Cooper and L Odell (eds.), Evaluating writing 33±69 NY: National Council of Teachers of English Lumley, T (forthcoming) Assessment criteria in a large-scale writing test: What they really mean to the raters? Language Testing Lynch, B K and Davidson, F (1994) Criterion-referenced language test development: Linking curricula, teachers and tests TESOL Quarterly 28 (4), 727±43 Mabry, L (1999) Portfolios plus: A critical guide to alternative assessment Thousand Oaks, CA: Corwin Madigan, R., Linton, P., and Johnson, S The paradox of writing apprehension In Levy, C M and S Ransdell (eds.), The science of writing Hillsdale, NJ: Lawrence Erlbaum Associates Markham, L R (1976) In¯uence of handwriting quality on teacher evaluation of written work American Educational Research Journal 13, (4), 277±283 Matalene, C (1985) Contrastive rhetoric: An American writing teacher in China College English 47, 789±807 McNamara, T F (1996) Measuring second language performance London and New York: Longman MELAB Technical Manual (1996) Ann Arbor, MI: University of Michigan Press Mendelsohn, D and Cumming, A (1987) Professors' ratings of language use and rhetorical organizations in ESL compositions TESL Canada Journal (1), 9±26 Messick, S (1989) Meaning and values in test validation: The science and ethics of assessment Educational Researcher 18 (2) 5± 11 Messick, S (1994) The interplay of evidence and consequences in the validation of performance assessments Educational Researcher 23, 2, 13±23 Moss, P A (1994) Can there be validity without reliability? Educational Research 23 (2), 5± 12 Murphy, S and Camp, R (1996) Moving towards systematic coherence: A discussion of con¯icting perspectives in portfolio assessment In R Calfee Bibliography 255 and P Perfumo (eds.), Writing portfolios in the classroom: Policy and practice, promise and peril Mahwah, NJ: Lawrence Erlbaum Associates Murphy, S and Ruth, L (1993) The ®eld testing of writing prompts reconsidered In M W Williamson and B A Huot (eds.), Validating holistic scoring for writing assessment: Theoretical and empirical foundations Cresskill, NJ: Hampton Press Nesi, H and Meara, P (1991) How using dictionaries affects performance in multiple-choice EFL tests Reading in a Foreign Language (1), 631±643 Nold, E W and Freedman, S W (1977) An analysis of readers' responses to essays Research in the Teaching of English 11, 164±174 Norris, J., Brown, J D., Hudson, T and Yoshioka, J (1998) Designing second language performance assessments University of Hawai'i at Manoa: Second Language Teaching and Curriculum Center North, B and Schneider, G (1998) Scaling descriptors for language pro®ciency scales Language Testing 15 (2) 217±263 Northwest Evaluation Association (1991) Portfolios Portfolio News (3), Ostler, S (1987) English in parallels: A comparison of English and Arabic prose In U Connor and R Kaplan (eds.), Writing across languages: Analysis of L2 text, Reading, MA: Addison-Wesley Page, E (1966) The imminence of grading essays by computer Phi Delta Kappan, 46, 238±243 Page, E (1968) The use of computers in analyzing student essays International Review of Education 14, 127±142 Page, E (1994) Computer grading of student prose: Using modern concepts and software Journal of Experimental Education 62 (2), 127±142 Page, E and Peterson, N S (1995) The computer moves into essay grading: Updating the ancient text Phi Delta Kappan March, 561±565 Palmquist, M., Kiefer, K., Hartvigsen, J and Goodlew, B (1998) Transitions: Teaching in computer-supported and traditional classrooms Greenwich, CT: Ablex Palmquist, M and Young, R (1992) The notion of giftedness and student expectations about writing Written communication (1), 137±168 Par®tt, M (1997) What kind of discourse? Thinking it through Annual Meeting of the Conference on College Composition and Communication (ERIC Document No ED 418 406) Peirce, B N (1995) Social identity, investment, and language learning TESOL Quarterly 29 (1), 9±31 Pennington, M (1996) The computer and the non-native writer Creskill, NJ: Hampton Press, Inc Perl, S (1979) The composing process of unskilled college writers Research in the Teaching of English 13 (4), 317±336 Peterson, N S (1997) Automated scoring of written essays: Can such scores be valid? Paper presented at NCME in Chicago, 26 March 256 ASSESSING WRITING Polio, C and Glew, M (1996) ESL Writing assessment prompts: How students choose Journal of Second Language Writing (1), 35± 49 (ERIC Document Reproduction Service No EJ 527 730) Pollitt, A (1990) Response to Charles Alderson's paper: `Bands and scores.' Alderson, J C (1991) In J C Alderson and B North (eds.), Language testing in the 1990s: The communicative legacy London: Modern English Publications/British Council/Macmillan, 87± 91 Popham, W J (1978) Criterion-referenced measurement Englewood Cliffs, NJ: Prentice Hall Powers, D E., Fowles, M E., Farnum, M and Ramsey, P (1994) Will they think less of my handwritten essay if others words process theirs? Effects on essay scores of intermingling handwritten and word-processed essays Journal of Educational Measurement 31 (3), 220±233 Powers, D E and Fowles, M E (1996) Effects of applying different time limits to a proposed GRE writing test Journal of Educational Measurement 33 (4) 433±452 Purves, A (1992) Re¯ection on research and assessment in written composition Research in the Teaching of English 26, 108± 122 Purves, A C., Soter, A., Takala, S and VaÈhaÈpaÈssi, A (1984) Towards a domainreferenced system for classifying assignments Research in the Teaching of English 18 (4), 385± 416 Quellmalz, E S., Capell, F J and Chou, C P (1982) Effects of discourse and response mode on the measurement of writing competence Journal of Educational Measurement 19 (4), 242± 258 Raimes, A (1985) What unskilled ESL students as they write: A classroom study of composing TESOL Quarterly 19 (2) 229± 258 Raimes, A (1991) Out of the woods: Emerging traditions in the teaching of writing TESOL Quarterly 25 (3) 407±430 Ransdell, S and Levy, M (1996) Working memory constraints on writing quality and ¯uency In C Levy and S Ransdell (eds.), The science of writing NJ: Lawrence Erlbaum Associates Reid, J (1990) Responding to different topic types: A quantitative analysis from a contrastive rhetoric perspective In B Kroll (ed.), Second language writing: Research insights for the classroom New York: Cambridge University Press Reid, J (1998) Responding to ESL student language problems: Error analysis and revision plans In P Byrd and J Reid (eds.), Grammar in the composition classroom: Essays on teaching ESL for college-bound students (pp 118±137) Reid, J and Byrd, P (1998) Writing to persuade and the language of persuasion In P Byrd and J Reid (eds.), Grammar in the composition classroom: Essays on teaching ESL for college-bound students (pp 101±117) Reid, J and Kroll, B (1995) Designing and assessing effective classroom Bibliography 257 writing assignments for NES and ESL students Journal of Second Language Writing (1), 17±41 Ruth, L and Murphy, S (1984) Designing topics for writing assessment: Problems of meaning College Composition 35 (4) 410±421 Ruth, L and Murphy, S (1988) Designing writing tasks for the assessment of writing Norwood, NJ: Ablex Saari, H and Purves A C (1992) The curriculum in mother-tongue and written composition In A C Purves (ed.), The IEA study of written composition II: Education and performance in fourteen countries Oxford: Pergamon (37±86) Sacks, H., Schegloff, E and Jefferson, G (1974) A simplest semantics for the organization of turn-taking conversation Language 50, 696± 735 Santos, T (1988) Professors' reactions to the academic writing of nonnativespeaking students TESOL Quarterly 22 (1), 69±90 Scharton, M (1996) The politics of validity In E M White, W D Lutz and S Kamusikiri (eds.), Assessment of writing: Politics, policies, practices New York: The Modern Language Association of America Schumann, J H (1978) The pidginization process: A model for second language acquisition Rowley, MA: Newbury House Publishers Scott, V M (1996) Rethinking foreign language writing Boston: Heinle and Heinle Shen, F (1988) The classroom and the wider culture: Identity as a key to learning English composition (Staffroom Interchange) College Composition and Communication 40, 4, 459± 66 Shermis, M and Burstein, J (forthcoming) Automated essay scoring: A cross-disciplinary perspective Hillsdale, NJ: Lawrence Erlbaum Associated Shohamy, E (1998) Critical language testing and beyond Studies in Educational Evaluation 24 (4), 331±345 Shohamy, E., Denitsa-Schmidt, S and Ferman, I (1996) Test impact revisited: Washback effect over time Language Testing 13 (3), 298± 317 Shohamy, E., Gordon, C and Kraemer, R (1992) The effect of raters' background and training on the reliability of direct writing tests Modern Language Journal 76(4), 513±521 Silva, T (1993) Toward an understanding of the distinct nature of L2 writing: The ESL research and its implications TESOL Quarterly 27, 657±77 Sloan, C and McGinnis, I (1982) The effect of handwriting on teachers' grading of high school essays Journal of the Association for the Study of Perception 17 (2), 15±21 Smith W L., Hull, G A., Land, R E., Moore, M T., Ball, C., Dunham, D E., Hickey, L S., and Ruzich, C W (1985) Some effects of varying the structure of the topic on college students' writing Written Communication 2, 73± 89 258 ASSESSING WRITING Sommer, N (1980) Revision strategies of student writers and experienced adult writers College Composition and Communication 31, 378±88 Spaan, M (1993) The effect of prompt on essay examinations In D Douglas and C Chapelle (eds.), A new decade of language testing research (pp 98±122) Alexandria, VA: TESOL Spack, R F (1988) Initiating ESL students into the academic discourse community: How far should we go? TESOL Quarterly 22 (1), 29± 52 Spalding, E and Cummins, G (1998) It was the best of times It was a waste of time: University of Kentucky students' view of writing under KERA Assessing Writing (2), 167±199 Sperling, M (1991) High school English and the teacher±student writing conference: Fine tuned duets in the ensemble of the classroom (Occasional Paper No #26) Berkeley, CA: Center for the Study of Writing Sperling, M (1996) Revisiting the writing±speaking connection: Challenges for research on writing and writing instruction Review of Educational Research 66, 53±86 Sproull, L and Kiesler, S (1986) Reducing social context cues: Electronic mail in organization communication Management Science 32, 1492±1512 Stewart, M and Grobe, C (1979) Syntactic maturity, mechanics of writing, and teachers' quality ratings Research in the Teaching of English 13 (3) 207±15 Stock, P L and Robinson, J L (1987) Taking on testing English Education 19, 93±121 Sullivan, F J (1987) Negotiating expectations: Writing and reading placement tests Paper presented at the meeting of the Conference of College Composition and Communication, Atlanta, GA Swales, J (1990) Genre analysis: English in academic and research settings Cambridge: Cambridge University Press Sweedler-Brown, C O (1985) The in¯uence of training and experience on holistic essay evaluation English Journal 74 (5) 49± 55 Sweedler-Brown, C O (1993) ESL essay evaluation: The in¯uence of sentence-level and rhetorical features Journal of Second Language Writing (1), 3± 17 Taylor, C., Jamieson, J., Eignor, D and Kirsch, I (1998) The relationship between computer familiarity and performance on computer-based TOEFL test tasks (TOEFL Research Report No 61) Princeton, NJ: Educational Testing Service Tedick, D (1990) ESL writing assessment: Subject-matter knowledge and its impact on performance English for Speci®c Purposes 9, 123±43 Tedick, D and Mathison, M (1995) Holistic scoring in ESL writing assessment: What does an analysis of rhetorical features reveal? In D Belcher and G Braine (eds.), Academic writing in a second language: Essays on research and pedagogy (pp 205± 230) Norwood, NJ: Ablex Bibliography 259 Truscott, J (1996) The case against grammar correction in L2 writing classes Language Learning 46 (2), 327±369 UCLES (1997) First Certi®cate in English: a handbook Cambridge: UCLES UCLES (2002) International English Language Testing System Cambridge: UCLES, The British Council, IDP Education, Australia ValdeÂz, G., Haro, P and Echevarriarza, M (1992) The development of writing abilities in a foreign language: Contributions toward a general theory of L2 writing Modern Language Journal, 76, 3, 333± 52 Vaughan, C (1992) Holistic assessment: What goes on in the rater's mind? In L Hamp-Lyons (ed.), Assessing second language writing in academic contexts (pp 111±26) Norwood, NJ: Ablex VaÈhaÈpaÈssi, A (1982) On the speci®cation of the domain of school writing In A C Purves and S Takala (eds.), An international perspective on the evaluation of written composition (pp 265±289) Oxford: Pergamon Wall, D (1996) Introducing new tests into traditional systems: Insights from general education and from innovation theory Language Testing 13 (3), 334±354 Weaver, F (1973) The composing process of English teacher candidates: Responding to freedom and constraint Unpublished doctoral dissertation, University of Illinois, Champaign-Urbana, IL Weigle, S C (1994) Effects of training on raters of ESL compositions Language Testing 11, 197± 223 Weigle, S C (1998) Using facets to model rater training effects Language Testing 15 (2), 263±87 Weigle, S C (1999) Investigating rater/prompt interactions in writing assessment: Quantitative and qualitative approaches Assessing Writing (2), 145±178 Weigle, S C and Jensen, L (1997) Assessment issues for content-based instruction In M A Snow and D Brinton (eds.), The content-based classroom: Perspectives on integrating language and content (pp 201± 212) White Plains, NY: Addison Wesley Longman Weigle, S C and Nelson, G (2001) Academic writing for university examinations In I Leki (ed.), Academic writing programs (pp 121±135) Alexandria, VA: TESOL Weigle, S C., Lamison, B and Peters, K (2000) Topic selection on a standardized writing assessment Paper presented at Southeast Regional TESOL, Miami, FL, October Weir, C J (1988) Construct validity In A Hughes, D Porter and C J Weir (eds.), ELTS Validation project report (ELTS Research reports (ii)) London: The British Council/UCLES Weir, C J (1990) Communicative language testing NJ: Prentice Hall Regents White, E M (1984) Holisticism College Composition and Communication 35 (4), 400±409 260 ASSESSING WRITING White, E M (1985) Teaching and assessing writing San Francisco, CA: JosseyBass White, E M (1994) Teaching and assessing writing: Recent advances in understanding, evaluating and improving student performance (2nd ed.) San Francisco: Jossey-Bass White, E M (1995) An apologia for the timed impromptu essay test College Composition and Communication 46, 30±45 White, E M (1996) Power and agenda setting in writing assessment In E M White, W D Lutz and S Kamusikiri (eds.), Assessment of writing: Politics, policies, practices New York: The Modern Language Association of America Willard-Traub, M., Decker, E., Reed, R and Johnston, J (1999) The development of large-scale portfolio placement assessment at the University of Michigan: 1992±1998 Assessing Writing (1), 41±84 Wolcott, W (with Legg, S M.) (1998) An overview of writing assessment: Theory, research and practice Urbana, IL: National Council of Teachers of English Yorkey, R (1977) Practical EFL techniques for teaching Arabic-speaking students In J Alatis and R Crymes (eds.), The human factors in ESL Washington: TESOL Zamel, V (1983) The composing processes of advanced ESL students: Six case studies TESOL Quarterly 17, 165± 187 Index academic writing skills assessment problems 93, 95 authenticity 52, 93, 95, 146 ± 47 content-based writing 187, 188 ± 89 construct de®nition 79, 80 correctness 17 ®rst-language education ± IELTS Academic Module 155 ± 56, 158, 160, 160t, 161 personal topics 92 portfolio assessment 204, 206, 229 process-oriented writing 202 school 4, ± 6, 6t, 11t scoring 188 ± 89, 190 ± 91f second language learner groups 6, 6t, 11t social aspects 19 ± 20, 178 stimulus material 94 ± 95 text-responsible writing 188 ± 89, 190 ± 91f textual interaction 94 ± 95 TLU domain, re¯ecting 189, 191 see also TOEFL accuracy 17, 146 content 189 ACTFL (American Council on the Teaching of Foreign Languages) 122 Pro®ciency Guidelines 166, 167 Adams, R 120 Alcaya, C 167 Alderson, J.C 41, 55, 77, 82, 83, 119, 122, 129, 135, 167 analytic scoring 114 ± 20 versus holistic 72 ± 73, 120 ± 22, 220 portfolio assessment 222, 224, 225 ± 27f ANOVA (analysis of variance) 135 anxiety, writing 37 Arnold, V 105 Arrington, P 95 Aschbacher, P 212 Ash, B.H 68 authenticity 51 ± 52, 121t academic writing skills 52, 93, 95, 146 ± 47 BEST 166 classroom assessment 175 ± 76, 186 ± 88 FCE 154 genre 96, 98 ± 100 IELTS 161 ± 62 portfolio assessment 203 TOEFL Writing Test 146 ± 47 word processing 147, 148 Bachman, L.F 3, 18, 29, 39, 40 ± 41, 42 ± 46, 47, 48 ± 56 77, 79, 80, 81, 82, 83, 96, 103, 119, 120, 121t, 123, 125, 126, 134, 135, 180, 200, 203, 204, 244 Bailey, K.M 55 Baron, N.S 231 Basic English Skills Test see BEST Bauer, B.A 73 Belanoff, P 198 Bensoussan, M 106 Bereiter, C Bereiter and Scardamalia model (1987) 29 ± 35, 44 Bernhardt, E 5, 6t BEST (Basic English Skills Test) 140 ± 41 content 162 ± 63, 164f distinctive features 163 ± 64 purpose 162 261 262 ASSESSING WRITING BEST (cont.) scoring 162 ± 63, 165f test usefulness 164 ± 66 Biber, D 100 Bloom, B.S 63 Boldt, H 189 Braine, G 233 Breland, H M 69, 71, 236 Bridgeman, B 142 British Council 155 Brossell, G 68 Brown, H.D 15 ± 16, 20 Brown, J.D 67, 71 Burnstein, J 235, 236 Butler, F.A 86, 87 ± 88f, 89, 96 Byrd, P 100 Cambridge First Certi®cate in English see FCE Camp, R 3, 59, 204 ± 6, 215, 221, 241, 242 Canale, M 29 Carlson, S 142 Carr, N 73 Carrell, P 21, 70 Carroll, J.B 50 Carson, J 103, 174, 188 ± 89 Cast, B M J 73 Center for Applied Linguistics (Washington) 140 ± 41 Chapelle, C.A 28, 42, 50 Chase, C.I 69, 105 Chisolme, I.M 237 Chodorow, M 236 Chung, G.K.W.K 234, 235 classroom assessment authenticity preparation 186 ± 87 revision 186, 187 ± 88 experience levels 177 interactiveness 186, 188 multiple writing tasks 185 ± 86 out-of-class writing 185 versus in-class writing 173 ± 75, 178 re¯ective writing 176 ± 77 scoring academic writing 188 ± 89, 190 ± 91f feedback 184, 192, 195f grading system 183 ± 84 inter-rater reliability 182 ± 83 speci®city 188 structure/composition course 191 ± 92, 193 ± 94f teaching tool 182 social aspects 177 ± 78 task design 179 ± 80 test usefulness 175 ± 76, 185 cognitive processes 19 knowledge, sources of 23, 24f, 26f, 28 versus linguistic skills 39 ± 40 oral and written language 15, 16, 17 ± 18 text types ± 9t, 10 ± 12 variance 63, 63t, 65 writing expertise 22 ± 23, 27, 31 ± 32, 33f, 35 see also writing process, models of Cohen, A.D 109, 172 Collado, A.V 21 Colorado Foreign Language Sample Pro®ciency Project 98, 99f Common European Framework of Reference: Learning, Teaching, Assessment (Council of Europe) 203 computer scoring 56, 242 E-rater 235 ± 36 Latent Semantic Analysis (LSA) 234 ± 35 opposition to 236 Project Essay Grade (PEG) 234 ± 35 Condon, W 198 ± 200, 202, 204, 206, 207, 208, 213, 217, 228, 240 Connor, U 70 construct de®nition 41 ± 42 BEST 164 ± 65 course syllabus based 79 CoWA 167, 170 e-mail 233 ± 34 FCE 153 ± 54 IELTS 160 ± 61 strategic competence 79 ± 80 technological change 233 ± 34 test development 79 ± 80, 92 ± 93 theory-based 79 TOEFL Writing Test 146 topic knowledge 44 ± 45, 80 construct validity 121t classroom assessment 175 ± 76 demonstrating 51 empirical evidence 50 ± 51, 81 personal topics 92 portfolio assessment 201 ± scoring procedures 136 ± 38, 139 speci®city 49 ± 50 content-based writing 187, 188 ± 89, 203 Index context 60, 61f, 73 ± 74 Contextualized Writing Assessment see CoWA correctness 15, 17, 83 CoWA (Contextualized Writing Assessment) 98, 141 content 166 ± 67, 168f distinctive features 167 purpose 166 scoring 167, 169f test usefulness 167, 170 critical thinking skills Crowhurst, M 65 cultural factors motivation 37 time allotment 101 ± writing ability 20 ± 22 Cumming, A 35, 70, 71 Cummins, G 207, 221 Daigon, A 234 Daly, J.A 37, 73 Davidson, F 77, 78, 83, 84, 84t, 85f diagnostic information 114, 120, 124 Dickson-Markman, F 73 Diederich, P.B 72 dif®culty, task 103 ± Dillon, W.T 202, 206 direct testing 3, 58 ± 59 discourse mode 100 ± 101 discrete-point testing 146 Dobson, B 95 Douglas, W.T 42, 43t, 44, 45, 80, 82, 83 Drechsel, J 236 Dweck, C 25 e-mail construct de®nition 233 ± 34 speech-like writing 231 ± 32 EAP (English for Academic Purposes) 52, 174 Educational Testing Service (US) 59, 236, 237 Eisterhold, J.C 21 Elbow, P 198 Elliot, N 202 English for Academic Purposes see EAP Ericsson, K.A 22 error correction 184 European Language Portfolio project (Council of Europe) 202 ± 3, 206 ± 263 FCE (Cambridge First Certi®cate in English) 140 content 149, 150 ± 51f distinctive features 153 purpose 148 scoring 149, 151 ± 53 test usefulness 153 ± 54 Feak, C 95 feedback 196 Ferris, D 105, 182 ± 83, 184 First Certi®cate in English see FCE ®rst-language writing ± Flower, L.S 22 Foltz, P.W 235 Fowles, M.E 101 Francis, J C 120 Freedman, S.W 69, 72, 73 functional literacy skills 6, 6t, 11, 11t Gardner, H 37 general impression marking 112, 149 general impression scale 149 genre 28, 29, 35 authenticity 96, 98 ± 100 Glew, M 75 GMAT (Graduate Management Admissions Test) 101, 235, 236 Gould, J.D 25 Grabe, W 20 ± 21, 34 ± 35, 37, 79 Grabe and Kaplan model (1996) 28 ± 29, 30 ± 31t Grabowski, J 4, 16 ± 17, 18 Graduate Management Admissions Test see GMAT Grischkowsky, N 25 Grobe, C 69 group scoring 129 ± 30, 145, 149, 151, 153 Haas, C 25 Hake, R 66 Hale, G 62, 63, 63t, 64, 174 Hales, L.W 73 Hamp-Lyons, L 19, 55, 58, 66, 67, 74, 75, 93, 108, 109, 112, 115, 119, 120, 124, 192, 198 ± 200, 202, 204, 206, 207, 208, 213, 217, 228, 240 Hartog, P.J 73 Hatch, E 135 Hayes, J.R 19, 22 Hayes-Flower model (1980) 23 ± 24 Hayes model (1996) 24 ± 29, 44 Hedgcock, J.S 105, 184 264 ASSESSING WRITING Herman, J.L 208, 209, 210, 211 ± 12, 213, 214, 220 Hinds, J 21 Hinkel, E 71 Hoetker, J 65, 68 Hoger, E.A 202 holistic scoring 65, 70, 71 versus analytic 72 ± 73, 120 ± 22, 220 college-level writing 222, 224f interpretation 114 portfolio assessment 222, 224f rating scales 109, 112 ± 14, 143 ± 45 Homburg, T.J 69 Horowitz, D 93, 95 Hughes, A 1, 45, 83, 105 Hughes, D.E 73 Huot, B 64, 69, 70, 242, 243t Hymes, D 29, 42 IELTS (International English Language Testing System) 140 content Academic Module 155 ± 56, 157f, 158f, 160 General Training Module 155, 156f, 160 distinctive features 160 purpose 155 scoring 157 ± 60, 161 test usefulness 161 ± 62 impact 53 ± 55, 121t classroom assessment 175 ± 76 portfolio assessment 204 ± TOEFL Writing Test 147 indirect testing 59, 239 ± 40 Intensive English Program (IEP), Georgia State University 188 inter-rater reliability 114, 135 classroom assessment 182 ± 83 portfolio assessment 208 ± timed impromptu writing 59 ± 60 interactiveness 53, 121t, 147 classroom assessment 175 ± 76, 186, 188 FCE 154 portfolio assessment 204 International English Language Testing System see IELTS Jacobs, H 115, 116f, 124 Jakobson, R 10 Janopolous, M 70 Jensen, L 187 Johns, A.M 20 Jones, R.J 69, 71 Kaplan, R.B 20 ± 21, 21, 34 ± 35, 37 Kean, D 37 Keech, C.L 71, 100 Keeling, B 73 Kenyon, D 60, 108 keyboarding skills 237 knowledge, telling versus transforming 31 ± 32 Kobayashi, H 71 Koretz, D 208 Krapels, A.R 35 Kroll, B 19, 35, 89, 91, 181, 182, 183t Lambert, W 37 Land, R.E 71 Language for Speci®c Purposes see LSP language knowledge, components of 42, 43t construct de®nition 41 ± 42 discourse 29, 30t functions 42, 43t grammar 42, 43t planning 43t sociolinguistic knowledge 29, 30 ± 31t, 42, 43t strategic competence 42 ± 44, 53 texts 42, 43t language use, and test performance construct de®nition 41 ± 42 topical knowledge 44 ± 45 emotional factors 44 ± 45, 46 personality factors 44, 45, 46 strategic competence 42 ± 44, 53 Lazaraton, A 135 learning, writing as tool for ± Leki, I 4, 20, 21, 188 ± 89 Lewkowicz, J 68, 95 Lloyd-Jones, R 110 LSP (Language for Speci®c Purposes) 67 topic knowledge 45 ± 46, 80 Lynch, B.K 77, 78, 83, 84, 84t, 85f Mabry, L 217 Markham, L.R 69, 105, 114 Matalene, C 21 Matthias, S.P 66 McGinnis, I 114 McNamara, T.F 42, 46 ± 47, 60, 61f, 108, 109 McNelly, M.E 71 Index mean score 135 Meara, P 106 memory 25, 26f, 28 Mendelsohn, D 71, 189 Messick, S 59, 61, 207 metacognitive strategies 42 ± 44, 53, 204, 233 Michigan English Language Assessment Battery (MELAB) 103, 104f Michigan Writing Assessment Scoring Guide 115, 118 ± 19f, 122 Miller, M.D 37 Minnesota Language Pro®ciency Assessments (MLPA) 166 Moss, P.A 241 ± 42 motivation and affect 25, 26f Murphy, S 71, 74, 75, 100, 102, 204 ± 6, 215 National Assessment of Educational Progress (NAEP), scoring guide 110, 111f Nesi, H 106 networked classrooms 188, 232 ± 34 Nold, E.W 69 Norris, J 77, 83 North, B 125 ± 26 O'Neil, H.F Jr 234, 235 online scoring 145 oral language, and written language cognitive processes 16, 17 ± 18 correctness 17 educational research view 15 language-use tasks 18 ± 19 linguistic view 15 reinforcing oral patterns 1, 11 ± 12 social and cultural context 16 ± 17 orthography, limitations of 16 Ostler, S 21 Page, E 234 Palmer, A.S 3, 18, 39, 40 ± 41, 42 ± 46, 47, 48 ± 56 77, 79, 80, 81, 82, 96, 103, 119, 120, 121t, 123, 125, 126, 134, 180, 200, 203, 204, 244 Palmquist, M 25 Par®tt, M 232 peer revision 188, 191 Peirce, B.N 37 Pennington, M 105 performance assessment 46 ± 48, 133 Perl, S 22 265 Peterson, N 234 pilot testing 181 ± 82 Polio, C 75 politics critical language testing 238 ± 39 educational testing 238 hermeneutical approach 241 ± 42 instructional versus programmatic perspective 240 principles for future 242, 243t stakeholders, con¯icts between 239 ± 40, 244 testing ®rms 239 ± 40 Pollitt, A 123 Popham, W.J 83 portfolio assessment 3, 59 characteristics 199 ± 200, 201f collection 199 ± 200, 201f contents 218f authenti®cation 216 background documentation 215 ± 16 control 213 number of pieces 214 process documentation 216 re¯ective essay 215 writing types 213 ± 14 context richness 199, 201f de®nition 198 delayed evaluation 199, 200, 201f development over time 199, 201f evaluation, external 201 feedback 200, 201f growth along speci®c parameters 199, 201f history 197 ± 98 logistical issues 227 ± 28 politics 240, 241 portfolio types 213 ± 14 purpose 211 ± 12 range, demonstrating 199, 201f re¯ection 199, 200, 201f scoring analytic 222, 224, 225 ± 27f criteria 217, 219 high school seniors 221 ± 22, 222 ± 23f holistic 222, 224f reporting 220 ± 21 selection 199, 200, 201f student-centered control 199, 201f, 213 test usefulness 200 ± 211 versus timed impromptu essays 52, 197 washback 55 266 ASSESSING WRITING Powers, D.E 72, 101, 105 practicality 48, 55 ± 56, 121t BEST 166 classroom assessment 175, 176 portfolio assessment 209 ± 10 scoring 56, 138 ± 39 test design stage 81 ± 82 TOEFL Writing Test 147 pre-testing versus pilot testing 89 prompts 89 ± 90, 134, 181 ± 82, 183t scoring levels 123 primary trait scoring 109, 110 ± 12, 132 ± 33 process-oriented writing 191 ± 92, 193 ± 94f, 196, 202, 232 prompts attributes 84t, 85f, 87f choice 147 ± 48 discipline-speci®c versus general 93 pre-testing 89 ± 90, 134, 181 ± 82, 183t test-taker interpretation of 74 ± 75 TOEFL Writing Test 142, 143f, 147 ± 48 psychometric methods 146, 178, 238, 239 ± 40, 241 Purves, A.C 3, 5, 60, 62, 63, 63t, 66, 101 ± Quellmalz, E.S 65 Raimes, A 20, 22 raters attributes 71 ± 72 behavior 137 online scoring 145 scoring rubric 125, 127 specialized writing 93 text attributes 70 ± 71 training 120, 130 ± 31 rating scales analytic 114 ± 20 versus holistic 72 ± 73, 120 ± 22 BEST 163, 165f CoWA 167, 169f FCE 152f holistic 109, 112 ± 14, 143 ± 45 IELTS 157 ± 58, 159f, 160t multi-trait scales 109 primary trait scoring 109, 110 ± 12, 132 ± 33 rating context 73 ± 74 scale descriptors 158 empirical approach 125 ± 26 levels-of-mastery approach 125, 126 ± 27 speci®c versus generalizable 109 ± 10, 109t TOEFL Writing Test 143 ± 45 weighting 124 ± 25 reading text, as basis for writing 95 ± 96 Reed, R 115 Reid, J 66, 89, 100, 181, 182, 183t, 184 reliability 121t, 170 BEST 166 classroom assessment 175, 176 portfolio assessment 208 ± scoring 50, 73, 120, 128 ± 30, 128 ± 30, 135 ± 36, 139 task design 90 ± 91 TOEFL Writing Test 147 reproducing information ± 9t, 10, 11 response attributes, specifying 84, 84t, 85f, 87f revision 27 Rinnert, C 71 Robinson, J.L 71 Ruth, L 71, 74, 75, 102 Saari, H Sacks, H 18 Santos, T 71, 189 scale descriptors 125 ± 27, 158 Scardamalia, M Scharton, M 239, 240 Schneider, G 125 ± 26 Schumann, J.H 37 scoring computer scoring 56, 234 ± 36, 242 handwritten versus keyed essays 105 ± incomplete responses 133 ± 34 independence 131 memorized scripts 133 off-task scripts 132 ± 33 practicality 56, 138 ± 39 reliability 50, 73, 120, 128 ± 30, 135 ± 36, 139 scoring rubric, designing functions, clarifying 122 language aspects 122 ± 23 reporting methods 124 ± 25, 220 ± 21 scale descriptors 125 ± 27 scoring levels 123 ± 24 total scores, calculating 127 ± 28 validity 60, 61f Index see also classroom assessment, scoring; portfolio assessment, scoring; raters; rating scales second-language writing issues speci®c to cultural and social factors 36 ± 37 motivation 36 ± 37 pro®ciency 35 ± 36 role of acquisition level learner groups ± writing purposes ± writing types 10 ± 12 Shen, F 37 Shermis, M.J 236 Shohamy, E 54, 71, 238 ± 39 Silva, T 35 ± 36, 202 Simon, H 22 skills, categorization of 14 ± 15 Sloan, C 114 Smith, W.L 68, 95 social aspects academic writing skills 19 ± 20, 178 classroom assessment 177 ± 78 hermeneutical approach 242, 243t networked classrooms 233 ± 34 social goals, promotion of 244 technology 233 Sommer, N 22 Spaan, M 65 Spack, R.F 17, 20 Spalding, E 207, 221 speaking, and writing 3, 14 ± 19 Sperling, M 17, 18, 19 standard deviation 135 standardization 17 Stewart, M 69 Stock, P.L 71 strategic competence, and language knowledge 42 ± 44, 53, 79 ± 80 Sullivan, F.J 69 survival, writing for 5, 6, 6t, 11, 11t, 12 Swain, M 29 Swales, J 17, 20, 28 Sweedler-Brown, C.O 71 task design 3, 106 ± clarity 90 dictionaries and reference materials 106 genre 96, 98 ± 101 instructions 103, 104f interest 91 reliability 90 ± 91 stimulus material 94 ± 96 subject matter 91 ± 94 task, choice of 103 ± time allotment 101 ± transcription mode 104 ± validity 90 task variables 60 ± 61, 69 comparability of results 62 construct-irrelevant variance 61 ± 62 content coverage 61 dimensions of variance 61 ± 62 cognitive demands 63, 63t, 65 discourse modes 65 ± 67 exposition, pattern of 62, 63t, 65 genre 62, 63t, 65 prompts 62 ± 64, 68 rhetorical task 62, 63t, 64, 65 scoring procedures 64, 66, 67 stimulus material 62, 63t, 67 ± 68 subject matter 62, 63t, 67 transcription mode 64 performance 61 ± 62 Taylor, C 105, 158, 237 teachers, stake in assessment process 239 ± 40 technology 230, 243 assessment 233 ± 34 computer access 237 computer scoring 234 ± 37 hypertext 232 networked classrooms 232 ± 33 social aspects 233 speech-like writing 231 ± 32 see also word processing Tedick, D 64, 67, 70, 93 TEEP (Test in English for Educational Purposes) 115, 117f test design see task design; test development test development administration stage 78, 86, 89 ± 90 classroom assessment 180 ± 81 design stage 77 ± 82 construct de®nitions 79 ± 80 design statement 82 mandate 78 test usefulness 81 ± 82 operationalization stage 89 test speci®cations 78, 82 ± 86, 106, 181 267 268 ASSESSING WRITING Test in English for Educational Purposes see TEEP Test of English as a Foreign Language see TOEFL Writing Test Test of Written English see TWE test purpose 40 ± 41 test speci®cations 78, 82 ± 86, 106, 181 test-taker variables 74 ± 75 test usefulness 3, 48 ± 49, 244 analytic versus holistic scoring 120 ± 22 authenticity 51 ± 52 classroom assessment 175 ± 76, 185 construct validity 49 ± 51 impact 53 ± 55 interactiveness 53 practicality 81 ± 82 reliability 49 test design 81 ± 82 test variables 69 ± 70 testing ®rms 239 ± 40 text types, communicative functions ± 12 think-aloud protocols 22, 50 time allotment 101 ± TOEFL Writing Test 112, 113f, 140, 141 authenticity 146 ± 47 construct 146 distinctive features 145 ± 46 practicality 147 purpose 141 ± 42 scoring 113f, 123, 143 ± 45 task ful®llment, relevance of 132 ± 33 technology gap 237 test usefulness authenticity 146 ± 47 construct 146, 147 impact 147 interactiveness 147 reliability 147 Tokar, E 73 topic knowledge construct de®nition 44 ± 45, 80 interactiveness 53 LSP 45 ± 46, 80 task design 80, 92 ± 93 testing academic writing 52 transcription mode 25, 104 ± 6, 142, 147, 148, 162 Truscott, J 184 TWE (Test of Written English) 89, 90, 142, 145 see also TOEFL Writing Test UCLES (University of Cambridge Local Examinations Certi®cate) 140, 148, 155 VaÈhaÈpaÈssi, A 4, ± 9, 10 ValdeÂz, G 167 validity hermeneutical approach 242 portfolio assessments 240, 241 scoring 59 ± 60, 61f task design 90 see also construct validity Vaughan, C 70 vocabulary 16 Wall, D 55 washback 54 ± 55, 137 ± 37, 147 ± 48, 177, 206 Weaver, F 74 Weigle, S.C 66, 71, 75, 187 Weir, C 73, 115, 119, 120 White, E.M 3, 59, 74, 90, 112, 114, 129, 130, 136, 180, 239, 240 Whitley, C 71 Willard-Traub, M 222, 225 ± 27f Wolcott, W 92, 201, 210, 222, 224f word processing 162 authenticity 147, 148 effect on writing process 25 versus handwriting 104 ± writer-responsible versus readerresponsible languages 21 ± 22 writing process, models of Bereiter and Scardamalia (1987) 29 ± 35, 44 Grabe and Kaplan (1996) 28 ± 29, 30 ± 31t Hayes (1996) 24 ± 29, 44 Hayes-Flower (1980) 23 ± 24 Yorkey, R 21 Young, R 25 Zamel, V 22 [...]... test writing ability? A common-sense answer to this question is that ``the best way to test people's writing ability is to get them to write'' (Hughes, 1989: 75) If we agree with this 1 2 ASSESSING WRITING statement, it follows that a test of writing involves at least two basic components: one or more writing tasks, or instructions that tell test takers what to write, and a means of evaluating the writing. .. assessment that go beyond the traditional timed impromptu writing test Chapter 8 discusses classroom evaluation of writing, looking at options for responding to and evaluating student writing at various stages of the writing process, from pre -writing through to a polished, ®nal text Chapter 9 discusses portfolio assessment, or the assessment of writing ability by collecting and evaluating a number of... future directions in second-language writing assessment Writing in ®rst- and second-language contexts Before we can discuss how to test writing, we must start by attempting to de®ne what we mean by writing ability As we will see, however, this is not a simple task, since, as researchers in both ®rstand second-language writing have pointed out, the uses to which writing is put by different people in... classroom to the real-world writing needs of these two groups, it is easy to imagine that the ®rst group ± immigrants in an L2 environment ± may have some use for informational (referential) writing ± for example, ®lling in forms, writing a narrative report of a workplace accident, or writing instructions One might also imagine some use for connative (persuasive) writing; for example, writing a letter of application... regard is the fact that, as discussed in Chapter 1, writing is highly valued in educational settings, and the standardization of writing means that accuracy in writing is frequently more important than accuracy in speaking The importance of correctness in writing as opposed to speaking is particularly relevant for writing in academic contexts, where writing is frequently seen as a key to entry into the... this chapter discuss these aspects of writing in more detail Writing as a social and cultural phenomenon Social aspects of writing The physical act of writing is sometimes thought of as mainly the result of cognitive effort on the part of an individual writer Indeed, the traditional approach to writing assessment has been to focus primarily on the cognitive aspects of writing, and these aspects will be... implications for the scoring of writing tasks and will be discussed again in Chapter 4 To summarize, writing is both a social and a cultural activity, in that acts of writing cannot be looked at in isolation but must be seen in their social and cultural contexts The implication for the testing of writing is that writing ability cannot be validly abstracted from the contexts in which writing takes place To some... Materials 1995, and IELTS Handbook 2002, UCLES Extracts from Contextualized Writing Assessment (CARLA 2001) Designing and assessing effective classroom writing assignments for NES & ESL students, reprinted from Journal of second language writing, Vol 4 no 1 pp 17±41, J Reid and B Kroll, copyright 1995, with permission from Elsevier Science Assessing the portfolio by L HampLyons and W Condon (2000), reprinted... of writing that are likely to be relevant for second-language writers Perhaps the best way to begin to appreciate the complexities in L2 writing is to contrast it with L1 writing As VaÈhaÈpaÈssi (1982), Leki (1992) and others have pointed out, ®rst language writing is inextricably linked to formal education While virtually all children are able to speak their native language when they begin school, writing. .. the testing of writing, both in terms of designing appropriate writing tasks and in terms of evaluating writing Classi®cation of written text types One important implication of the variety of background, experience, and needs of second-language writers is that the types of writing produced by these different groups vary considerably as well To continue our discussion of what is meant by writing ability,