A Task-Based Analysis of Undergraduate Assessment: A Tool for the EAP Practitioner

BRIEF REPORTS AND SUMMARIES TESOL Quarterly invites readers to submit short reports and updates on their work These summaries may address any areas of interest to Quarterly readers Edited by JOHN FLOWERDEW University of Leeds ALI SHEHADEH United Arab Emirates University A Task-Based Analysis of Undergraduate Assessment: A Tool for the EAP Practitioner KATIE DUNWORTH Curtin University of Technology Perth, Western Australia Ⅲ Within the field of English for academic purposes (EAP), one issue that attracts consensus in the literature is the importance of needs analysis As Benesch (2001) observes, “needs analysis offers detailed information about the linguistic and cognitive challenges students face in academic settings” (p 61) Although there is rather less unanimity on what those needs might be—see, for example, Pennycook’s (1997) critique of vulgar pragmatism—all approaches nevertheless accept certain fundamentals One fundamental is the need for EAP teachers to be aware of the discourse types that occur in the domains to which EAP students will progress However, it is not always a straightforward matter to determine what these discourse types might be Thus it is necessary for EAP tutors or curriculum designers to be familiar with the work that their students, who may be from a range of discipline areas, will be required to Certainly numerous studies have been conducted into specific, and usually written, academic genres, particularly at the graduate level, but there is a paucity of widely available information on the complete range of tasks that undergraduates are required to undertake This brief report summarises an empirical study that identified the range of tasks undertaken by neophyte undergraduate students and thereby provides EAP practitioners with information for their ongoing efforts to design programs TESOL QUARTERLY Vol 42, No 2, June 2008 315 THE STUDY The research took place on the main campus of an Australian university with approximately 35,000 students, more than 25% of whom are international The single site was selected because it was anticipated to generate richer data than a less focused approach and because it permitted local inspection of primary sources In other words, the study would use the material that students themselves encounter rather than filtered descriptions from respondents from a variety of institutions Within the study university, the tasks identified for analysis were compulsory, summatively assessed (i.e., allocated a mark that contributed to the final grade), and set in a student’s first semester of undergraduate study It is now widely acknowledged that summative assessment strongly influences and may even drive student learning (Birenbaum, 1997; Brown & Knight, 1994; Dochy & McDowell, 1997; Hanrahan & Isaacs, 2001; Maclellan, 2004; O’Donovan, Price, & Rust, 2004; Shepard, 2000; Struyven, Dochy, & Janssens, 2005; Swanson, Norman, & Linn, 1995; Williams, 2005) The choice of first-year tasks was predicated on the belief that the influence of a pretertiary EAP course is most likely to be felt at an early stage of a student’s academic career, prior to experiential familiarisation with the linguistic, academic, and social norms of the university In Australia, academic degrees consist of specified groups of study units within a certain discipline Students are provided with unit outlines, substantial documents that summarise the content of the unit of study and, among other things, describe the assessed tasks and assessment procedures These documents provided the major source of data for the study When interpretation of tasks was problematic, clarification or expansion was sought through semistructured interviews with unit coordinators, a process that involved 45 academic staff From a campus-wide total of 139 units identified, 112 unit outlines, or 80%, from 32 discipline areas were obtained This percentage was considered sufficiently high to yield reliable results, particularly because no disciplinary area or academic department was unrepresented in the obtained sample So that it would be possible to identify any interdisciplinary differences, the results were stratified according to the four academic divisions of the university at that time: business, engineering and science, health sciences, and humanities The number of compulsory units for each area varied considerably In business, all majors share a common first year, and so the division produced only five compulsory units, while engineering and science produced a total of 59 Nevertheless, the diversity of the assessed tasks the outlines generated was as great for the former as for the latter, providing further support to the argument that a sample higher than 80% of the total would not have generated markedly different results 316 TESOL QUARTERLY More than 360 separately assessed tasks were identified, each of which was then classified into a task type according to six criteria that had previously been determined Lack of space precludes a detailed description of how the criteria were selected, but they were devised following a process of integration of criteria used in a number of previous studies into university-level tasks, such as those of Bridgeman and Carlson (1983), Hale et al (1996), Horowitz (1986), and Moore and Morton (1999) Reference was also made to taxonomies of language proficiency, such as the Common European Framework (Council for Cultural Cooperation Education Committee, Modern Languages Division, Strasburg, 2001) and the model devised by Bachman (1990), so that the final classification system would be of particular use in an EAP context The six criteria were Nomenclature of the task as named in the unit outline, for the insight it provided into the academic’s intentions Quantity of output, as measured by the number of words Time allocated to task completion Predominant macro-skill required for completion, selected because many EAP courses are skills based Source of information for task fulfilment, selected in order to recognise the contribution of all macro-skills to the task Cognitive demands, evaluated through the use of Bloom’s (1956) still frequently cited Taxonomy and the work of Hale et al (1996) For example, tasks could be categorized as belonging to a domain of “knowledge of specifics,” which required the linguistic skills of describing, explaining, listing, or summarizing; or to a domain of “knowledge of universals and abstractions in a field” (Bloom, 1956, p 202), which required the functions of evaluating, extrapolating, and analysing abstractions From the process of applying these criteria to each task, 19 different categories of task emerged, as described in Table All assessed tasks, including those set in examinations or class tests were placed within the 19 categories In passing, it should be noted that tests and examinations featured prominently in three of the four divisions, in spite of the expanding body of literature that details and recommends alternative forms of assessment in higher education Tests and exams absorbed 50% of all marks in business, 51% in engineering and science, 31% in health sciences, and 12% in humanities The impact of their content in reference to the overall frequency and value of certain tasks is therefore extremely important BRIEF REPORTS AND SUMMARIES 317 TABLE Categories of Task Task type Task category Predominantly speaking and listening based Formal presentations Participation Predominantly writing and reading based Annotated bibliographies Article/book reviews or critiques, literature reviews Case study reports Extended essays, more than 1,200 words Journals, diaries, and learning logs Multiple-choice and true–false questions Reports on experiments, research, or field experience Short answers, including formulae, algorithms, definitions, and labelling or describing diagrams Short essays up to 1,200 words Summaries of texts, information synthesis Timed essays, usually part of a test Practical Computer tasks (emphasis on the technology) Designs, drawings, folios Laboratory work and workbook reports Library task (becoming familiar with the library and its resources) Nonlaboratory-based practicals Other (a few tasks which did not merit their own category but could not be incorporated into any other group) The tasks were then compared according to their relative frequency of occurrence in each division, and for their relative status in each division in terms of marks awarded Both measurements were necessary to gain an impression of the overall relevance of the task from the perspective of a broadly based EAP program For example, an individual task might be valued at, say, only 10% of the total marks for a unit but might occur in a large number of units Conversely, a task might attract a high percentage of marks in one unit but not appear in any other units within the division and might therefore be seen as an outlier in the overall context RESULTS Table shows how frequently, in percentage terms, a given task occurred within each division For example, the first listed task, participation, occurred in 100% of the compulsory first-year units offered within the division of business, 18% of those in the division of engineering and science, and so on The table shows that the most frequently occurring tasks, that is, those tasks which occurred in more than half the total units for each division, are participation, multiple-choice questions, short answers, and timed essays in business; short answers, multiple-choice questions, and 318 TESOL QUARTERLY TABLE Frequency of Occurrence of Tasks Within Academic Divisions (Percentages) Task Category Business Engineering and science Health sciences Humanities Annotated bibliographies Case studies Computer tasks Critiques, reviews Designs, drawings, folios Experiments/field trip reports Essays, extended (1,200+ words) Essays, short (up to 1,200 words) Essays, timed Formal presentations Journals, diaries, learning logs Laboratory/workbook reports Library task Nonlaboratory-based practicals Multiple-choice questions Short answers Summaries, synthesis Other Total task types 20 40 20 20 0 20 60 40 20 0 100 60 0 11 0 18 27 14 57 0 50 82 11 27 8 15 15 23 15 19 38 65 85 18 19 13 32 48 26 32 10 19 32 23 16 13 10 17 laboratory/workbook reports in engineering and science; short answers and multiple-choice questions in health sciences; and participation in humanities Table illustrates the mean percentage score (rounded to the nearest whole number) allocated to a particular task when averaged across all units obtained for each division For example, the first listed task, participation, was allocated an average of 7% of the total marks in a unit within the division of business, 3% of the total marks in engineering and science, and so on Thus, whereas participation occurred as a task in all units within the division of business, it was not accorded many marks and was therefore not highly valued as an assessed activity Those activities which did attract the most marks in three of the four divisions were short answers and multiple-choice questions, with 47% or more of the total marks being allocated to these activities In humanities, which includes the departments of art, design, and architecture as well as social sciences, the highest marks were allocated to designs, drawings, and folios and reports on experiments/field trips Thus short answers and multiple-choice questions featured strongly in both frequency of occurrence and value in terms of marks awarded in three of the four divisions In terms of the results by skill, tasks which involved some kind of oral output appeared in all divisions, although they were not allocated a high status with regard to marks, ranging from 4% of the total in engineering and science to 15% of the total in humanities In all cases, participation BRIEF REPORTS AND SUMMARIES 319 TABLE Status of Tasks Within Academic Divisions (Percentages) Task Category Business Engineering and science Health sciences Humanities Annotated bibliographies Case studies Computer tasks Critiques, reviews Designs, drawings, folios Essays, extended (1,200+ words) Essays, short (up to 1,200 words) Essays, timed Experiments/field trip reports Formal presentations Journals, diaries, learning logs Laboratory/workbook reports Library task Nonlaboratory-based practicals Multiple-choice questions Participation Short answers Summaries, synthesis Other 10 0 17 4 0 23 25 0 0 5 10 0 11 53 0 4 10 17 31 1 21 13 12 1 was considerably more important with regard to both frequency and value than formal presentations, although the nature of participation and the criteria by which it might be assessed remained somewhat opaque, even following interviews with unit coordinators Perhaps unsurprisingly, assessment of written work varied among discipline areas Timed essays in examinations or tests were accorded the highest status in business, constituting 17% of the total marks awarded, but nontimed essays (short essays and extended essays) only attracted marks in double figures within humanities Written tasks which were experientially focused in some way, through conducting an experiment or undertaking a field trip, appeared in all divisions except business, while problem-solving tasks in the form of case studies appeared in all divisions except engineering and science Reflective writing, in the form of journals, diaries, or learning logs, appeared in almost one third of units in humanities, but featured less strongly elsewhere In summary, extended writing of some kind represented nearly half of all marks awarded in humanities, around 40% of marks in business and health sciences, and under a quarter of total marks in engineering and science Tasks overtly based on reading, such as critiques, reviews; annotated bibliographies; and summaries, synthesis, were comparatively rare across the board Indeed, reading of any kind did not appear to play the major role that might have been expected Subsequent analysis of all unit outlines revealed that required reading matter often consisted of a set of lecturer320 TESOL QUARTERLY compiled readings, lecture notes, or single chapters from text books Few units listed more than two or three set reading texts DISCUSSION What have these findings to with EAP courses? Whatever the paradigm in which an EAP course is framed, it is important for teachers to be aware of the practical demands that will be made of students after graduation Given the large body of research that demonstrates the primacy to undergraduate students of assessed tasks, it can be argued that the EAP syllabus should prioritise those tasks which course participants are most likely to encounter, whether it be from a pragmatic perspective, to emulate them, or from a critical perspective, to learn how to contest them EAP professionals, who are themselves frequently graduates from humanities, education, or social science disciplines, are not automatically aware of the range of disciplinary distinctions that exist, so it is important that professionals have access to such data With this knowledge it is possible not only to incorporate the most featured and relevant tasks into the syllabus, but also to explore with students the many issues that might arise with regard to assessment, such as misunderstanding the task itself, differing interpretations of assessment rubrics, lack of familiarity with assessment criteria, potential conflicts about the nature and function of the task, and uncertainty over the rights and responsibilities of students For example, research indicates how open to interpretation are the abstract terms used in the rubric of assessed tasks (Chanock, 2000) and in assessment criteria (O’Donovan, Price, & Rust, 2004) Furthermore, familiarity with the assessed activities undertaken at an institution not only indicates to university students “the kind of intellectual work which is valued” (Maclellan, 2004, p 20), but also provides EAP teachers, who are not always used within a tertiary environment themselves, with a window into the organisation’s underlying value system and epistemological position This does not mean, of course, that those forms of assessment which students are most likely to encounter and which will most impact on their grades should be uncritically foregrounded in EAP programs Instead, assessment must take into account both the activities which precede the actual task and the way in which students are taught at the tertiary level For example, a narrow focus solely on future assessed output would, as the results of this study indicate, heavily weight content toward the development of certain forms of writing at the expense of other skills Yet as we know, the construction of the knowledge used in producing the content for the various writing genres is facilitated BRIEF REPORTS AND SUMMARIES 321 through a wide range of activities, only some of which involve writing For example, the literature has long made reference to the difficulties experienced by international students participating in tertiary-level discussion groups and seminars or in integrating with the local student population (e.g., Chalmers & Volet, 1997; Mullins et al., 1995) Space does not permit a discussion of the reasons for this, which are numerous and complex However, it could certainly be argued that unless students are also encouraged to develop the aural/oral skills that will facilitate their acquisition and construction of disciplinary content knowledge, the advantages conferred by the development of any assessment- related writing skills are greatly diminished Nevertheless, if EAP courses are to dedicate at least part of their syllabi to the modelling and emulation or critique of those tasks that students are later to encounter (and it is certainly the case in Australia that this is what many programs purport to do), then it is essential that EAP practitioners have an active and ongoing awareness of the assessment environment for which they are preparing their students, whatever their discipline Although this research is a single-site study carried out in Australia, it contributes to the body of knowledge and provides a comparison for other broad-based investigations EAP is a wide field that incorporates a range of learning contexts, as Casanave (2004) points out Within that overarching paradigm, decisions on local course content need to be made, and there is considerable value in having available to EAP practitioners the kind of data that will assist with making sensitive, informed, and reflective choices THE AUTHOR Katie Dunworth is an associate professor in the Department of Languages and Intercultural Education at Curtin University of Technology, Perth, Western Australia She has more than 25 years of experience in TESOL as a teacher, administrator, and researcher Her particular research interests are educational assessment and the internationalization of higher education REFERENCES Bachman, L F (1990) Fundamental considerations in language testing Oxford: Oxford University Press Benesch, S (2001) Critical English for academic purposes: Theory, politics, and practice Mahwah, NJ: Lawrence Erlbaum Birenbaum, M (1997) Assessment preferences and their relationship to learning strategies and orientations Higher Education, 33, 71–84 Bloom, B S (Ed.) (1956) Taxonomy of educational objectives London: Longman 322 TESOL QUARTERLY Bridgeman, B., & Carlson, S (1983) Survey of academic writing tasks required of graduate and undergraduate foreign students Princeton, NJ: Educational Testing Service Brown, S., & Knight, P (1994) Assessing learners in higher education London: Kogan Page Casanave, C P (2004) Controversies in second language writing: Dilemmas and decisions in research and instruction Ann Arbor, MI: University of Michigan Press Chalmers, D., & Volet, S (1997) Common misconceptions about students from South-East Asia studying in Australia Higher Education Research & Development, 16, 87–98 Chanock, K (2000) Comments on essays: Do students understand what tutors write? Teaching in Higher Education, 5(1), 95–105 Council for Cultural Co-operation Education Committee, Modern Languages Division, Strasburg (2001) Common European framework of reference for language: Learning, teaching, assessment Cambridge: Cambridge University Press Dochy, F., & McDowell, L (1997) Assessment as a tool for learning Studies in Educational Evaluation, 23, 279–298 Hale, G., Taylor, C., Bridgeman, B., Carson, J., Kroll, B., & Kantor, R (1996) A study of writing tasks assigned in academic degree programs Princeton, NJ: Educational Testing Service Hanrahan, S., & Isaacs, G (2001) Assessing self- and peer-assessment: The students’ views Higher Education Research & Development, 20, 53–70 Horowitz, D (1986) What professors actually require: Academic tasks for the ESL classroom TESOL Quarterly, 20, 445–482 Maclellan, E (2004) Authenticity in assessment tasks: A heuristic exploration of academics’ perceptions Higher Education Research & Development, 23, 19–33 Moore, T., & Morton, J (1999) Authenticity in the IELTS academic module writing test: A comparative study of Task items and university assignments In R Tulloh (Ed.), IELTS research reports 1999 (Vol 2; pp 64–106) Canberra: IELTS Australia Mullins, G., Quintrell, N., & Hancock, L (1995) The experiences of international and local students at three Australian universities Higher Education Research & Development, 14, 201–223 O’Donovan, B., Price, M., & Rust, C (2004) Know what I mean? Enhancing student understanding of assessment standards and criteria Teaching in Higher Education, 9, 325–335 Pennycook, A (1997) Vulgar pragmatism, critical pragmatism, and EAP English for Specific Purposes, 16, 253–269 Shepard, L (2000) The role of assessment in a learning culture Educational Researcher, 29(7), 4–14 Struyven, K., Dochy, F., & Janssens, S (2005) Students’ perceptions about evaluation and assessment in higher education: A review Assessment & Evaluation in Higher Education, 30, 325–341 Swanson, D B., Norman, G R., & Linn, R L (1995) Performance-based assessment: Lessons from the health professions Educational Researcher, 24(5), 5–11, 35 Williams, K (2005) Lecturer and first year student (mis)understandings of assessment task verbs: “Mind the gap.” Teaching in Higher Education, 10, 157–173 BRIEF REPORTS AND SUMMARIES 323