Đang tải... (xem toàn văn)
Microsoft Word 03 Chap 40 Chapter 3 Evaluation Introduction The first section of this chapter discusses the broad aims of evaluation in education, and the need for a systematic approach to materials e[.]
Chapter Evaluation Introduction The first section of this chapter discusses the broad aims of evaluation in education, and the need for a systematic approach to materials evaluation Important terms like microevaluation and macro-evaluation are also explained in this section The second section stresses the role of criterion-referenced approaches to evaluation The closing section includes the salient features of the CBSE’s Interact in English syllabus in India, and explains the criteria used in the evaluation of the Interact in English syllabus Brown (1989: 231) defines evaluation as “the systematic collection and analysis of all relevant information necessary to promote the improvement of a curriculum, and assess its effectiveness and efficiency, as well as the participants’ attitudes within the context” From Brown’s definition cited above, it is evident that information and systematic analysis are primary for evaluation The term ‘evaluation’ was originally associated with testing and assessment This notion is reflected in Murphy’s (1985: 10) complaint "the necessity for evaluation is not understood and recognised" However, this tradition has changed due to the strong interest of applied linguists like Rea-Dickens and Germaine (1992) and Weir and Roberts (1994) regarding the goals, roles and methods of evaluation in language teaching This development is partly due to applied linguists’ recent interest in education theory and the need to conduct large-scale evaluations for external agencies like the British Council and the United States agencies (USAID) that fund programmes There have been plenty of studies or “organised educational activities” to use the words of Weir and Roberts (1994: 1) However, most of the evaluations are funded to meet a particular goal Beretta (1992) and Weir and Roberts (1994) argue that most of the evaluations are macro-evaluations, except for Slimani’s (1992) study on classroom interaction, which is not an examination of a programme, but an examination of a classroom technique 40 Alderson and Beretta (1992) have included a considerable number of case studies and macro-evaluation projects like Coleman’s (1992) Key English Language Teaching Project (KELT) in Indonesia and Beretta’s (1992) evaluation of the Communicative Language Teaching Project (CLTP) or Prabhu’s Bangalore-Madras experiment between 1979 and 1984 in India Macro-evaluation Macro-evaluation aims to answer one of the questions listed below To what extent was the programme/project effective and efficient in meeting its goals? In what ways can the programme/project be improved? (Ellis, 1998: 218) The first question relates to Weir and Roberts’ (1994) ‘accountability evaluation’, and the second to ‘development evaluation’ The evaluator needs information on administrative matters (financial implications, duration of the programme and other logistical factors), and curriculum matters (materials, teachers and learners) to undertake a macro-evaluation of any programme Hence it can be concluded that a macro-evaluation is meant either for accountability or for the development of an existing programme Although macro-evaluations provide useful information, they tend to ignore the teachers’ perspective (which any ideal evaluation should include) If the teachers are involved in any macro-evaluation, the evaluation will be a result of a series of micro-evaluations (conducted for each lesson) during the term According to Ellis (1998), that any teachermediated evaluation is likely to be a micro-evaluation, and not a macro-evaluation 41 Micro-evaluation Ellis defines micro-evaluation in this manner: A micro-evaluation is characterised by a narrow focus on certain aspects of the curriculum All the curricular and administrative aspects can be subjected to a micro-evaluation In the case of materials one can find out the efficiency of a task or an activity in a lesson unit of a course book to fulfil the objectives of the book and the curriculum One way of understanding evaluation is to accord the teacher perspective in the evaluation, which can be achieved by a microevaluation (Ellis, 1998: 219) Detailed studies on methods of evaluation, dimensions of evaluation, types of evaluation and purposes of evaluation can be found in Brown (1989), Alderson and Beretta (1992), Weir and Roberts (1994) and Ellis (1998) The ensuing sections focus on materials development and materials evaluation in language teaching Materials evaluation: A systematic approach Most people apply the term "language teaching materials" to language coursebooks According to Mariani: The relationship between an EFL teacher and his/her course book is often not a very easy and straightforward one: seldom is he/she enthusiastic over it; sometimes a love-and-hate relationship is established; and, more often, the teacher seems to have to come to terms with the book, in a sort of compromise through which a temporary armistice has been reached (1980: 27) Although this study is restricted to coursebook evaluation, the term ‘materials’ include audio/video tapes, CD-roms, newspaper stories, grammar activities, cartoons or anything that facilitates language learning Criteria for selecting materials A coursebook cannot be selected without a grasp of the needs, and the limitations posed by the learning environment like infrastructure, length of the course, class strength, teachers' abilities and examination constraints Therefore, it is difficult to find a global coursebook that will account for the factors listed above But teachers in most of the 42 institutions are unable to produce materials for their professional contexts due to time factors and government regulations Hence, teachers are obliged to adapt published coursebooks for their classrooms Mariani (1980) outlines the following criteria for selecting a coursebook: Define your local teaching situation Define your programme Evaluate your course book Devise suitable adaptations Teachers' role in materials production The selection of textbooks has been the responsibility of administrators and academics who may not be conversant with the classroom needs For instance, public (private and residential) schools in India were the testing sites for textbooks during the colonial period before importing them to England Scholars like Viswanathan (see p.2.) and Howatt (1984) have worked extensively on this issue Currently, teachers are given a prominent role in materials design and evaluation in language teaching The Central Board of Secondary Education (CBSE) in India trained a group of teachers in England with the British Council support to produce interactive/communicative materials English teachers in Malaysia, Indonesia and the Philippines are involved in materials production as well Materials evaluation Rea-Dickens and Germaine (1992) believe that teaching materials should be defined in the following manner before they are evaluated: What can materials mean to teachers? a) Does the teacher confine herself/himself solely to the textbook or does the teacher refer to other sources like a teacher’s guide, video, listening tapes etc.? b) Does the teacher distinguish between traditional materials meant for teaching/learning purposes and authentic materials? 43 c) Does the teacher use materials created by teachers and learners? The relation between materials and the social context: a) What roles are the materials likely to play? b) What goals and objectives are the materials likely to accomplish? How are the materials to be used? a) Are the materials the only resource for teaching? b) Are they one of several other resources? According to Hutchinson and Waters (1987), materials can be evaluated based on the beliefs of the learning context He divides this process in the following manner: • Stage one involves the definition of the criteria used to evaluate materials • Stage two is based on a subjective analysis to determine the nature and principles of the teaching context where the materials are used • Stage three involves the actual evaluation • Stage four involves the matching of the subjective and the objective analyses to conclude the evaluation Printed materials according to Hutchinson and Waters (1987: 19) are mostly built on theories of ‘language learning’ and ‘language use’ as much as the language teaching context, which operates on some premise Therefore, it is clear that the whole process of evaluation in this model is a matching exercise in four stages Candlin and Breen (1979) divide materials evaluation into two stages or phases The first phase focuses on the usefulness of materials The second phase focuses on selection and use of materials in ways that are sensitive to the language classroom Phase one The three major concerns to evaluate the usefulness of materials are listed below: What learners need to know should be matched with what learners will be able to with the materials 44 How language is best learnt should be matched with the kinds of learning actually offered by the materials The above questions open up the weak points of the materials with regard to teachers’ freedom to adapt the materials Moreover, this analysis can reveal whether the materials force the teacher to take inappropriate roles and isolate classroom activities in the materials The conditions to determine the usefulness of materials before evaluation are based on: a) The contribution of the teacher to classroom work b) The resources provided by the learners c) The contributions of the classroom Phase two Breen and Candlin (1987) propose the following questions to evaluate the usefulness of materials that are sensitive to language learning How the materials synchronise with the learners’ perception of language needs? Can learners perceive for themselves whether the materials can meet their language needs? Do the materials relate to learners’ values, attitudes and feelings? Do the materials capture learners’ attention? Are the materials divided in a principled manner? Are they continuous? Can the learners impose their own sequencing and division on the materials according to their needs? Are the materials flexible enough to accommodate teachers’ and learners’ preferences? Breen and Candlin (1987) propose that materials need to account for learners’ views and perceptions This proposition may not be possible in diverse learning contexts where the learners come from different cultures and backgrounds The models of Rea-Dickens and Germaine and Candlin and Breen place a high premium on the learners' role in materials production 45 Although the socio-cultural factors in India are likely to prevent learners' participation in materials production, the Interact in English series has been largely teacher-driven, unlike earlier materials, which were produced by university lecturers and curriculum specialists in India Criterion-referenced approach to materials evaluation According to Cunningsworth (1984), even ELT professionals have not always evaluated materials in a systematic manner in the past A random evaluation is usually based on superficial aspects like layout, print, writer's/publisher's credibility or the cost implications But Tomlinson (1999a) advocates a criterion-referenced approach instead of an impressionistic and random evaluation Types of evaluation Tomlinson (1999a: 10) recommends the following types of evaluation: Pre-use evaluation: The material is evaluated before it is taught, and it is the most common type of evaluation But even a meticulous pre-use evaluation can be unreliable, as there may not be adequate information about the impact of the material in the classroom Whilst-use evaluation: This is done while using the material in the classroom but it is seldom done in a methodical way If the whilst-evaluation is done with planning and focus it can generate useful data Post-use evaluation: According to Tomlinson (1999a: 11), it is difficult for most teachers to conduct a systematic post-use evaluation owing to time constraints and lack of expertise The post-use evaluation in this study will discuss the preferences of the former pupils of the Interact in English syllabus The post-use questionnaire in this study will focus on ‘process-based learning’, ‘classroom interaction’ and ‘product-based learning’ The post-use evaluation will not investigate learners’ language abilities after completing the Interact in English syllabus in the CBSE schools of Madras and Nasik 46 Tomlinson’s criterion-referenced model includes universal criteria apart from contentspecific, gender-specific and age-specific criteria to match the course objectives and the contents of the material This model is useful for evaluating other types of materials like CDRoms and audio/visual materials The criterion-referenced model of Tomlinson (1999a) is appropriate for evaluating the CBSE’s Interact in English syllabus prescribed for Classes IX and X (14–15 year olds) Tomlinson's interest in affective materials and visualisation are additional attractions of this model Purposes of materials evaluation According to Tomlinson (1999a), the objectives, the types and the instruments for the evaluation determine the reasons for evaluating materials For instance, if the evaluation aims to improve materials then the evaluator will investigate learners’ problems through classroom observation sessions Therefore, the instruments for evaluation depend on the objectives of the evaluation Tomlinson (1999a: 11) lists the following reasons for evaluating materials: To select a textbook for a course To select materials to supplement a coursebook To select materials from different sources in an eclectic manner As a basis for adaptation of materials in order to make them more suitable for a particular course As a basis for improving materials (trialling or piloting materials) In order to edit materials produced by others In order to review proposed materials for a publisher In order to review published materials for a journal In order to help teachers or trainee teachers develop their understanding of methodology and/or materials writing 10 In order to recommend a coursebook for an institution or a ministry of education 11 As part of a research experiment 47 According to Tomlinson (1999a: 11), each reason has an objective Therefore, one cannot apply a model for materials evaluation as it is determined by factors like the reasons for evaluation, the objectives of evaluation and the context of evaluation Principles of evaluation Generally, coursebooks are used for a semester or an academic year, and it is quite unlikely for the evaluator to make a systematic analysis of all the units in a coursebook Usually, a few units are evaluated in a systematic manner with a set of criteria A systematic evaluation is based on principled procedures with a set of criteria depending on the nature of evaluation and the type of material (coursebook, audio/video tapes etc) Although an ideal evaluation is based on principled criteria, the labels given below are arbitrary Universal criteria Content-specific criteria Context-specific criteria Gender-specific criteria Age-specific criteria Universal criteria: A set of self-explanatory criteria, which are applicable to all types of language learning materials irrespective of the teaching methodology Examples: a) Are the instructions clear? b) Does the material promote critical thinking? Content-specific criteria: These relate to the contents or the topics in the coursebook Thus, a set of criteria meant to evaluate a business English coursebook cannot be used to evaluate a language-through-literature coursebook Therefore, evaluators must devise criteria according to the contents of the materials Context-specific criteria: The cultural values, ideas, beliefs and attitudes of the learning environment are evaluated Therefore, a set of context related criteria developed for 48 school level course books in India would not be entirely applicable for textbooks published for schools in Nigeria Gender-specific criteria: These relate to issues in the treatment and portrayal of men and women in language coursebooks Age-specific criteria: These relate to the age of the learners Therefore, what applies to children may not be emotionally and intellectually challenging to teenagers or young adults Classification of criteria The classification of criteria is not uniform among evaluators, but still consistent within an evaluation The classification of criteria helps the evaluator to group generalisations of a similar nature, and move on to specific issues within that category This study has the following types of criteria: i Physical aspects ii Functional aspects i) Physical aspects: In this section, importance is given to the presentational aspects of the coursebook like layout, design, print quality etc ii) Functional aspects: Here the focus is on learning objectives, whether importance is given to all the four skills, are the activities analytical or experiential According to Tomlinson (1999a: 11), "the most realistic way of developing criteria is to brainstorm what principles I think the materials should ideally follow" The principles are written as statements about what the materials should actually do, and then converted into questions as a tool for evaluation The questions are classified into several categories The questions are framed with an objective of gathering data about the coursebook Thus questions, which fail to gather information in the list, will not have any role in the evaluation The questions, which aim to elicit Yes/No answers, begin with Does/Do, Is/Are 49 Points to develop criteria Distinguish between analytical questions and evaluative questions Analytical questions have a direct Yes/No answer Evaluative questions usually evoke a subjective response with adjectives Good questions are measurable For example, "Are the materials interesting"? Can be rephrased as, "Are the reading topics likely to interest (male or female – genderspecific, Singaporean/Indian teenagers? – context-specific) Each question should aim to evaluate only one aspect of the material rather than too many aspects The use of "and" is avoided in the questions Application of criteria The criteria can be used in a multiplicity of ways In this research, I wish to answer each question with a numerical value on a three-point scale The ensuing section will stress the need for evaluating materials in India Materials evaluation in India Materials evaluation is an under-explored field in India due to the lack of new initiatives in materials production However, the CBSE has changed this trend with the launch of the Interact in English syllabus for Classes IX and X Since teachers treat literature as a product rather than a process in the ESL classrooms materials have become static and repetitive Although there is very little evidence of any systematic evaluation of materials in the Indian context, specialists appointed by the CBSE have evaluated the Interact in English syllabus “to observe…teaching, acquire familiarity with current/teaching learning practices, talk with teachers and students etc The visits were not representative of CBSE schools in general, but were weighted towards the upper end of the range” (Tickoo, 2001: 118) 50 School materials English coursebooks are usually published by the department of education The textbooks published for Central and State schools are low priced in order to help the economically weaker sections of the country However, coursebooks for the Matriculation and Anglo-Indian schools are expensive and usually published by private agencies in the country English language textbooks are rarely changed by the boards owing to several constraints related to publishing, printing and training of teachers English language teaching materials in India are synonymous with the coursebook, but the Interact in English Syllabus includes a listening tape apart from the printed materials However, the students are not expected to buy the listening tape, as they have access to tape recorders in their schools The authorities have probably refrained from prescribing the listening tapes fearing that students from resource poor families may not have access to tape recorders According to Tickoo (2001: 118), “a majority of the schools find the package [Interact in English] expensive and not worth all the cost and effort” Materials production A vast majority of ELT materials focus on literature, grammar and vocabulary in India The popular tradition of using literature in compulsory English courses at the tertiary level is being replaced with functional English courses in some universities, but literature continues to be popular at the tertiary level in the regions of this study This evaluation is based on the compulsory English course meant for Classes IX and X in the CBSE system University lecturers prepare tasks for literary texts in coursebooks The professionals involved in the teaching of new materials are rarely invited to participate in materials production: Teachers and school principals not have much say; they are outweighed, both in size and power, by the ‘higher’ level members of committees, that is, university dons and senior officials of the education system (neither is actively involved in the day-to-day processes of teaching-learning at school) It is also found that these ‘senior’ members have a coloured view of that teachers’ or students’ views in matters of education are worth (Mathur, 1995: 302) 51 Most of the students in India belong to diverse groups in terms of language proficiency, cultural background, values and interests Trialling results show that more than 70 per cent of the students found the Interact in English materials interesting According to Bhatt (1995: 330), “trialling of Class IX materials was done in two stages 150 schools were covered in the first trialling phase and 304 schools in the second Class X materials were trialled once” Beretta’s (1992) evaluation of Prabhu’s Bangalore-Madras Project shows teachers’ difficulties in employing innovative methods in the Indian ESL classrooms even after inservice training workshops Narang observes that grammar teaching in most of the contexts is far removed from reality with its artificially created sentences Narang (1985: 38) states that “linguistic theories have changed the role of grammar from teaching structures in isolation to teaching structures in contexts” Furthermore, “a communicative grammar… gives priority to the use of linguistic structures, i.e, the functions that linguistic structures perform rather than the structures themselves It is thus a departure from formal grammar to functional grammar” (ibid) Although Narang’s comments on grammar teaching reflect some general tendencies, Bhatt highlights the objectives of teaching grammar in the Interact in English syllabus He states that “the grammar workbook was also brought in as the validity of a strong grammar tradition could not be overlooked Grammar teaching, however, was to be contextualized and students trained in inductive reasoning” (1995: 329-330) The Main Coursebook provides opportunities for free writing (composition) in the form of essays on a wide range of topics, which benefit proficient students According to Miller (1980: 3), teaching, “literature without composition is like teaching dance without music” However, composition activities are common in English medium schools but quite uncommon in the rural schools in India 52 Landmarks in materials development in India Language teaching in India has gone through several phases from ‘grammar translation’ to the ‘communicative approach’ The ambitious Madras English Language Teaching (MELT) in 1952, which is referred to by D.A Smith (1968) as the ‘Madras snowball’, proposed to train 27,000 teachers in the structural/oral method West describes the unfavourable conditions that led to the failure of the Madras campaign in the following comment: By ‘unfavourable circumstances’ we mean a class consisting of over thirty pupils (more usually forty or even fifty), congested on benches (not sitting in individual or dual desks), accommodated on an unsuitably shaped room, illgraded with a teacher, who perhaps does not speak English very well or very fluently, working in a hot climate (1960: 1-2) Prabhu’s Bangalore-Madras experiment was based on a procedural syllabus, which encouraged learning by doing things This project, which was conducted between 1979 and 1984, has drawn considerable attention among ELT specialists in the world The findings of Beretta’s (1992) evaluation of Prabhu’s syllabus state that the approach achieved its goals among the trainee teachers of the project where the instruction was on ‘meaning’ rather than ‘form’ But most of the regular teachers in the project resorted to ‘form focused’ instruction rather than ‘meaning focused instruction’ Studies on the Interact in English syllabus show that it has been evaluated only for the purposes of accountability and innovation by the CBSE According to Breen (1989: 193), “evaluation should be a part of teaching and learning work in the language class, not merely for the sake of developing more sensitive tasks but as a reflexive means for revealing individual processes” The Interact in English teachers with the support of the heads of many CBSE schools have reported that the method of teaching has remained unchanged in terms of teacher-student relationship, testing techniques and the physical conditions of the language classroom 53 According to Srivatsava (1985), during the days of the ‘grammar translation method’ “literature formed a part of the curriculum” to teach language The ‘audiolingual method’ according to Srivastava (1985: 73), “got promoted as it was ‘quite’ effective in producing a practical speaking knowledge of the language in the shortest possible time” Reading and writing have always played supreme roles in language learning compared to listening and speaking Hence the notion of comprehension preceding production seems to be true in the Indian context Although students listen intensively during their lessons, there is hardly any scope for extensive listening due to the lack of classroom interaction Generally, students from educated families in the metropolitan areas have copious opportunities for extensive listening through media, entertainment and socialisation Dimri makes the following statement regarding the role of literature in the ‘communicative methodology’: With the domination of the communicative approach …which aims at equipping the with the student with the knowledge of FL to such a level that he can use it effectively in real life situations he takes up the issue whether the study of literature can form part of the curriculum under the changed methodology (1985: 29) The Interact in English package is considered to be appropriate for L2 students (aspirants see p 6) in the CBSE schools compared to the students (elites see p 6) in the private schools Conclusion This chapter has stressed the importance of a systematic approach to evaluation The final section has highlighted the nature of materials evaluation and the role of teachers in the production of the Interact in English mateials It must be mentioned that materials evaluation plays a significant role in language teaching, as an appropriate coursebook contributes a great deal to the success of any course However, careful evaluation becomes a key issue in the present day context where a wide 54 variety of materials are available Therefore, a systematic approach to materials evaluation will help teachers, curriculum planners, publishers and researchers The next chapter will provide an overview of the Indian education system followed by a discussion on the instruments, research sites and the subjects of this evaluation 55 ... teachermediated evaluation is likely to be a micro-evaluation, and not a macro-evaluation 41 Micro-evaluation Ellis defines micro-evaluation in this manner: A micro-evaluation is characterised by a narrow... given below are arbitrary Universal criteria Content-specific criteria Context-specific criteria Gender-specific criteria Age-specific criteria Universal criteria: A set of self-explanatory criteria,... in English syllabus The post-use questionnaire in this study will focus on ‘process-based learning’, ‘classroom interaction’ and ‘product-based learning’ The post-use evaluation will not investigate