Dynamic Speaking Assessments TRƯỜNG ĐẠI HỌC MỞ BÀI GIẢNG GIÁO TRÌNH

BRIEF REPORTS AND SUMMARIES TESOL Quarterly invites readers to submit short reports and updates on their work These summaries may address any areas of interest to TQ readers Edited by CHRIS DAVISON University of New South Wales CONSTANT LEUNG King’s College, London Dynamic Speaking Assessments KENT HILL Seigakuin University Saitama, Japan MEHRAN SABET Seigakuin University Saitama, Japan Ⅲ This article describes an attempt to adopt dynamic assessment methods in classroom speaking assessments As an alternative or supplement to standardized testing, interest in classroom assessment has been on the increase (Leung, 2005; Leung & Mohan, 2004) The teacher as assessor creates discourses of assessment that are of value to enhancing our understanding of classroom teacher assessment (Rea-Dickins, 2004), and dynamic assessment (DA) offers another valuable means of advancing this understanding DA is a development-oriented process of collaborative engagement that reveals the underlying causes of learners’ performance problems and helps learners overcome them (Poehner, 2007, 2008) DA distinguishes itself from nondynamic assessment (NDA) in a number of ways First, NDA taps more into a developed state (i.e., intelligence), whereas DA taps more into a developing process (i.e., cognition) Second, in NDA there is no feedback from examiner to test-taker regarding quality of performance In DA, feedback is given, either explicitly or implicitly Third, the quality of the examiner–examinee relationship is different In NDA, the examiner attempts to be as neutral and as uninvolved as possible to avoid any error in measurement In DA, the mediator and learner relationship TESOL QUARTERLY Vol 43, No 3, September 2009 537 have a two-way interactive relationship (Sternberg & Grigorenko, 2002) Finally and perhaps most important, DA does not separate instruction from assessment BACKGROUND The study reported in this article focused on four particular applications of dynamic speaking assessment (DSA) The first, mediated assistance, involves interaction between an assistor and a learner to reveal problems in spoken performance Mediated assistance (MA) is the way in which a mediating agent, guided by her intentions, culture, and emotional investment, selects and organizes the world of stimuli for the learner (Sternberg & Grigorenko, 2002) MA has several components relevant to this study: (a) selection of stimuli that the assistor believes will profit the learner, (b) sequencing of stimuli so this learner will be able to learn from them in an optimal way, and (c) anticipation of the stimuli by the learner (Feuerstein, Rand, Falik, & Feuerstein, 2003) The mediation by the assessor is designed to assist learners in solving problems (Lantolf & Thorne, 2006) In many ways, such mediation functions like corrective feedback in second language acquisition (Ohta, 2001) In particular, the strategic use of recasts to reformulate learners’ erroneous utterances (Loewen & Philp, 2006) may help in the assessment of learners’ developmental levels The second DSA approach is to discover learners’ ability to transfer what they have already internalized to novel problems This transfer-oflearning (TOL) ability to overcome performance problems also represents the primary means of assessing genuine development (Brown & Ferrara, 1985) In addition, because of the symmetrical nature of DSA, some TOL from higher to lower level pairs may be observable If the learner is able to solve a problem of more graduated difficulty by extending what he or she has learned, it would indicate that the learner has developed a deeper understanding of the concepts at work in the original assessment (Lantolf & Thorne, 2006) Vygotsky’s (1986) zone of proximal development (ZPD) is the third DSA application However, the proximal zone between more and less competent individuals is not the only construct under observation The same developmental and problem-solving ability between individuals can also become collective and interconnected in groups (Poehner, this issue) A group ZPD (GZPD) breaks from the cognitively biased preoccupation with the autonomous individual and shifts the focus to the sociocultural nature of the ZPD Comparisons among individuals in the GZPD cease to make sense because the dynamics of their cooperative activity becomes more relevant (Poehner, 2007) Mediation within the group not only 538 TESOL QUARTERLY moves the GZPD forward; it also promotes the development of individuals within it The final DSA approach is collaborative engagement Collaborative engagement (CE) involves diagnosing problem areas during DSA, especially when there has not been indication of TOL or the internalization of any cognitive tools (Poehner, 2007) During periods of instruction, the mediator engages collaboratively with learners, providing them with the intervention needed to overcome problem areas, and then learners are once again assessed as to the extent to which the CE was successful and the necessary cognitive were tools internalized THE STUDY The method of speaking assessment at our university prior to this study was through peer-to-peer interaction, and assessors identified three possible areas for DSA application: use of three methodologies (i.e., questions, prompts, and role-plays), evaluation by checking off evaluation criteria, and lower level learners negatively affecting the performance of higher level partners A sociocognitive perspective to DSA (Hill, 2006a, 2006b) suggested the following solutions: Develop TOL between roleplays, provide MA in the form of questions and prompts, and pair learners from higher to lower level and allow them to observe the previous pair’s performance This year-long study involved four speaking assessments of a firstyear speaking and listening class at a university in Saitama, Japan (two in the spring semester and two in the fall) There were 18 students (12 female and male) Each assessment involved two role-plays The first assessment was a control NDA The second assessment was the first DSA In this assessment, the second role-play was graduated in TOL difficulty (i.e., more difficult than the first), the students were paired from most to least competent, and MA was administered in the form of recasts The third assessment (i.e., the second DSA) again used graduated TOL role-plays and MA in the form of recasts but added comprehension checks, and students were paired and sequenced differently (i.e., the GZPD was reorganized with the top 1–9 learners paired with the lower 10–18, respectively) The final DSA again used graduated TOL role-plays; however, MA was concerned with negotiation of intended meaning and sought evidence of internalization of TOL from a learner who had not previously shown any such internalization This learner therefore participated in ongoing CE between the second and third DSA All assessments were recorded and transcribed BRIEF REPORTS 539 RESULTS The Effectiveness of TOL The purpose of using role-plays with TOL was to see if transfer might occur from the first role-play to the more difficult second one Learners could prepare for the first role-play but not the second The initial DSA’s first role-play was authentic in that learners used their own answers (discussing the semester), and the second role-play was designed to be graduated in difficulty (two friends running into each other after a long time) However, some learners (predominantly male) lacked engagement in the second role-play perhaps because of its fictional nature (van Lier, 1989) but also perhaps from a lack of task-intrinsic motivation (Haywood & Lidz, 2007) Task-intrinsic motivation refers to motivation that is inherent in tasks (i.e., inherently rewarding) The second DSA was much more dynamic because of a need to alter its mediation, and this difference resulted in more inherent task-intrinsic motivation In the role-play, pairs first gave directions using a simple road map (e.g., to the store) Then the TOL task required using a subway map of Tokyo to tell how to get from one station to the next Tokyo’s subway system is quite complex, and it turned out that many learners were not familiar with it Thus, completion of the task required some dynamic mediation on the part of the assistor so that pairs could complete it First, the departing and arriving stations were highlighted on the map Then the routes were highlighted, and at that point pairs could successfully transfer their knowledge from the first road map to Tokyo’s subway system It was this need to graduate the task that resulted in greater TOL and consequently began to reveal learners’ developmental levels and problem-solving abilities In the final DSA, pairs engaged in mini-debates In the first minidebate, they knew their topics and could prepare either pro or arguments In the second, although they had been told all of the different possible debate topics, they did not know which topic they would debate Though they were unaware of which topic they would debate, all learners showed internalization of the cognitive tools to co-construct debates, and each was able to extend from the first topic to any of the possible topics in the second role-play (see Table for the sequence of DSA role-plays) The Effectiveness of MA As should be expected with DSA, the forms of MA developed from one assessment to the next The first and most predominant form of MA was 540 TESOL QUARTERLY TABLE The Sequence of Dynamic Speaking Assessment (DSA) Role-Plays Nondynamic assessment DSA DSA DSA Fictional Nongraduated No transfer of learning (TOL) No problem solving Fictional Graduated Some TOL No problem solving Authentic Graduated More TOL Some problem solving Authentic Graduated Effective TOL Effective problem solving recasts The following is a typical example of a teacher-produced MA recast: S: What did you before this class? MA: What were you doing S: What were you doing before this class? The target of the DSA was the past tense, and the past progressive emerged as the form that best represented the particular or potential level of development for the learners in the GZPD The majority of these recasts occurred in the first role-play of the first DSA, and they had a significantly high occurrence of uptake (81%) In contrast, Ohta (2001) found that only about half of the recasts in a teacher-fronted lesson had uptake In addition, the majority of recasts occurred with less competent pairs The following excerpt from the first DSA is an example of correct usage after a recast and uptake First the MA recasts three times for thrice, and the learner shows uptake through repetition of three times Then the MA does another recast for the other learner, changing from past tense to present perfect aspect Note how the recast becomes more implicit, specifically with the past participial gone This learner also shows uptake Finally, the previous learner demonstrates correct usage by using three times instead of thrice A: Have you ever went to English stu English conversation school? S: Yes A: How many times have you went to English conversation school? S: I went to there thrice a week MA: Three times S: Three times (pause) MA: Have you ever GONE to conversation school A: Have you ever— MA: –gone— A: gone to conversation school S: Yes S: Th-Three times I went to English conversation school three times BRIEF REPORTS 541 This learner, who had the present perfect recast in the excerpt earlier, exhibited no uptake on her first recast With her second recast, she exhibited uptake but she repeated an incorrect part of the recast In this last example, however, she has cumulatively learned how to spot the target of the recast and produce correct uptake She has also developed reciprocity toward the benefits of MA (Poehner, 2008) In the second DSA, the topic was likes and dislikes, and the comparative emerged as the concurrent MA form: A: Harry Potter I like Harry Potter bet—I like Harry Potter it’s better than Lord of the Rings MA: I like Harry Potter better than A: I like Harry Potter better than Lord of the Ring but Lord of the Ring is as famous as Harry Potter This is the same learner from the previous recast example She has now learned not only to show uptake of a recast but also to co-construct on it In the three DSAs, MA developed from recasts in the first to comprehension checks in the second, to, finally, negotiations of intended meaning Negotiation of intended meaning began as suggesting possible alternatives to learners’ use of L2 communication strategies such as the L1 or substituting other words (Yoshida-Morise, 1998): S: I have three point the first point that brand name goods is nan to iu ii daro uh brand name goods can buy at wi wi little money MA: Not expensive? S: Not not expensive and the second Along with co-constructing language, negotiation of intended meaning develops a more conceptual understanding of the language within the learner (Poehner, 2008) The cumulative effect of recasts tended to make them more implicit, especially from the first to second role-play Once an intersubjective focus was established with recasts, newer forms of MA then developed which became more explicit (see Figure 1) FIGURE The Sequence of MA 542 TESOL QUARTERLY The Effectiveness of the GZPD Gal’perin (cited in Poehner, 2008) suggests that learners need to orient themselves to the task by devising successful methods to complete it Observation of the previous pair appears to be a successful method to achieve this aim However, pairing learners from most to least competent does not necessarily initiate the most effective GZPD, because the most competent are at a ceiling level and the lowest cannot make successful gains In the second DSA, learners were paired by matching the higher selfregulatory learners with the lower self-regulatory learners In this DSA, the less self-regulatory learners’ scores improved significantly; however, the scores of their more competent partners dropped by almost exactly the same amount This decrease in level from convergence of performance is quite possibly another indication of the reality of the ZPD, an issue to be explored in future studies Possible indications of cumulative GZPD development from one DSA to the next included an approximate 10% increase in scores for the entire group as well as lesser variance between scores The shift from explicit to implicit MA between pairs may also indicate a GZPD That is, if MA was offered in a previous pair, the following pair may only require an implicit recast to make similar uptake This pairto-pair cumulative effect was similar to that found between pairs from one role-play to the next (see Table 2) The Effectiveness of CE Following the second DSA, there was only one learner who had not shown any internalization of TOL He had continually been successful in completing the first role-play but did not exhibit any TOL in the second It was decided to undertake CE with him during class instruction His position had also usually been near the end in the DSA; thus, his position was also moved to a higher level Through triadic CE with his partner and the teacher, this learner finally began to internalize the cognitive tools for TOL This interactive dialectic between assessment and instruction reveals the promise of DA: Attempting to show a learner how to achieve self-regulation may also increase his or her accurate usage of language TABLE Explicit/Implicit Relationship Between MA and GZPD Role-play to role-play explicit → implicit BRIEF REPORTS Pair to pair explicit → implicit Assessment to assessment explicit → implicit 543 Following CE, in his final DSA, this learner displayed the ability for TOL, reciprocity to all forms of MA, and performed symmetrically to others in the GZPD DISCUSSION AND CONCLUSION Our data suggest that TOL in the form of role-plays of graduated difficulty is a genuine means of assessing development of second language acquisition Pairing and sequencing learners according to their levels and allowing them to observe previous pairs (i.e., a GZPD) may interconnect their performance, improve DSA performance, and reduce variation in performance MA data suggest it has significant cumulative improvement not only in learners’ speaking performance but in their reciprocity (primarily in the form of recasts) However, MA should not be thought of as simply a repertoire of techniques The potential developmental level of the learner must always be kept in mind Through joint-attention interaction, the mediator engages in triadic interaction (i.e., establishing and developing intersubjectivity between oneself and learner pairs) By feeding information derived from MA back into the curriculum, MA can then help to create positive washback for both learners and assistors Allowing learners to observe the previous pair undertaking the assessment grants these learners the time to orient to the task and frees their cognitive attention to focus on performance In this sense, the GZPD highlights the fact that the ZPD is a sociocultural rather than simply cognitive phenomenon and that development rather than imitation takes precedence within it Indeed, if learners repeat previous errors, they should be assessed lower; however, if they can observe an error and correct it, then they should be assessed more positively In addition, it is possible to alter the GZPD sequence during the DSA If learners are not performing as expected, they could be asked to wait and observe other pairs or they could be paired differently Within pairs, it should also be decided if the higher learner would like a higher partner in order to develop further or would like a lower partner to assist in development Finally, each additional assessment should begin at the level of proximal development exhibited in the previous assessment, thereby allowing learning to dynamically lead development In sum, the exhibition of TOL, reciprocity to MA, and an awareness of CE may make good criteria for DSAs To summarize, DA contributes to optimizing the match between learners and tasks, to developing learners’ cognitive as well as second language skills, and to providing useful assessments of language development Future research into DA should investigate how approaches like MA can extend to NDAs such as oral proficiency interviews 544 TESOL QUARTERLY ACKNOWLEGMENTS The authors would like to thank Matt Poehner for his comments on an earlier draft THE AUTHOR Kent Hill is a lecturer at Seigakuin University in Saitama, Japan He is currently pursuing ways to take the static out of standardized testing Mehran Sabet is the director of the English program at Seigakuin University in Saitama, Japan He is currently looking into how to put the noise back into standardized testing REFERENCES Brown, A., & Ferrara, R (1985) Diagnosing zones of proximal development In J V Werstch (Ed.), Culture, communication and cognition: Vygotskian perspectives Cambridge: Cambridge University Press Feuerstein, R., Rand, Y., Falik, L., & Feuerstein, R S (2003) Dynamic assessment of cognitive modifiability Jerusalem: ICELP Haywood, H., & Lidz, C (2007) Dynamic assessment in practice: Clinical and educational applications Cambridge: Cambridge University Press Hill, K (2006a) A sociocognitive perspective: The best of both worlds TESOL Quarterly, 40, 819–825 Hill, K (2006b) Sociocognitive metaphorm Unpublished doctoral dissertation, University of Nottingham, England Lantolf, J., & Thorne, S (2006) Sociocultural theory and the genesis of second language development Oxford: Oxford University Press Leung, C (2005) Classroom teacher assessment of second language development: Construct as practice In E Hinkel (Ed.), Handbook of research in second language teaching and learning Mahwah, NJ: Erlbaum Leung, C., & Mohan, B (2004) Teacher formative assessment and talk in classroom contexts: Assessment as discourse and assessment of discourse Language Testing, 21, 335–359 Loewen, S., & Philp, J (2006) Recasts in the adult English L2 classroom: Characteristics, explicitness and effectiveness Modern Language Journal, 90, 536–556 Ohta, A (2001) Second language acquisition processes in the classroom Mahwah, NJ: Erlbaum Poehner, M (2007) Beyond the test: L2 dynamic assessment and the transcendence of mediated learning Modern Language Journal, 91, 323–340 Poehner, M (2008) Dynamic assessment: A Vygotskian approach to understanding and promoting L2 development Berlin: Springer Rea-Dickins, P (2004) Understanding teachers as agents of assessment Language Testing, 21, 249–258 Sternberg, R., & Grigorenko, E (2002) Dynamic testing Cambridge: Cambridge University Press van Lier, L (1989) Reeling, writhing, drawling, stretching, and fainting in coils: Oral proficiency interviews as conversation TESOL Quarterly, 23, 489–508 Vygotsky, L S (1986) Thought and language Cambridge: MIT Press Yoshida-Morise, Y (1998) The use of communication strategies in language proficiency interviews In R Young & A W He (Eds.), Talking and testing Amsterdam: John Benjamins BRIEF REPORTS 545