Đang tải... (xem toàn văn)
Although student generated questions have been studied to link assessment and learning for decades, little empirical research has been available to support student generated question activities. This study investigated the effects of webbased student generated questions (hereafter eSGQ) on EFL university students’ increasing class content knowledge and business correspondence skills. For the study, 46 EFL university students in a business correspondence skills course were asked to design midterm prep tests based on six chapters of the course textbook. In small groups, each student generated at least three items from the six chapters. The students made approximately 170 test items, and the instructor reviewed all the items and used them to form 18 webbased assessments with a total of 160 test items. Due to practical constraints the setting provided, the study employed a single group design with pre and post test analysis. Survey data were also examined. The results illustrated that the students’ posttest scores were improved compared to the pretest scores, and the students found creating test questions useful for improving content knowledge and business correspondence skills.
http://journal.kamall.or.kr/wp-content/uploads/2018/6/O_21_2_03.pdf http://www.kamall.or.kr Multimedia-Assisted Language Learning 21(2) 57-76 An Investigation of the Effects of Web-based Student-generated Questions to Improve Learning* Kyung-Mi O (Dongduk Women’s University) O, Kyung-Mi (2018) An investigation of the effects of web-based student-generated questions to improve learning Multimedia-Assisted Language Learning, 21(2), 57-76 Although student generated questions have been studied to link assessment and learning for decades, little empirical research has been available to support student generated question activities This study investigated the effects of web-based student generated questions (hereafter e-SGQ) on EFL university students’ increasing class content knowledge and business correspondence skills For the study, 46 EFL university students in a business correspondence skills course were asked to design mid-term prep tests based on six chapters of the course textbook In small groups, each student generated at least three items from the six chapters The students made approximately 170 test items, and the instructor reviewed all the items and used them to form 18 web-based assessments with a total of 160 test items Due to practical constraints the setting provided, the study employed a single group design with pre- and post- test analysis Survey data were also examined The results illustrated that the students’ post-test scores were improved compared to the pre-test scores, and the students found creating test questions useful for improving content knowledge and business correspondence skills Key words student generated questions, alternative assessment, online assessment, student-centered assessment, student-designed test, student-made test doi: 10 15702/mall.2018.21.2.57 I INTRODUCTION Over the past few decades, the significance of connecting assessment and learning has been widely accepted (e.g., Black & William, 1998; Fuchs & Fuchs, 1986), and it has been considered * This work was supported by the research fund of Dongduk Women’s University by some to be impossible to work in one area without taking into consideration of the other (Kafrani & Afshari, 2017) In this interrelationships between assessment and learning, the role of students has increased due to the raised awareness of individual students’ learning processes and their own construction of knowledge (Black, Harrison, Lee, Marshall, & William, 2003) In addition, the idea of having students understand what they think is relevant, important, and meaningful in the learning materials, and then create questions by themselves serving the assessment purposes, has been addressed by a number of contemporary educational theories and assessment approaches (e.g., constructivism, self-regulation, self-assessment, self-determination theories and multiple assessments) (Yu & Sun, 2015) In this respect, student question-generation (hereafter SQG) is one of such attempts However, there has been lack of research studies investigating the usefulness of the SQG in the students’ understanding of their own content materials and learning processes In a systematic review of research studies highlighting the role of students in assessment, Dinsmore and Wilson (2017) indicated that there has been a dearth of evidence regarding the benefits of student involvement in assessment practices, even though they did not reject the notion that ‘‘participation in assessment can be beneficial in terms of self-regulation’’ (p 164) Under such circumstances, the current study was motivated to examine whether the process of SQG could help students to better understand their class content and improve their language skills, specifically business correspondence skills In addition, the study has been designed to find out if the technology tools (i.e., an e-Learning software and a cloud platform) involving SQG can facilitate the process and provide the student/test-generators with online learning and assessing environment serving the intended educational goals To meet the ends, this study was designed to answer the following three research questions: 1) Does e-SGQ have any influence on the EFL students’ content knowledge and business correspondence skills? 2) How the students perceive the experience of SQG? 3) How the students perceive taking e-SGQ as an assessment? II LITERATURE REVIEW As an instructional technique to facilitate students’ learning outcomes, SQG has been investigated since the 1980s (Cohen, 1983; King, Biggs, & Lipsky, 1984; King, 1992) According to King (1992), SQG has been described as “prompting students to generate specific 58 An Investigation of the Effects of Web-based Student-generated Questions to Improve Learning thought-provoking questions pertaining to the material to be learned” (p 111) Simply put, SQG means having students design test questions based on the learning content Through the process, the students are expected to identify the important information about the content materials, and to generate questions about the aspects they think are important (Van Blerkom, Van Blerkom, & Bertsch, 2006) The concept of SQG is based on elaborative interrogation (Pressley, McDaniel, Turnure, Wood, & Ahmad, 1987) and the generative learning theory (Wittrock, 1974; 1990) The elaboration techniques in the cognitive process include relating new information to familiar topics of relevance, focusing on distinctive differences among similar ideas, and constructing an overview in various forms (Hoffman, 1997; Reigeluth, 1992, 1999) In the generative learning theory, learners are expected to actively integrate new ideas into their memory by connecting new with old ideas to facilitate their educational experience Both the elaboration technique and the generative learning theory highlight the importance of learners’ activity of making new information meaningful by linking the new with previously learned information, and this has been accepted by many researchers However, learners rarely engage in such a meaningful cognitive process unless they are prompted to (Britton, Van Dusen, Glynn, & Hemphill, 1990) and rarely activate their relevant prior information without prompting (Pressley, Woloshyn, King, Wood, Martin, & Menke, 1992) It has not been difficult to find such unsuccessful cases in point Given this, researchers have paid attention to self-generated elaboration (Wittrock, 1990) Prompting self-generated elaborations have been reported to be more effective in learning than elaborations provided by external sources including a teacher and a textbook (Wittrock, 1990) According to the schema theory, since personal knowledge is already schematized, when new information is related to it, it is less demanding to process and recall (King, 1992) Thus, learners may find the activities of engaging in one’s own elaboration easier than processing external elaborations, and this may be influential on the recall of the elaboration (King, 1992) Therefore, prompting students to generate questions has been expected to be more useful than other types of elaboration activities to successfully recall processed information The potential benefits of SQG have been addressed in multiple-folds The addressed strengths include the aspects of increasing student autonomy (Bruner, 1990; King, 1992; Marbach-Ad & Sokolove, 2000) and motivation (Barak & Rafaeli, 2004; Chin & Brown, 2002; King, 1992; Lan & Lin, 2011; Wilson, 2004), reducing students’ test anxiety (Kafrani & Afshari, 2017), helping teachers diagnose their students’ weak areas (Kafrani & Afshari, 2017), allowing students to recognize important points (Nicol, 2010), internalizing specific test contents Kyung-Mi O 59 (Havnes, Smith, Dysthe, & Ludvigsen, 2012), facilitating students’ high-level thinking (King, 1992; Papinczak, Peterson, Babri, Ward, Kippers, & Wilkinson, 2012) and/or deep-learning (Hawe & Dixon, 2014; Roscoe & Chi, 2007), and enhancing the abilities in problem-solving activities (Brown & Walter, 2005) In sum, SQG has been believed to motivate learners to voluntarily involve in discussion and to provide them with appropriate starting points from their prior knowledge leading to increase their performance (Byun, Lee, & Cerreto, 2014) Although most of the reviewed studies reported its usefulness (e.g., King, 1992), not all studies investigating SQG have revealed beneficial results Byun, et al (2014) found that SQG was not useful on ill-structured problem solving Byun et al (2014) investigated the effects of questioning strategies in ill-structured problem solving involving three groups with 61 undergraduate students: i) student-generated questioning group (n=21), ii) revising student-generated questions using instructor-generated question prompts (n=21), and iii) question prompts provided by the instructor (n=19) The study found that the control group, the third group with instructor’s question prompts performed better than the other groups in overall performance in ill-structured problem solving The researchers suggested that the level of SQG might have been affected by passive participation in discussion, but also by a lack of prior knowledge Based upon the analyses, Byun et al (2014) implied that the instructor’s intervention and students’ active participation were important variables contributing to the success of implementing SQG Kafrani and Afshari (2017) gained similar results in an Iranian EFL context Kafrani and Afshari examined how the process of SQG would help students and their teachers in assessing the students’ learning outcomes For the study, the researchers employed test analysis and interview procedure after having 120 students design test questions based on their course book The results concluded that the students did not pay enough attention to certain important content, and their test-item formats lacked in variety, predominantly limited to multiple-choice tasks Through the study, Kafrani and Afshari provided some insightful implications indicating that the students’ final products should be used in any type of possible form (e.g., school website, classroom boards, or classroom quizzes) in order to give them a sense of achievement and ownership The studies of Byun et al (2014) and Kafrani and Afshari (2017) have suggested that the SQG can benefit from teacher intervention, and the reviewed test questions should be formed into a finished product Yu and Chen (2014) is a good case in point employing recent technologies to develop student-generated questions into an online test by introducing an online SQG activity for their study In the study, Yu and Chen aimed to investigate the influence of online drill-and-practice 60 An Investigation of the Effects of Web-based Student-generated Questions to Improve Learning activities using student-generated questions on the students’ academic performance and motivation For the experiment specifically, the researchers used three groups: i) online SQG and answering teacher-generated questions (treatment group 1), ii) online SQG and answering student-generated questions (treatment group 2), and iii) online SQG alone (comparison group) Through a five week treatment with 145 participants, the researchers found the students in treatment group with online SQG and teacher-generated questions performed significantly better on the academic assessment than the other two groups, and the students in treatment group with online SQG and student-generated questions failed to show any meaningful difference from the students in the comparison group with only online SQG Based on the findings, the researchers suggested that the online SQG should be coupled with teacher-generated questions for increasing students’ academic performance and motivation They added that the student-generated questions should not be used solely under time constraint Although Yu and Chen’s study is meaningful in that they adopted the current technology in the student question generation study; however, the hypothesis of their research design is not clear They employed online SQG for all three groups and added drill-and-practice tasks for the online SQG The researchers did not explicate clearly what they meant by SQG and why they included drill and practice tasks for the online SQG Through the unclear research design and treatments, the purpose of the research seemed to be unclear and lack interval validity Another important variable in the study, the amount of time, did not appear to have been controlled uniformly for the three groups This lack of control in the experiment procedure might have negatively influenced the results of the study Inspired by a number of student-generated questioning studies and available technology to support the activity, this study was designed to examine if e-SGQ could help students in increasing class content knowledge and business correspondence skills To reach the goals, the study adopted a simple research design employing a single group of students taking pre-and post-tests measuring content knowledge and business correspondence skills Moreover, the study was aimed at understanding the learners’ perceptions toward SQG and e-SGQ Kyung-Mi O 61 III METHOD Single-group Study For the study, a single-group research design was chosen to test the efficiency of the SQG activity because it was not practically possible or ethical to arbitrarily assign a control group (Harris, et al., 2006) Generally, the use of both a treatment group and a control group with the pre- and post-test design has been the most broadly adopted form of research design in much educational research However, with the advantage requiring researchers only a modest amount of resources and efforts (Nock, Michel, & Photos, 2007) to observe the effects of interventions in a relatively short treatment time (Nock et al., 2007), the single-group experiment design is another type of quasi-experimental research design which is also widely used in the field of social science and psychology, and medical science (Harris, et al., 2006) In the single group study, special attention needs to be given in interpreting the gathered data Because the design has no external validity, the intervention shown to be effective for the participants in this study may not have the same outcome with other populations Moreover, since it has only minimal internal validity, method triangulations, using multiple data collection methods as survey and/or interview procedure, may enlarge the internal validity of the study For this study, survey data were additionally used Participants For the study, 46 university students taking a Business Correspondence Skills class at a women’s university in Seoul were selected The students in the class were all females in mid-twenties In the class, the students were expected to enlarge their knowledge of the business world and develop the essential written communication skills needed in business contexts The 46 students in the class were chosen through convenience sampling by simply being selected from one of the researcher/instructor’s classes This sampling method is not ideal due to the risk of sampling error However, when it is the only possible option for the research study, the method is used with caution as observed in many educational and/or medical research Thus, the 46 students were all involved in both data collection procedures of pre- and post- test and survey 62 An Investigation of the Effects of Web-based Student-generated Questions to Improve Learning Instruments 1) Web‐based Student‐generated Questions (e‐SGQ) In order to create a student-generated assessment, the students were randomly put into 14 groups consisting of or students each Each individual in every group was asked to design at least three items from any chapters they chose from the six chapters covered in the class The students made approximately 170 test items with various test tasks including multiple-choice, multiple response, open-ended (type-in), matching, sequencing, and fill-in tasks To develop the test items into web-based tests, all the student-generated test items were reviewed by the researcher to examine if there were any typos, items with incorrect keys or distractors, overlapping items, and any potential problems Through the instructor’s reviewing process, ten items, mostly overlapping questions, were deleted For putting the tests on the web, an e-Learning software, iSpring QuizMaker was used This app was chosen because it bears resemblance with MS office packet, which has been seen to be user-friendly to the researcher The versatile functions of the app including many test task choices, multiple-trial option, autoscoring, and feedback were judged to be useful Through the process, 18 versions of web-based quizzes with a total of 160 test items were made The various test tasks that the student employed were transferred to the web-version with minimal revision Each test item was labeled with the generator’s name(s) so that the students could identify their own tests and feel ownership to the test items The e-SGQs (the actual student names in Figure 1~4 were removed for the presentation) were published on the iSpring Cloud and the Cloud address was posted on the university e-class community bulletin With the tests available on the Cloud, the students had easy access to their personal computers and smartphones anywhere, anytime [Figure 1] Title Image of Each e-SGQ Version by Section Kyung-Mi O 63 [Figure 2] Sample Image of Matching Task [Figure 3] Sample Image of Open-ended (Type-in) Task [Figure 4] Sample Image of Score Report 64 An Investigation of the Effects of Web-based Student-generated Questions to Improve Learning 2) Pre‐ and Post‐Test In order to measure the students’ class content knowledge and business correspondence skills, test items covering business correspondence skills as well as the class content in the commercial TOEIC preparation material were carefully selected Since the pre- and post-test procedures should not take up all the class time which is 75 minutes, a short modified version with 30 items was used as the pre-and post-test, and each test item was assigned with one point for scoring 3) Questionnaire A survey was formed to measure the students’ perceptions toward their experiences of SQG and taking the questions online The survey was formed with a point Likert-scale and an open-ended question each allowing the researcher to quantify the survey result and collect unasked responses It is mainly comprised of two sections: students’ experiences of 1) designing test items and 2) taking the e-SGQ For the first section investigating the students’ experiences of designing test items, four subsections are included: i) question generating experience, ii) perceived cooperation, iii) perceived outcome, and iv) satisfaction degree The questions in this section were basically adopted from Song (2015) where he employed for his student-generated question study Specifically, seven survey questions (#1~4, #8~10) were used from Song (2015, p 160) without modification, and three questions (#5~7) were adopted from Song (2015) with minor modification to fit into the current study Since a large number of the questions were directly or partly adopted from Song, which have already been tested in his study, the items could be considered rather proper indicators of the researcher’s areas of inquiry In the second section regarding the students’ experiences of taking the e-SGQ, inspired by Song (2015), eight questions with two subsections: i) perceived outcome (#11~13) and ii) satisfaction degree (#14~17) were newly added With the new items and the minor modifications with some questions, the instrument was pilot tested with ten individuals before administrating the instrument to the participants The Chronbach’s Alpha was 0.886, which is the scale of 0.80 and 0.90 implying very good reliability of the instrument (DeVellis, 1991) An 'alpha if item deleted' was also calculated for refining the survey items Three items including two from Song (2015) and a newly made one were judged to be not important items Kyung-Mi O 65 with 'alpha if item deleted' of 0.894, 0.893 and 0.892, respectively Thus, the three items were deleted All the survey questions were provided in English since the students’ English reading skills were advanced enough to read the items with ease The survey questions with the framework and specifications are summarized in Table [Table 1] Survey Questions with Framework and Specifications I Students’ experiences of SQG A Question generation experience Item # I learned how to make good questions I am able to make good questions B Perceived cooperation I had enough support to make good questions My teacher supported me enough to make good questions C Perceived outcome Making questions was helpful for me to understand the class contents Making questions improved my business correspondence comprehension skills Making questions improved my business correspondence production skills D Satisfaction degree I enjoyed making questions I was satisfied with making questions I would like to make questions for the final test 10 II Students’ experiences of taking e-SGQ E Perceived Outcome Solving e-SGQ was helpful for me to understand the class contents 11 Solving e-SGQ improved my business correspondence comprehension skills 12 Solving e-SGQ improved my business correspondence production skills 13 F Satisfaction degree I enjoyed solving e-SGQ in general 14 I enjoyed solving e-SGQ by myself 15 I am satisfied with solving e-SGQ 16 I would like to solve e-SGQ for the final test 17 III Deleted items I learned what a good question was 66 NA My group supported me to make good questions NA I enjoyed solving e-SGQ as a group NA An Investigation of the Effects of Web-based Student-generated Questions to Improve Learning Data Collection Procedures and Reliability Analyses At the onset of the study, the students took the pre-test and learned about SQG and various test tasks The students were also informed that they could use any type of test tasks they liked to use for creating their own questions After making student-generated questions for a couple of weeks through group and individual works, the students submitted the test items All the test items were reviewed, corrected, and formed by the researcher/instructor into 18 versions of e-SGQ and the students took the 18 versions of the tests with a total of 160 test items for three weeks repeatedly to prepare for their mid-term test They took them multiple times in groups in class and individually in class and/or at home either through computers or smartphones In the 6th week, the post-test and survey were administered to the participants For the statistical analysis of the quantitative data collected from the pre- and post-tests and the survey, SPSS 23 was used To ensure the reliability of the test instrument, the reliability coefficient was calculated from the post-test data The gained Cronbach’s coefficient alpha was 0.749 which was larger than the standard acceptability of Cronbach’s coefficient alpha of 0.7 suggesting that the test instrument was reliable enough to be used (Nunnally, 1978) The survey was administered to the students, and the Cronbach’s coefficient alpha value was 0.910 indicating an acceptable reliability coefficient The student comments gathered from the open-ended question were collected and analyzed Some portions of the answers were translated from Korean to English and presented in quotes with pseudonyms IV RESULTS The Effect of e-SGQ For the purpose of answering to first research question asking about the effect of e-SGQ on the class contents and students’ business correspondence skills, the quantitative analysis of the pre- and post-test data has rendered a positive outcome To investigate if the taking the e-SGQ was effective for increasing the students’ content knowledge and business correspondence skills, the students’ post-test results were compared with pre-test results Since the same participants’ performances were measured twice, resulting in pairs of observations, paired samples T-test, a widely used statistical procedure, was employed The use of the parametric paired samples T-test was seen to be valid as the two related dependent Kyung-Mi O 67 variables were normally distributed [Table 2] Paired Samples T-test Result of the Pre- and Post- Test (n=46) Test Mean (SD) t df p-value Pre- 23.22 (3.01) -3.789 45 000* Post- 24.78 (3.24) * p