Comparing the Impact of Two Different Designs for Online Discussion 369 3 Results The online version of the volume will be available in LNCS Online. Members of institutes subscribing to the Lecture Notes in Computer Science series have access to all the pdfs of all the online publications. Non-subscribers can only read as far as the abstracts. If they try to go beyond this point, they are automatically asked, whether they would like to order the pdf, and are given instructions as to how to do so. Please note that, if your email address is given in your paper, it will also be in- cluded in the meta data of the online version. Table 1. A Summary of Online Discussion Participations by the Students _________________________________________________________________________ Fall 2007 Spring 2008 __________________________________ ____________________________________ Topics Accesses Messages Words Topics Accesses Messages Words per per message message __________________________________ ____________________________________ Lesson 1 About me 477 32 83 High stakes test 714 32 200 Assessment Benefit of topic 488 36 70 assess. know. 537 28 130 Lesson 2 Reliability 405 34 126 Test reliability 481 24 111 True score 449 33 54 True score 523 17 215 Lesson 3 Validity Test example 346 31 113 validity 717 35 101 Test Test validation 363 30 116 validation 490 25 95 Lesson 4 Test bias 354 32 107 Biased assess. 344 28 140 Test Test accommod. 240 28 99 accommod. 415 35 117 Lesson 5 Cognitive Cognitive Level 306 28 76 level 522 30 146 Affective/ Affective/ Psychomotor 278 26 93 psychomotor 564 30 121 Lesson 6 Guideline Guideline for item for selected construction 201 26 113 response item 314 25 125 Type of test Selected item 175 27 106 response item 314 26 106 ________________________________________________________________________ Average 340 30 96 Average 495 28 134 370 Y. Yao 3.1 Student Participation A summary of quantitative data regarding student participation in the online discussions is provided in Table 1. The data shows that the students in the online course visited each forum on an average of 340 times in Fall 2007, and 495 times in Spring 2008, an increase of 46% over the previous semester. There was an average of 30 messages per discussion topic provided by students in Fall 2007, and about the same number (an average of 28) of student messages in Spring 2008. The average number of words a student used in a posted message was 96 in the fall semester, and 134 in the spring, an increase of 40% over the previous semester. Table 2. A Summary of Online Discussion Participations by the Instructor __________________________________________________________________________ Fall 2007 Spring 2008 ___________________________________ ____________________________________ Topics Accesses Messages Words Topics Accesses Messages Words per per message message ___________________________________ ____________________________________ Lesson 1 About me 66 6 34 High stakes test 72 4 100 Assessment Benefit of topic 145 21 14 assess. know. 43 2 124 Lesson 2 Reliability 184 24 11 Test reliability 22 1 373 True score 196 24 10 True score 32 3 95 Lesson 3 Validity Test example 119 19 34 validity 20 1 145 Test Test validation 194 23 32 validation 16 1 424 Lesson 4 Test bias 96 18 28 Biased assess. 14 1 411 Test Test accommod. 85 20 16 accommod. 13 1 460 Lesson 5 Cognitive Cognitive Level 121 22 15 level 29 2 169 Affective/ Affective/ psychomotor 122 21 12 psychomotor 30 2 221 Lesson 6 Guideline Guideline for item for selected construction 73 21 12 response item 6 1 244 Type of test Selected item 66 21 19 response item 5 1 239 __________________________________________________________________________ Average 122 20 20 Average 25 2 250 Comparing the Impact of Two Different Designs for Online Discussion 371 3.2 Instructor Involvement The involvement of the instructor in terms of the number of visits to the discussion forums, the number of messages, and the number of words per message is summa- rized in Table 2. The instructor visited each forum on an average of 122 times in Fall 2007 and 25 times in Spring 2008. Given that there were 24 students in each semester, the instructor visited each student’s postings about five times in the Fall semester, and only once per student in the Spring semester. The average number of messages posted by the instructor for each discussion topic also dropped significantly, from 20 in the Fall to 2 in the spring. On the other hand, the average number of words used by the instructor per message increased drastically, from an average of 20 words per mes- sage in the Fall 2007 semester, to an average of 250 words per message in the Spring 2008 semester. The sharp increase in the words reflected the fact that the instructor used relatively brief comments to each student in the Fall semester and much detailed feedback at the end of the discussions in the Spring semester. The total number of words the instructor used for each discussion forum averaged 400 for the Fall semes- ter and 500 for the Spring semester. 3.3 Student Interactivity Aside from the quantitative data, the content of the discussion forums was examined to compare the potential impact of the discussion designs. One difference that was relatively easy to identify was the extent of interactivity between the students. Al- though there was some interactivity in the Fall semester, it was only discernible in 5 out of the 30 messages students posted in response to the topic of test validation. Overall, the postings in that semester’s discussion board read more like a dialogue between each individual student and the instructor, with each student responding to the message originally posted by the instructor, and the instructor following the re- sponse with a brief comment on the merit of the student’s response. In contrast, in the Spring semester, each student was required to respond to the previous post. It was evident from the postings that each student had read at least the previous post before providing any response in the forum. Many messages included acknowledgment of several postings that were provided earlier. 3.4 Opportunities for Questions Related to the difference of interactivity is the opportunity for students to raise mean- ingful questions. In the Spring section, there was sufficient evidence of students asking their fellow students to clarify their positions or challenge them to a new per- spective. For instance, in the discussion forum on the topic of test accommodation, a student mentioned a friend of hers who needed accommodation due to dyslexia. She observed that test accommodations were important, but should be only provided on a case by case basis. Another student followed up by drawing attention to potential issues with test accommodations, even when the provision of accommodations was justified in this case: 372 Y. Yao Did she say whether or not the reader’s voice intonation, eye contact, or body language ever helped her get the right answer, or maybe even mis- guided her to a wrong answer? I ask this because I would be concerned about an added advantage or disadvantage to the tester. In another post, a student questioned the benefit of accommodating students in testing over the long term: I have questions about whether or not students benefit in the long run by hav- ing exams read to them when they are poor readers… I question the wisdom in constantly reading exams to students. Students will learn that they have no need to try harder… There were five instances where such thought provoking questions were asked re- garding test accommodation in the Spring section. In contrast, no such question was found in a corresponding forum in the Fall section. Instead, most of the messages posted in the semester were attempts to answer the instructor’s original question without raising any further questions for either the instructor or the rest of the class. 3.5 Evidence of New Learning Not only was there a difference in the opportunity to raise thoughtful questions, there was also a difference in the opportunity for students to engage in meaningful inquires that would provide for them insights or new perspectives. Such opportunities were rare in the discussions in the Fall semester, where the conversations were mostly restricted to those between the instructor and an individual student. A typical response by the instructor to a student’s post on the topic of test validation was: “I think that would be a great idea to validate a test, by giving it to students in another grade!” Although the instructor’s comments were often worded in a constructive way, seldom did the conversation between the instructor and the student proceed much further. Occasionally, a conversation might occur among the students themselves in the Fall section. For instance, the following is a message pertaining to the topic of test validation: Since I am the only teacher for my area of instruction, it makes it difficult to have another teacher review the exam; however, your post got me thinking that there are a few other instructors in the local area, some with many years [of] experience and they could be a great resource in developing my pro- gram. Thanks. This message suggests that a previous post brought to the attention of this student the idea of using peers at another location to review her test. However, the idea was new to the student and needed to be tested out in the field. It is possible that some other students in the class had relevant experience and could provide valuable feed- back to this student. Unfortunately, this conversation was not followed up in the Fall semester. Comparing the Impact of Two Different Designs for Online Discussion 373 Such follow-ups, however, were common in the Spring section of the course. For instance, one student responded to a previous post by suggesting a new idea for pre- testing when validating a test: I believe the pre-test is a good idea, but why does it need to be a test? An- other good way to test if the students know the material could be by a game, such as a relay game, or different things like that. This material you cover in the game would be related to what the test is going to be like and then if the students have [a] question they are more than welcome to ask… Adding in some fun and action will allow the students to be more involved and feel more comfortable on the material. The message that immediately followed this post provided additional support for an alternative pre-test, as well as reflecting upon other ideas for test validation: I completely agree that the "pre-test" does not have to be an actual test. Mak- ing it a test penalizes the students for not knowing information that they may have never been exposed to, and [it is inappropriate] since you're mostly us- ing it to assess what they do and don't know in order to better plan your curriculum… The first post introduced the idea of using a game as an alternative pre-test and suggested that this approach would help with student motivation. The second post added to this argument from a different perspective, by suggesting that providing an alternative pre-test was not only useful but also very necessary. Such posts constituted evidence that new learning had occurred among the students, as a result of the follow- ups of messages made possible by the interactions of the students in the Spring semester. 4 Conclusions and Discussions This study examined the impact of two different designs of online discussion on stu- dent participation in the discussions and the type of learning that was generated. The results showed that when students were given an opportunity to respond to each other with the instructor mostly absent in the discussion process until the end of the discus- sions, they would visit the discussion forums more frequently. They were also more likely to pose follow-up questions for their classmates and engage in meaningful inquires. On the other hand, when the instructor posted frequently in the discussion forums, students would respond to him instead of their peers, and the discussions were seldom carried forth in much depth. The findings confirmed what was reported in the research literature regarding the negative relationship between instructor involvement and student participation in online discussions [11]. Although the instructor may need to provide more detailed feedback at the end of the discussion, the release from making frequent visits and posts in the forum would save the instructor valuable time. When online discussion is . forth in much depth. The findings confirmed what was reported in the research literature regarding the negative relationship between instructor involvement and student participation in online. re- garding test accommodation in the Spring section. In contrast, no such question was found in a corresponding forum in the Fall section. Instead, most of the messages posted in the semester were attempts. Comparing the Impact of Two Different Designs for Online Discussion 369 3 Results The online version of the volume will be available in LNCS Online. Members of institutes subscribing to the Lecture