Webinar: Creating Rubrics to Assess Writing and Speaking 9-12 January 18, Bangkok Readings for interested participants before the webinar Make Room for Rubrics: http://teacher.scholastic.com/professional/assessment/roomforubrics.htm Understanding Rubrics: http://www.middleweb.com/rubricsHG.html RubiStar: http://rubistar.4teachers.org/ Writing Objectives: http://ets.tlt.psu.edu/learningdesign/objectives/writingobjectives/ Classifying Objectives (a revised Bloom's Taxonomy): http://edtech.tennessee.edu/~bobannon/classifications.html 9:05 Poll at the outset – ideal if people can check more than one box: What level you teach? - Primary - Secondary - Post-secondary adults - Teachers What subjects you usually teach? - Writing - Speaking - Reading - Listening - Pronunciation - Grammar - Vocabulary - English for Specific Purposes 9:10 PPT - slide numbers are on the left Creating Rubrics to Assess Writing and Speaking title slide Poll: What is the correct form of the verb in parentheses in the following sentence? I in Bangkok last Sunday (arrive) - arrive - arrived - have arrived - will arrive How many of your classroom assessments look like this? For most people, this type of discrete-answer (one answer is correct) is very common We all use these in formative and summative assessment These are quick to score, and there are rarely arguments about what counts as correct 9:15 How you assess writing? Q: Why are you learning English? I am and work in museum So, if my English vocabulary, grammar and pronunciation could be enhance, I think that I could get some advantage like the following My job could be more interesting when I meet somebody from other countries Now, the English guide take charge of these groups When it happen that I had to guide English groups in the museum, i had research my words everytime and the conversation wasn't nice for me If I want to change job, I think that the English, even if its not necessary for the work that I will do, will be part of application Where will you start? What will you focus on? Talk with your group; respond in the chat if you’re working by yourself People in the room will respond, too 9:25 What are rubrics? Go ahead and answer in the chat; people in the room will answer, too 9:30 Checklists are one way people have created assessments SHIFT to website and Word file Sample checklist for writing http://www.scribd.com/doc/35772510/Writing-Checklist - see My Writing Checklist.doc Do you use these? Are they helpful to weak students as well as to strong students? Poll: Are they helpful to weak students as well as to strong students? - Yes - No - Sometimes 9:40 Back to PPT A rubric looks a little different Rubrics are often in matrix format: Characteristic or Task element Weak (1 point) Average (2 points) Very Good (3 points) Characteristic or Task element Characteristic or Task element Can you think of some benefits? Respond in the chat and in the room 9:45 Some benefits of rubrics for students Students can fully understand what they need to in order to succeed with a task Students work on self-assessment, which is a critical thinking skill Students can more easily peer assessment if they are using rubrics Grading is fairer Some benefits for teachers The teacher can fully understand the task and decide what is important The teacher fully understands what he or she is asking students to Grading is fairer Grading is much faster and easier (More? Respond in the chat and in the room What about drawbacks?) 10 Drawbacks of rubrics They take time to well They take practice to well You can’t change your mind as you are grading, even if you realize you forgot something important The rubric is a contract that binds the teacher as well as the student A poorly-designed rubric is not helpful to the students or to the teacher (The contract is both good and bad Even if you forgot something important, you can’t add it If your rubric is good, students’ work is better Any poorly-designed assessment is not helpful.) 9:50 Let’s look at how to create a rubric, first for writing, then for speaking I’ll also show how to use Rubistar to help you create your rubrics 11 Six steps to create a rubric, adapted from Create a Rubric Tutorial at http://health.usf.edu/publichealth/eta/Rubric_Tutorial/default.htm Start with the performance objective Identify the characteristics/tasks comprising the performance Identify the potential levels of quality (e.g., very good, good, fair, weak, very weak) Assign a point value to each level, and a total point value for the assessment Identify the criteria for each level of quality within a characteristic/task Create the rubric table – put the dimensions as rows and the levels as columns 9:55 12 Step 1: Performance Objectives with the ABCD Model "How to Write Clear Objectives" from Pennsylvania State University's Teaching and Learning office gives the basic overview of the ABCD model that we will be using: http://ets.tlt.psu.edu/learningdesign/objectives/writingobjectives/ Objectives need to be based on specific, observable behavior (e.g., not “learn” and “understand” but “create” or “list”) A = Audience (who is being evaluated?) B = Behavior (observable!) C = Condition (under what conditions will the behavior occur?) D = Degree (how much/many, how fast, how accurately, etc – what makes it good?) – also called criteria 13 Examples of WellWritten Objectives – from PSU Below are some example objectives which include Audience (A), Behavior (B), Condition (C), and Degree of Mastery (D). Note that many objectives actually put the condition first Cognitive (application level) "Given a sentence written in the past or present tense, the student will be able to rewrite the sentence in future tense with no errors in tense or tense contradiction (i.e., I will see her yesterday.)." Affective "Given the opportunity to work in groups, the student will demonstrate an positive increase in attitude towards cooperation, as measured by a survey." 14 Cognitive (comprehension level) "Given examples and nonexamples of constructivist activities in a college classroom, the student will be able to accurately identify the constructivist examples and explain why each example is or isn't a constructivist activity in 20 words or less." Participants in this webinar (Audience), after listening carefully and participating actively in group work (Condition), will create rubrics (Behavior) that will contain at least three rows for characteristics and at least three columns for levels (Degree) The Degree/criteria for more complex assessment are best addressed with rubrics 10:00 15 Let’s go back to that initial student writing: Q: Why are you learning English? I am and work in museum So, if my English vocabulary, grammar and pronunciation could be enhance, I think that I could get some advantage like the following My job could be more interesting when I meet somebody from other countries Now, the English guide take charge of these groups When it happen that I had to guide English groups in the museum, i had research my words everytime and the conversation wasn't nice for me If I want to change job, I think that the English, even if its not necessary for the work that I will do, will be part of application Please create a separate pod with this, preferably at 20 point size or larger Q: Why are you learning English? I am and work in museum So, if my English vocabulary, grammar and pronunciation could be enhance, I think that I could get some advantage like the following My job could be more interesting when I meet somebody from other countries Now, the English guide take charge of these groups When it happen that I had to guide English groups in the museum, i had research my words everytime and the conversation wasn't nice for me If I want to change job, I think that the English, even if its not necessary for the work that I will do, will be part of application PPT 16 How would you write the performance objective? Work with a partner, or watch the chat A = Audience (who is being evaluated?) B = Behavior (observable!) C = Condition (under what conditions will the behavior occur?) D = Degree (how much/many, how fast, how accurately, etc – what makes it good?) – also called criteria Later: Create a performance objective for one activity in one lesson (not for the whole course!) 10:10 (keep the student writing pod open, if possible) 17 PPT Step 2: Identify the characteristics/tasks comprising the performance This is often the hardest part What exactly are you looking for? What are some characteristics that you think are important here? Spelling/capitalization/punctuation (mechanics) Grammar Organization Content Interest Specific examples Make sure you focus on enough but not too many Try to keep the rubric to one or two pages at most 10:20 18 Step 3: Identify the potential levels of quality More levels is a lot more work Have enough, but no more Three levels: weak, average, very good (9 cells with characteristics) Weak (1 point) Average (2 points) Characteristic or Characteristic or task element Characteristic or task element Very Good (3 points) Four levels: very weak, almost satisfactory, satisfactory, excellent (12 cells with characteristics) Five levels: very weak, weak, average, very good, excellent (15 cells with characteristics – and you often have more than That’s a lot of writing) 10:30 19 Step 4: Assign a point value to each level, and a total point value for the assessment Suggestions from the Rubric Tutorial: If you already know the total number of points for this assessment Divide the total number of points by the number of dimensions to get the maximum point value for achieving the highest gradation for a dimension For example, if the assessment is worth 15 points, and there are dimensions, the highest gradation will be worth points Each gradation below this highest level should be valued at less than the points If you not have a total number of points for this assessment Identify the maximum number of points for achieving the highest level of quality Assign a number to each of the lower levels of quality Typically, the gradations are in increments of point Multiply the maximum point value by the total number of dimensions This is the total point value for the assessment For example, if you have levels (poor, average, very good) the highest level is worth points, the middle level is worth points and the lowest level is worth point If there are dimensions, the total point value for this assessment is 21 points You can assign a point value of 0, typically for something that was not addressed at all You can also have a point spread for each level, though this makes fairness more difficult Sample calculation – if points in characteristics, total can be 20 (or 15, if one of the levels is 0) 10:40 20 Step 5: Identify the criteria for each level of quality within a characteristic/task This is what takes time Start with what makes something excellent How would a student get the maximum number of points for each characteristic/task? Be as complete as possible in thinking about this Now, look at the weakest What would a very weak characteristic/task look like? Finally, look at the middle areas What is missing in each characteristic/task? Describe each of the levels carefully and clearly, so that students can understand what you want Please show the student writing pod again Let’s go back to that initial student writing again: I am and work in museum So, if my English vocabulary, grammar and pronunciation could be enhance, I think that I could get some advantage like the following My job could be more interesting when I meet somebody from other countries Now, the English guide take charge of these groups When it happen that I had to guide English groups in the museum, i had research my words everytime and the conversation wasn't nice for me If I want to change job, I think that the English, even if its not necessary for the work that I will do, will be part of application 11:00 (keep the student writing pod open, if possible) 21 Step 6: Create the rubric table Now that you’ve figured out what should go into each cell, you can create your rubric I like to put the course name, specific assignment, and the date on the rubric Tables in Word work well for creating templates for rubrics 22 Sample table with three characteristics, and three points Weak (1 point) Average (2 points) Content No examples that At least one example (interesting ideas, explain why you that explains why you examples) want to learn want to learn English English Grammar Several mistakes or small mistakes, Very Good (3 points) At least two examples that explain why you want to learn English No mistakes in what Mechanics that make it hard to understand your meaning Several mistakes that make it hard to understand your meaning but the reader can understand your meaning or small mistakes, but the reader can understand your meaning we have studied so far (simple present and simple past) No mistakes in what we have studied so far (capitalization, punctuation, common spelling words) What your characteristics look like? 11:05 23 Online help with rubrics: Rubican and Rubistar Switch to viewing websites Rubican examples - http://www.rubrician.com/writing Persuasive writing rubric (US 8th graders): http://www.intercom.net/local/school/sdms/mspap/wkidpers.html Informative writing rubric (US 8th graders): http://www.intercom.net/local/school/sdms/mspap/wkidinf.html 11:15 RubiStar examples – persuasive writing, persuasive speaking http://rubistar.4teachers.org/index.php? screen=CustomizeTemplate&bank_rubric_id=127§ion_id=5& (persuasive essay) RubiStar gives you ideas about what you might find helpful YOU need to decide what fits your behavioral objective! You can change any of the characteristics or descriptions Persuasive writing 11:20 PPT 24 Speaking characteristics Oral presentation 25 Create a rubric Now you create a rubric with a partner and share it in your group If you want, I can add it to the wiki for this presentation – email it to me and I’ll share it there Make sure you include your name, your school, the class, the age of the students, and the objective Poll (if possible, otherwise in PPT) What you think about using rubrics? - I’m ready! - I need a bit more practice - I don’t think I’ll use rubrics - What are rubrics? 11:45 PPT 27 Q&A ... poorly-designed assessment is not helpful.) 9:50 Let’s look at how to create a rubric, first for writing, then for speaking I’ll also show how to use Rubistar to help you create your rubrics 11 Six steps to. .. respond, too 9:25 What are rubrics? Go ahead and answer in the chat; people in the room will answer, too 9:30 Checklists are one way people have created assessments SHIFT to website and Word... characteristics – and you often have more than That’s a lot of writing) 10:30 19 Step 4: Assign a point value to each level, and a total point value for the assessment Suggestions from the Rubric Tutorial: