Terry-Rhodes-Using-Value-Rubric-Results-for-Learning-Improvement

Using VALUE Rubric Results for Learning Improvement, Professional Development and Equity: Assessment that Empowers Faculty to Take Risks with Pedagogical Innovation LEAP Texas Terrel L Rhodes Association of American Colleges and Universities March 25, 2018 The Traditional Assessment & Accountability Landscape Movement from course-embedded and program-level assessment to more global, institutional assessment = increase cost, harder to tie results directly to improving teaching & learning at the local level; quality assurance mechanisms evidence generated requires valid and reliable measures that transcend local conditions in order to set effective policy The traditional measure preferred at the policy level – commercially available standardized tests – lack transparency in design and the ability to disaggregate data below the institutional level to make changes to improve teaching and learning at the course and program level Regional Accreditation Teaching & Learning Course Level Teaching, Learning & Assessment Program Assessment Disciplinary and/or Professional Accreditation General Education/ Core Curriculum Assessment Institutional Assessment System and/or State Level Accountability Federal Level Accountability Other Accountability Mechanisms (e.g., VSA) Policy & Quality Assurance Institutional Level Course-Level Recognize and promote student agency and faculty development and expertise in order to improve teaching and through the adoption of active learning pedagogies and enhanced assignment design Create guided learning pathways – including successful 2- to 4-year transfer - to promote retention and completion for all students, while addressing quality assurance and accountability requirements through general education and beyond The VALUE Model Evidence of quality student learning to: Program Level Design curricula that leverage high-Impact practices within and across degree areas that respect disciplinary paradigms and professional standards while promoting the attainment of higher order necessary abilities to thrive in work, citizenship, and life for all students Policy Level To create a common language of evidence that facilitates collaboration across the triad – system/state, federal, and regional accreditation – and enables the development of sound public policy to promote individual student success and educational attainment for the common good The Key Elements for a Compelling Quality Framework Already Are in Hand  Consensus Aims and Outcomes  Practices that Foster Achievement AND Completion  Evidence on “What Works” for Underserved Students  Assessments That Raise – and Reveal – the Level of Learning The VALUE Rubric Approach to Assessing Student Learning Valid Assessment of Learning in Undergraduate Education www.aacu.org/value VALUE Rubric Approach - Assumptions  Learning is a process that occurs over time  Student work is most robust representation of student motivated learning  Focus on what student does in terms of key dimensions of learning outcomes  Faculty and educator expert judgment  Results are useful and actionable for improvement of learning VALUE Embraces Imperfection as Part of the Learning Process “Never Let the Perfect Get in the Way of the Good” VALUE embraces the variables that other assessment approaches control or eliminate in their consideration of student learning, including:  Individual, faculty-designed assignments taken straight off the syllabus and out of the classroom There are no required common prompts  An approach to sampling that is designed to raise up, not wash out, the inherent diversity—from race, ethnicity, and socioeconomic status to the diversity of courses, creditlevels, and disciplinary backgrounds—found on campuses  Scorer training sessions that are equal parts calibration to reach a consensus score and a rich faculty development opportunity, and that are open to all faculty whether they are contingent or tenure-track, two-year or four-year, curricular or co-curricular www.aacu.org/OnSolidGroundVALUE VALUE Project map: The Multi-State, Minnesota, and Great Lakes Colleges Association Collaboratives Multi-state Collaborative Multi-state and Minnesota Collaboratives “ Participation in VALUE Hamline University “ has diversified and expanded how we understand learning Math faculty one-hour assessment charrette and offered it as a companion to an “assessment salon” – feedback on assignments and discussion of assessment French faculty they would like to something like this again…grow a stronger culture of assessment at the college the faculty really enjoyed sharing what they were doing • one whole department talked about how they are collecting data and working alongside career services, counseling, and advising to figure out if students are on the right pathway.” Campus Benefits and Uses PROGRAM AND COURSE IMPROVEMENT Applying lessons learned to our local initiatives on campus • MSC provided a means for better general education assessment • Potential: – Sustainable model to assess general education – Less work for individual faculty member (than current model) – Broader faculty engagement across campus – Assess what a student knows at 90+ credits –use major and gen ed courses – Use results for VSA/College Portrait??? • Mini MSC- Assessment Retreat –CCSU faculty scoring same artifacts – Compare home scores to MSC scores – Very manageable, especially with use of Taskstream for uploading and scoring of artifacts Reflections on the Pilot Year Learning Outcome & Faculty Faculty (N) Number of Artifacts First Year Soph Junior Senior CCSU Total MSC Total Critical Thinking (33 Majors) 12 16 21 58 130 225 119 Quantitative Reasoning (19 majors) 29 82 117 78 Written Communication (28 Majors) 13 13 19 62 97 191 87 Grand Total (45 majors – 75% of majors) (27 faculty – 45% of dept) 27 29 46 140 318 533 283 “…data…collected over the last eight or nine years to look at programs “more holistically” and evaluate staffing, course sequencing, or programwide curricula…” • Math, Statistics, and Computer Science department…started using a statistical software manual “because they realized that students didn’t have quite the competence level … as they wanted [them] to have when they graduated…” Project and Institution De-Identified Institutional Data Criterion by Criterion Took Apart the Rubrics Focus Sources and Evidence Action Project • Project as a Whole Institutional • Local Scoring Course • Individual Faculty Research highlights importance of faculty and student success and equity 33 Assignment Difficulty Critically Important to Collect INTRODUCE Assignment designed to introduce the outcome PRACTICE Assignment designed to afford student practice with the outcome REINFORCE Assignment designed to reinforce previously practiced outcome MASTERY Assignment designed for students to demonstrate level of mastery of the outcome Analysis of student work assessed using the Critical Thinking and Written Communication VALUE Rubrics, seniors’ work was significantly more likely to be scored at the “Capstone” level—the highest level of performance—when the assignment was designed to produce work at the Capstone level When the assignment was “easier” seniors’ performance, on average, went down to meet the lower expectations of the assignment When asked for less, they produced less Achieving it would transform the learning outcomes of American higher Inclusive excellence is an affordable, feasible education goal if the highest impact high-impact practice is high quality and appropriately demanding assignments non-white and lower income students gain access to high quality and demanding assignments less often But insisting that all Our notion of high-impact faculty give assignments practices consists that are both demanding primarily of things like Weresearch, need to begin thinking of high and andquality intentional about student-faculty the highest higher-order learning overseasdemanding study, and assignments as perhaps and lowest cost high-impact practice goals not just content participatingimpact in learning learning goals is not communities—practices financially costly, only where increasing the politically and supply is costly in higher managerially difficult education Lessons Learned from VALUE/MSC • Context or landscape is important • Local data are critical • Data need deconstruction/disaggregation at local level • Interdisciplinary/integrative experience is required to attain high quality levels associated with graduation • What faculty/educators is foundational to achieve quality student learning Questions or Comments? rhodes@aacu.org