Students Performance on Institutional Learning Outcomes

St Cloud State University theRepository at St Cloud State Culminating Projects in Higher Education Administration Department of Educational Leadership and Higher Education 5-2017 Students' Performance on Institutional Learning Outcomes Cheryl R Norman St Cloud State University, crnorman@stcloudstate.edu Follow this and additional works at: https://repository.stcloudstate.edu/hied_etds Part of the Higher Education Commons, and the Higher Education Administration Commons Recommended Citation Norman, Cheryl R., "Students' Performance on Institutional Learning Outcomes" (2017) Culminating Projects in Higher Education Administration 12 https://repository.stcloudstate.edu/hied_etds/12 This Dissertation is brought to you for free and open access by the Department of Educational Leadership and Higher Education at theRepository at St Cloud State It has been accepted for inclusion in Culminating Projects in Higher Education Administration by an authorized administrator of theRepository at St Cloud State For more information, please contact rswexelbaum@stcloudstate.edu Students’ Performance on Institutional Learning Outcomes by Cheryl Ruth Norman A Dissertation Submitted to the Graduate Faculty of St Cloud State University in Partial Fulfillment of the Requirements for the Degree of Doctor of Education in Higher Education Administration May, 2017 Dissertation Committee: Michael Mills, Chairperson Frances Kayona Steven McCullar Krista Soria Abstract In response to increased attention within higher education to develop essential learning outcomes, faculty and administrators have worked to measure students’ performance in relation to these outcomes Students’ learning is often measured through standardized tests or students’ perception of their learning, rather than authentic learning that is connected to the curriculum This study incorporated results from one institution’s involvement in the Association of American Colleges and Universities (AAC&U) project to assess learning outcomes using the Valid Assessment of Learning in Undergraduate Education (VALUE) rubrics The learning outcomes assessed included critical thinking, quantitative literacy, and written communication The purpose of this quantitative study was to identify relationships between students’ demographic background characteristics, collegiate experiences, and performance on three learning outcomes The independent variables included race/ethnicity, age, gender, socioeconomic status, program of study, credit hours and degree level Astin’s I-E-O model was used as a framework for my study to determine whether students grow or change differently based on the characteristics students bring to college and what they experience in college, and compared these variables with their performances on outcomes after exposure to the environment The institution was a Midwestern 2-year public community college Data collected were analyzed using multiple linear regressions to determine which independent variables had statistically significant relationships with higher scores on the three learning outcomes The findings suggested that health majors and students’ increase in age scored higher in critical thinking, African American students and business and math majors scored higher in quantitative literacy, and females and Asian students scored lower in critical thinking Acknowledgements This has been an amazing journey to becoming a Doctor of Education in Higher Education Administration Thank you to my advisor Dr Michael Mills for your constructive feedback, and to my committee members Dr Frances Kayona and Dr Steven McCullar I want to especially thank Dr Krista Soria for her support and advice in analyzing the data I would like to thank all of the instructors listed above as well as many others in this program for their passion, insight, and for taking the time to develop me as a future administrator in higher education Thank you to my parents, Pearl and Duane, for modeling hard work and for being so proud of me as I strived to complete my doctoral program I appreciated the bed and breakfast you provided each weekend of my classes Mom, I wish you were here to attend my graduation I miss you, and even though you are no longer physically here, your presence will be felt Thank you to my children Alexis and Taylor, Clarke and Shelby, Jericho, and Abigail for your support, love, prayers, and encouragement I know this has taken so much of my time away from you and I am incredibly thankful that God has blessed me with each one of you Thank you to my sisters, Carolyn and Pam, for the many phone calls that kept me going during my long drives to and from school as I attended classes Thank you also for listening to me through the highs and lows, and encouraging me to keep going Thank you to Cohort 8, for stretching me in many ways and opening my mind to many new thoughts and ideas Thank you to collaborative Cohort 7, for welcoming me into your classes and writing sessions Thank you especially to my husband, Ken I could not have completed this program or this dissertation without your support and encouragement You believed in me even when I did not You are an amazing, faithful, and gracious life partner I love you so much I am thrilled to be completing this degree just in time to become grandparents together and welcoming baby boy Jones Table of Contents Page List of Tables…………………………………………………………………………………… List of Figures………………………………………………………………………………….…8 Chapter I Introduction………………………………………………………………… .9 Statement of the Problem……………………………………………… 12 Description and Scope of the Research………………………………….13 Research Question……………………………………………………….19 Purpose of the Study…………………………………………………… 20 Assumptions…………………………………………………………… 21 Delimitations…………………………………………………………… 22 Summary…………………………………………………………………23 II Literature Review……………………………………………………………… 25 Definition of the Assessment of Learning Outcomes……………………27 Historical Impact of the Assessment of Learning Outcomes in Higher Education………………………………… .29 Current Trends in the Assessment of Student Learning in Higher Education……………………………………………………… 33 Assessment Tools……………………………………………………… 38 Demographics……………………………………………………………43 Theoretical Framework………………………………………………… 63 Relevance of the Synthesis for the Research Project…………………….81 Summary…………………………………………………………………83 III Method………………………………………………………………………… 84 Research Design………………………………………………………….85 Description of the Sample……………………………………………… 86 Instruments for Data Collection………………………………………….90 Data Collection Procedures………………………………………………95 Analysis………………………………………………………………… 98 Institutional Review Board Process…………………………………….102 Summary……………………………………………………………… 102 IV Results………………………………………………………………………… 104 Demographic Information………………………………………………105 Testing Multiple Linear Regression Assumptions…………………… 110 Multiple Linear Regression Results…………………………………….123 Synthesis……………………………………………………………… 131 Summary……………………………………………………………… 136 V Conclusions ……………………………………………………………………137 Conclusions ……………………………………………………………139 Limitations…………………………………………………………… 151 Recommendations…….……………………………………………… 153 Summary……………………………………………………………… 163 References………………………………………………………………………………164 Appendices………………………………………………………………………… …179 A Critical Thinking VALUE Rubric………………………………………….179 B Quantitative Literacy VALUE Rubric…………………………………… 181 C Written Communication VALUE Rubric………………………………… 183 D Institutional Review Board Approval………………………………………185 List of Tables Table Page Comparison of Population and Sample………………………………………………….89 Coding Scheme…………………………………………………………………………100 Descriptive Statistics for Demographic Independent Variables……………………… 107 Descriptive Statistics for Dependent Variables……………… ……………………….109 Critical Thinking Pearson Correlation………………………………………………….112 Quantitative Literacy Pearson Correlation…………………………………………… 113 Written Communication Pearson Correlation………………………………………… 114 Model Summary for Critical Thinking…………………………………………………124 ANOVA for Critical Thinking………………………………………………………….124 10 Critical Thinking Coefficients………………………………………………………….125 11 Model Summary for Quantitative Literacy………………….………………………….126 12 ANOVA for Quantitative Literacy…………………………………………………… 127 13 Quantitative Literacy Coefficients…………………………………………………… 127 14 Model Summary for Written Communication………………………………………….128 15 ANOVA for Written Communication………………………………………………….129 16 Written Communication Coefficients………………………………………………… 130 17 Significant Predictive Variables……………………………………………………… 133 List of Figures Figures Page Astin’s I-E-O Model…………………………………………………………………… 78 Histogram of Standardized Residuals in Critical Thinking…………………………….117 Histogram of Standardized Residuals in Quantitative Literacy……………………… 117 Histogram of Standardized Residuals in Written Communication…………………… 118 Normal P-P plot for Standardized Residuals for Critical Thinking………………….…119 Normal P-P plot for Standardized Residuals for Quantitative Literacy……………… 119 Normal P-P plot for Standardized Residuals for Written Communication…………… 120 Scatterplot of Standardized Residuals in Critical Thinking…………………………….121 Scatterplot of Standardized Residuals in Quantitative Literacy……………………… 121 10 Scatterplot of Standardized Residuals in Written Communication…………………….122 Chapter I: Introduction Students’ learning is often measured through standardized tests, students’ self-reported evaluation of their learning, or the grades that instructors apply to the assignments students complete that culminates in final semester grades (Berrett, 2016) Those measures of learning are often faulty because standardized tests are not connected to the curriculum, self-reported attitudes regarding learning may not be accurate, and grades not always reflect how much a student has learned For decades, higher education has been accused of having a lack of accountability in measuring student learning The public places pressure upon higher education institutions to carry out assessments of what, and how much, students are learning and to ensure that students are progressing in their learning from point of entry to exit in college (Blaich & Wise, 2011) Yet, even with this focus on student learning, evidence regarding the factors that have a positive impact on student learning remains scarce (Stes, Maeyer, Gijbels, & Petegem, 2012) In response to public pressures and accreditation requirements, higher education administrators and faculty have worked to establish student learning outcomes—and systems of assessment for those outcomes—at their institutions Student learning outcomes assessment in higher education is designed to evaluate the effectiveness of teaching, while gaining an understanding of the outcomes students have learned The assessment and evaluation of teaching and learning is potentially unclear; therefore, it is beneficial to define assessment of learning outcomes, consider why we assess learning outcomes, and understand how to use assessment on college campuses Accrediting bodies have required higher education institutions to assess student learning to be accountable in collecting data and transparent in reporting information regarding student performance on institutional learning outcomes (Blaich & Wise, 2011) 172 gender, prior qualifications and discipline studied Assess & Evaluation in Higher Education 22(3), 317-328 Doi: 10.1080/0260293970220305 JBHE Foundation (2015) Black student college graduation rates remain low, but modest progress begins to show The Journal of Blacks in Higher Education, 50(1), 88-96 Jankowski, N., Hutchings, P., Ewell, P., Kinzie, J., & Kuh, G (2013) The degree qualifications profile: What it is and why we need it now Change, 45(6), 6-15 doi:10.1080/00091383.2013.841515 Jaschik, S (2010, June 21) New evidence of racial bias on SAT Inside Higher Ed Retrieved from https://www.insidehighered.com/news/2010/06/21/sat Kantrowitz, M (2009) Analysis of why some students not apply for financial aid Retrieved from http://www.finaid.org/educators/20090427CharacteristicsOfNonApplicants.pdf Kaplin, W A & Lee, B A (2014) The law of higher education San Francisco, CA: JosseyBass Kim, Y K., & Sax, L J (2011) Are the effects of student-faculty interaction dependent on academic major? An examination using multilevel modeling Research in Higher Education, 52(6), 589-615 King, P M., Baxter Magolda, M B Barber, J P., Perez, R J., & Taylor, K B (2012, November) Researcher learning about assessing student development: Lessons from the Wabash National Study Symposium conducted at the annual conference of the Association for the Study of Higher Education, Las Vegas, NV Kugelmass, H & Ready, D D (2011) Racial/ethnic disparities in collegiate cognitive gains: A 173 multilevel analysis of institutional influences on learning and its equitable distribution Research in Higher Education, 42(4), 323-348 Kuh, G D (2011) What educators and administrators need to know about college student engagement In Harper, S R & Jackson, J F L (Eds.), Introduction to American higher education (pp 189-212) New York, NY: Routledge Kuh, G D., Ikenberry, N A., Cain, T R., Ewell, P T., Hutchings, P., & Kinzie, J (2015, Sept – Oct) Beyond compliance: Making assessment matter Change Retrieved from http://www.changemag.org/Archives/Back%20Issues/2015/SeptemberOctober%202015/assessment_full.html Kuh, G D., Kinzie, J., Buckley, J A., Bridges, B K., & Hayek, J C (2011) Piecing together the student success puzzle: Research, propositions, and recommendations ASHE Higher Education Report, Vol 32, No San Francisco: Jossey-Bass Lakin, J M., Elliot, D C., & Liu, O L (2012) Investigating ESL students’ performance on outcomes assessments in higher education Educational and Psychological Measurement, 72(5), 734-753 Lederman, D (2013, May 6) Public university accountability 2.0 Inside Higher Ed Retrieved from https://www.insidehighered.com/news/2013/05/06/public-university-accountabilitysystem-expands-ways-report-student-learning Lightweis, S (2014) The challenges, persistence, and success of White, working-class, first-generation college students College Student Journal, 48(3), 461-467 Liu, O., & Roohr, K (2013) Investigating ten-year trends of learning outcomes at community 174 Colleges ETS Research Report Series, 2013(2), i–45 doi:10.1002/j.23338504.2013.tb02341.x Loacker, G., Cromwell, L., & O’Brien, K (1985, October) Assessment in higher education: To serve the learner Paper presented at the National Conference on Assessment in Higher Education, Columbia, SC Lumina Foundation (2011) The Degree Qualifications Profile Indianapolis, IN: The Lumina Foundation Maki, P L (2004) Assessing for learning: Building a sustainable commitment across the institution Sterling, Virginia: Stylus Publishing Maki, P (2015) Assessment that works: A national call, a twenty-first-century response Washington, DC: Association of American Colleges and Universities McDonough, M (1997) An assessment of critical thinking at the community college level Dissertation Abstracts International, 58, 2561A Merriam, S B (2009) Qualitative research: A guide to design and implementation San Francisco, CA: Jossey-Bass Money, S (1997) The relationship between critical thinking scores, achievement scores, and grade point average in three different discipline Dissertation Abstracts International, 58, 3401A Morante, E A (2003) A handbook on outcomes assessment for two year colleges College of the Desert, Palm Desert, California National Center for Education Statistics (2013) Digest of education statistics Washington, DC: 175 National Center for Educational Statistics, Institute of Educational Sciences, US Department of Education Retrieved from https://nces.ed.gov/programs/digest/d13/tables/dt13_326.10.asp New Leadership Alliance for Student Learning and Accountability (2012) Assuring quality: An institutional self-assessment tool for excellent practice in student learning outcomes Assessment Washington, DC Nunley, C., Bers, T., & Manning, T (2011) Learning outcomes assessment in community colleges National Institute for Learning Outcomes Assessment Retrieved from http://www.learningoutcomesassessment.org/documents/CommunityCollege.pdf Office of Strategy, Planning & Accountability (2016) MCTC data shop: Data as of spring 2016 Minneapolis, MN: Author Pascarella, E T., & Blaich, C (2013) Lessons from the Wabash National Study of Liberal Arts Education Change: The Magazine of Higher Learning, 45(2), 6-15 Pascarella, E., & Terenzini, P (1991) How college affects students: Findings and insights from twenty years of research San Francisco, CA: Jossey-Bass Pascarella, E., & Terenzini, P (2005) How college affects students: A third decade of research (Vol 2) San Francisco, CA: Jossey Bass Pascarella, E T., Edison, M I., Nora, A., Hagedorn, L., & Terenzini, P T (1995/1996) Cognitive effects of community colleges and four-year colleges Community College Journal, 66(3), 35-39 Pascarella, E T., Mayhew, M J., Rockenbach, A B., Bowman, N A., Seifert, T A., Wolniak, G C., & Terenzini, P T (2016) How college affects students: 21st century evidence that 176 higher education works (Vol 2) San Francisco, CA: Jossey Bass Penn, J D (2011) Future directions for assessing complex general education student learning outcomes New Directions for Institutional Research, (149), 5-14, 109-117 Perna, L W., & Swail, W S (2001) Pre-college outreach and early intervention Thought & Action, 17(1), 99-110 Pike, G R., & Killian, T (2001) Reported gains in student learning: Do academic disciplines make a difference? Research in Higher Education, 42(4), 429-454 Pike, G R., Kuh, G D., & Gonya, R M (2003) The relationship between institutional mission and students’ involvement and educational outcomes Research in Higher Education 44(2), 241-261 Poon, O., Squire, D., Kodama, C., Byrd, A Chan, J., Manzano, L., Furr, S., Bushandat, D., (2016) A critical review of the model minority myth in selected literature on Asian Americans and Pacific Islanders in higher education Review of Educational Research, 86(2), 469-502 doi:10.3102/003465431561205 Punch, K F (2009) Introduction to research methods in education London: Sage Publications Raosoft (2004) Raosoft: Sample size calculator Raosoft, Inc Retrieved from http://www.raosoft.com/samplesize.html Research Consultation (2007) The t-test in dissertation and thesis research Retrieved from http://www.researchconsultation.com/t-test-statistics.asp Reese, R (2013, February 21) Minority testing bias persists The Huffington Post Retrieved from http://www.huffingtonpost.com/ronnie-reese/test-bias-minorities_b_2734149.html Rhodes, T L (Ed.) (2010) Assessing outcomes and improving achievement: Tips and tools for 177 using rubrics Association of American Colleges and Universities Rhodes, T L (2011) Emerging evidence on using rubrics Association of American College and Universities, 13/14, 4(1) Retrieved from https://www.aacu.org/peerreview/20112012/fall-winter/rhodes Rhodes, T L., & Finley, A (2013) Using the VALUE rubrics for improvement of learning and authentic assessment Washington, DC: Association of American Colleges and Universities Roach, R (2001) Is higher education ready for minority America? Black Issues in Higher Education, 18(8), 29 Ross, T., Kena, G., Rathbun, A., Kewal-Ramani, A., Zhang, J., Kristapovich, P., & Manning, E (2012) Higher Education: Gaps in Access and Persistence Study (NCES 2012-046) U.S Department of Education, National Center for Education Statistics Washington, DC: Government Printing Office Samuel, K R., & Scott, J A (2014) Promoting Hispanic student retention in two Texas community colleges Research in Higher Education Journal, 25 Saunders, L (2011) Information literacy as a student learning outcome: The perspective of institutional accreditation Santa Barbara, CA: Library of Congress Sebrell, K., & Erwin, T (1998) Assessment of critical thinking: One performance method analyzed Unpublished paper, Center for Assessment and Research Studies, James Madison University, Harrisonburg, VA Shireman, R (2016) The real value of what students are learning in college New York, NY: 178 The Century Foundation Retrieved from https://tcf.org/content/report/the-real-value-ofwhat-students-do-in-college/ Schmidt, P., (2003, November 28) Academes Hispanic future: The nation’s largest minority group faces big obstacles in higher education, and colleges struggle to find the right ways to help The Chronicle of Higher Education Retrieved from http://www.chronicle.com/article/Academes-Hispanic-Future/20497 Sheard, M (2009) Hardiness commitment, gender, and age differentiate university academic performance British Journal of Educational Psychology, 79(1), 189-204 Doi: 10.1348/000709908X304406 Spaulding, S., & Kleiner, K (1992) The relationship of college and critical thinking: Are critical thinkers attracted or created by college disciplines? College Student Journal, 26, 162166 Stes, A., De Maeyer, S., Gijbels, D., & Van Petegem, P (2012) Instructional development for teachers in higher education: Effects on students' learning outcomes Teaching in Higher Education, 17(3), 295-308 Sternberg, R J (2011, February 8) Who is really adrift? Inside Higher Ed Retrieved from https://www.insidehighered.com/views/2011/02/08/a_critique_of_academically_adrift_a d_the_test_behind_many_of_the_findings Sullivan, D F (2015) The VALUE breakthrough getting the assessment of student learning in college right Washington, DC: Association of American Colleges and Universities Suskie, L (2009) Assessing student learning: A common sense guide San Francisco, CA: Jossey-Bass 179 Terenzini, P., Springer, L., Yaeger, P., Pascarella, E., & Nora, A (1994, November) The Multiple influences on students’ critical thinking skills Paper presented at the meeting of the Association for the Study of Higher education, Orlando Thelin, J R (2011) A history of American higher education Baltimore, MD: John Hopkins University Press Tremblay, K., Lalancette, D., & Roseveare, D (2012) Assessment of higher education learning outcomes: Feasibility study Volume – Design and Implementation OECD Willingham, D T (2007, Summer) Critical thinking: Why is it so hard to teach? American Educator Retrieved from http://www.aft.org/sites/default/files/periodicals/Crit_Thinking.pdf Willie, C V., Garibaldi, A M & Reed, W L (Eds.) (1991) The education of African Americans New York: Auburn House Wright, B D (2002) Accreditation and the scholarship of assessment In T W Banta and Associates (Eds.), Building a scholarship of assessment (pp 223- 239) San Francisco, CA: Jossey-Bass 180 Appendix A: Critical Thinking VALUE Rubric for more information, please contact value@aacu.org The VALUE rubrics were developed by teams of faculty experts representing colleges and universities across the United States through a process that examined many existing campus rubrics and related documents for each learning outcome and incorporated additional feedback from faculty The rubrics articulate fundamental criteria for each learning outcome, with performance descriptors demonstrating progressively more sophisticated levels of attainment The rubrics are intended for institutional-level use in evaluating and discussing student learning, not for grading The core expectations articulated in all 15 of the VALUE rubrics can and should be translated into the language of individual campuses, disciplines, and even courses The utility of the VALUE rubrics is to position learning at all undergraduate levels within a basic framework of expectations such that evidence of learning can by shared nationally through a common dialog and understanding of student success Definition Critical thinking is a habit of mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion Framing Language This rubric is designed to be transdisciplinary, reflecting the recognition that success in all disciplines requires habits of inquiry and analysis that share common attributes Further, research suggests that successful critical thinkers from all disciplines increasingly need to be able to apply those habits in various and changing situations encountered in all walks of life This rubric is designed for use with many different types of assignments and the suggestions here are not an exhaustive list of possibilities Critical thinking can be demonstrated in assignments that require students to complete analyses of text, data, or issues Assignments that cut across presentation mode might be especially useful in some fields If insight into the process components of critical thinking (e.g., how information sources were evaluated regardless of whether they were included in the product) is important, assignments focused on student reflection might be especially illuminating • • • • • Glossary The definitions that follow were developed to clarify terms and concepts used in this rubric only Ambiguity: Information that may be interpreted in more than one way Assumptions: Ideas, conditions, or beliefs (often implicit or unstated) that are "taken for granted or accepted as true without proof." (quoted from www.dictionary.reference.com/browse/assumptions) Context: The historical, ethical political, cultural, environmental, or circumstantial settings or conditions that influence and complicate the consideration of any issues, ideas, artifacts, and events Literal meaning: Interpretation of information exactly as stated For example, "she was green with envy" would be interpreted to mean that her skin was green Metaphor: Information that is (intended to be) interpreted in a non-literal way For example, "she was green with envy" is intended to convey an intensity of emotion, not a skin color 181 CRITICAL THINKING VALUE RUBRIC for more information, please contact value@aacu.org Definition Critical thinking is a habit of mind characterized by the comprehensive exploration of issues, ideas, artifacts, and events before accepting or formulating an opinion or conclusion Evaluators are encouraged to assign a zero to any work sample or collection of work that does not meet benchmark (cell one) level performance Capstone Milestones Benchmark Explanation of issues Issue/problem to be considered critically is Issue/problem to be considered critically is stated, stated clearly and described comprehensively, described, and clarified so that understanding is not delivering all relevant information necessary for seriously impeded by omissions full understanding Issue/problem to be considered critically is stated but description leaves some terms undefined, ambiguities unexplored, boundaries undetermined, and/or backgrounds unknown Issue/problem to be considered critically is stated without clarification or description Evidence Selecting and using information to investigate a point of view or conclusion Information is taken from source(s) with enough interpretation/evaluation to develop a comprehensive analysis or synthesis Viewpoints of experts are questioned thoroughly Information is taken from source(s) with enough interpretation/evaluation to develop a coherent analysis or synthesis Viewpoints of experts are subject to questioning Information is taken from source(s) with some interpretation/evaluation, but not enough to develop a coherent analysis or synthesis Viewpoints of experts are taken as mostly fact, with little questioning Information is taken from source(s) without any interpretation/evaluation Viewpoints of experts are taken as fact, without question Influence of context and assumptions Thoroughly (systematically and methodically) analyzes own and others' assumptions and carefully evaluates the relevance of contexts when presenting a position Identifies own and others' assumptions and several relevant contexts when presenting a position Questions some assumptions Identifies several relevant contexts when presenting a position May be more aware of others' assumptions than one's own (or vice versa) Shows an emerging awareness of present assumptions (sometimes labels assertions as assumptions) Begins to identify some contexts when presenting a position Student's position (perspective, thesis/hypothesis) Specific position is imaginative, taking into account the complexities of an issue Limits of position are acknowledged Others' points of view are synthesized within position Specific position (perspective, thesis/hypothesis) takes into account the complexities of an issue Others' points of view are acknowledged within position (perspective, thesis/hypothesis) Specific position (perspective, thesis/hypothesis) acknowledges different sides of an issue Specific position (perspective, thesis/hypothesis) is stated, but is simplistic and obvious Conclusions and related outcomes (implications and consequences) Conclusions and related outcomes are logical and reflect student’s informed evaluation and ability to place evidence and perspectives discussed in priority order Conclusion is logically tied to a range of information, including opposing viewpoints; related outcomes (consequences and implications) are identified clearly Conclusion is logically tied to information; some related outcomes (consequences and implications) are identified clearly Conclusion is inconsistently tied to some of the information discussed; related outcomes (consequences and implications) are oversimplified 182 Appendix B: Quantitative Literacy VALUE Rubric for more information, please contact value@aacu.org The VALUE rubrics were developed by teams of faculty experts representing colleges and universities across the United States through a process that examined many existing campus rubrics and related documents for each learning outcome and incorporated additional feedback from faculty The rubrics articulate fundamental criteria for each learning outcome, with performance descriptors demonstrating progressively more sophisticated levels of attainment The rubrics are intended for institutional-level use in evaluating and discussing student learning, not for grading The core expectations articulated in all 15 of the VALUE rubrics can and should be translated into the language of individual campuses, disciplines, and even courses The utility of the VALUE rubrics is to position learning at all undergraduate levels within a basic framework of expectations such that evidence of learning can by shared nationally through a common dialog and understanding of student success Definition Quantitative Literacy (QL) – also known as Numeracy or Quantitative Reasoning (QR) – is a "habit of mind," competency, and comfort in working with numerical data Individuals with strong QL skills possess the ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations They understand and can create sophisticated arguments supported by quantitative evidence and they can clearly communicate those arguments in a variety of formats (using words, tables, graphs, mathematical equations, etc., as appropriate) Quantitative Literacy Across the Disciplines Current trends in general education reform demonstrate that faculty are recognizing the steadily growing importance of Quantitative Literacy (QL) in an increasingly quantitative and data-dense world AAC&U’s recent survey showed that concerns about QL skills are shared by employers, who recognize that many of today’s students will need a wide range of high level quantitative skills to complete their work responsibilities Virtually all of today’s students, regardless of career choice, will need basic QL skills such as the ability to draw information from charts, graphs, and geometric figures, and the ability to accurately complete straightforward estimations and calculations Preliminary efforts to find student work products which demonstrate QL skills proved a challenge in this rubric creation process It’s possible to find pages of mathematical problems, but what those problem sets don’t demonstrate is whether the student was able to think about and understand the meaning of her work It’s possible to find research papers that include quantitative information, but those papers often don’t provide evidence that allows the evaluator to see how much of the thinking was done by the original source (often carefully cited in the paper) and how much was done by the student herself, or whether conclusions drawn from analysis of the source material are even accurate Given widespread agreement about the importance of QL, it becomes incumbent on faculty to develop new kinds of assignments which give students substantive, contextualized experience in using such skills as analyzing quantitative information, representing quantitative information in appropriate forms, completing calculations to answer meaningful questions, making judgments based on quantitative data and communicating the results of that work for various purposes and audiences As students gain experience with those skills, faculty must develop assignments that require students to create work products which reveal their thought processes and demonstrate the range of their QL skills This rubric provides for faculty a definition for QL and a rubric describing four levels of QL achievement which might be observed in work products within work samples or collections of work Members of AAC&U’s rubric development team for QL hope that these materials will aid in the assessment of QL – but, equally important, we hope that they will help institutions and individuals in the effort to more thoroughly embed QL across the curriculum of colleges and universities Framing Language This rubric has been designed for the evaluation of work that addresses quantitative literacy (QL) in a substantive way QL is not just computation, it is a habit of mind, a way of thinking about the world that relies on data and on the mathematical analysis of data to make connections and draw conclusions Teaching QL requires us to design assignments that address authentic, data-based problems Such assignments may call for the traditional written paper, but we can imagine other alternatives: a video of a PowerPoint presentation, perhaps, or a well designed series of web pages In any case, a successful demonstration of QL will place the mathematical work in the context of a full and robust discussion of the underlying issues addressed by the assignment Finally, QL skills can be applied to a wide array of problems of varying difficulty, confounding the use of this rubric For example, the same student might demonstrate high levels of QL achievement when working on a simplistic problem and low levels of QL achievement when working on a very complex problem Thus, to accurately assess a students QL achievement it may be necessary to measure QL achievement within the context of problem complexity, much as is done in diving competitions where two scores are given, one for the difficulty of the dive, and the other for the skill in accomplishing the dive In this context, that would mean giving one score for the complexity of the problem and another score for the QL achievement in solving the problem 183 QUANTITATIVE LITERACY VALUE RUBRIC for more information, please contact value@aacu.org Definition Quantitative Literacy (QL) – also known as Numeracy or Quantitative Reasoning (QR) – is a "habit of mind," competency, and comfort in working with numerical data Individuals with strong QL skills possess the ability to reason and solve quantitative problems from a wide array of authentic contexts and everyday life situations They understand and can create sophisticated arguments supported by quantitative evidence and they can clearly communicate those arguments in a variety of formats (using words, tables, graphs, mathematical equations, etc., as appropriate) Capstone Interpretation Ability to explain information presented in mathematical forms (e.g., equations, graphs, diagrams, tables, words) Provides accurate explanations of information presented in mathematical forms Makes appropriate inferences based on that information Milestones Provides accurate explanations of information presented in mathematical forms For instance, accurately explains the trend data shown in a graph Provides somewhat accurate explanations of information presented in mathematical forms, but occasionally makes minor errors related to computations or units Attempts to explain information presented in mathematical forms, but draws incorrect conclusions about what the information means Representation Skillfully converts relevant information into Competently converts relevant information Ability to convert relevant information into an insightful mathematical portrayal in a way into an appropriate and desired mathematical various mathematical forms that contributes to a further or deeper portrayal understanding Completes conversion of information but Completes conversion of information but resulting mathematical portrayal is only partially resulting mathematical portrayal is inappropriate appropriate or accurate or inaccurate Calculation Calculations attempted are essentially all Calculations attempted are essentially all successful and sufficiently comprehensive to successful and sufficiently comprehensive to solve the problem solve the problem Calculations attempted are either unsuccessful or represent only a portion of the calculations required to comprehensively solve the problem Calculations are attempted but are both unsuccessful and are not comprehensive Application / Analysis Ability to make judgments and draw appropriate conclusions based on the quantitative analysis of data, while recognizing the limits of this analysis Uses the quantitative analysis of data as the basis for deep and thoughtful judgments, drawing insightful, carefully qualified conclusions from this work Uses the quantitative analysis of data as the basis for workmanlike (without inspiration or nuance, ordinary) judgments, drawing plausible conclusions from this work Uses the quantitative analysis of data as the basis for tentative, basic judgments, although is hesitant or uncertain about drawing conclusions from this work Assumptions Ability to make and evaluate important assumptions in estimation, modeling, and data analysis Explicitly describes assumptions and Explicitly describes assumptions and provides Explicitly describes assumptions provides compelling rationale for why each compelling rationale for why assumptions are assumption is appropriate Shows awareness appropriate that confidence in final conclusions is limited by the accuracy of the assumptions Communication Uses quantitative information in connection Expressing quantitative evidence in with the argument or purpose of the work, support of the argument or purpose of the presents it in an effective format, and work explicates it with consistently high quality Uses the quantitative analysis of data as the basis for competent judgments, drawing reasonable and appropriately qualified conclusions from this work Uses quantitative information in connection with the argument or purpose of the work, though data may be presented in a less than completely effective format or some parts of the explication may be uneven Uses quantitative information, but does not effectively connect it to the argument or purpose of the work Attempts to describe assumptions Presents an argument for which quantitative evidence is pertinent, but does not provide adequate explicit numerical support 184 Appendix C: Written Communication VALUE Rubric for more information, please contact value@aacu.org The VALUE rubrics were developed by teams of faculty experts representing colleges and universities across the United States through a process that examined many existing campus rubrics and related documents for each learning outcome and incorporated additional feedback from faculty The rubrics articulate fundamental criteria for each learning outcome, with performance descriptors demonstrating progressively more sophisticated levels of attainment The rubrics are intended for institutional-level use in evaluating and discussing student learning, not for grading The core expectations articulated in all 15 of the VALUE rubrics can and should be translated into the language of individual campuses, disciplines, and even courses The utility of the VALUE rubrics is to position learning at all undergraduate levels within a basic framework of expectations such that evidence of learning can by shared nationally through a common dialog and understanding of student success Definition Written communication is the development and expression of ideas in writing Written communication involves learning to work in many genres and styles It can involve working with many different writing technologies, and mixing texts, data, and images Written communication abilities develop through iterative experiences across the curriculum Framing Language This writing rubric is designed for use in a wide variety of educational institutions The most clear finding to emerge from decades of research on writing assessment is that the best writing assessments are locally determined and sensitive to local context and mission Users of this rubric should, in the end, consider making adaptations and additions that clearly link the language of the rubric to individual contexts.This rubric focuses assessment on how specific written work samples or collectios of work respond to specific contexts The central question guiding the rubric is "How well does writing respond to the needs of audience(s) for the work?" In focusing on this question the rubric does not attend to other aspects of writing that are equally important: issues of writing process, writing strategies, writers' fluency with different modes of textual production or publication, or writer's growing engagement with writing and disciplinarity through the process of writing Evaluators using this rubric must have information about the assignments or purposes for writing guiding writers' work Also recommended is including reflective work samples of collections of work that address such questions as: What decisions did the writer make about audience, purpose, and genre as s/he compiled the work in the portfolio? How are those choices evident in the writing in the content, organization and structure, reasoning, evidence, mechanical and surface conventions, and citational systems used in the writing? This will enable evaluators to have a clear sense of how writers understand the assignments and take it into consideration as they evaluate The first section of this rubric addresses the context and purpose for writing A work sample or collections of work can convey the context and purpose for the writing tasks it showcases by including the writing assignments associated with work samples But writers may also convey the context and purpose for their writing within the texts It is important for faculty and institutions to include directions for students about how they should represent their writing contexts and purposes Faculty interested in the research on writing assessment that has guided our work here can consult the National Council of Teachers of English/Council of Writing Program Administrators' White Paper on Writing Assessment (2008; www.wpacouncil.org/whitepaper) and the Conference on College Composition and Communication's Writing Assessment: A Position Statement (2008; ww.ncte.org/cccc/resources) Glossary • Content Development: The ways in which the text explores and represents its topic in relation to its audience and purpose • Context of and purpose for writing: The context of writing is the situation surrounding a text: who is reading it? Under what circumstances will the text be shared or circulated? What social or political factors might affect how the text is composed or interpreted? The purpose for writing is the writer's intended effect on an audience Writers might want to persuade or inform; they might want to report or summarize information; they might want to work through complexity or confusion; they might want to argue with other writers, or connect with other writers • Disciplinary conventions: Formal and informal rules that constitute what is seen generally as appropriate within different academic fields Writers will incorporate sources according to disciplinary and genre conventions, according to the writer's purpose for the text Through increasingly sophisticated use of sources, writers develop an ability to differentiate between their own ideas and the ideas of others, credit and build upon work already accomplished in the field or issue they are addressing, and provide meaningful examples to readers • Evidence: Source material that is used to extend, in purposeful ways, writers' ideas in a text • Genre conventions: Formal and informal rules for particular kinds of texts and/or media that guide formatting, organization, and stylistic choices, e.g academic papers, poetry, webpages, or essays • Sources: Texts (written, oral, behavioral, visual, or other) that writers draw on as they work for a variety of purposes to extend, argue with, develop, define, or shape their ideas, for example 185 WRITTEN COMMUNICATION VALUE RUBRIC for more information, please contact value@aacu.org Definition Written communication is the development and expression of ideas in writing Written communication involves learning to work in many genres and styles It can involve working with many different writing technologies, and mixing texts, data, and images Written communication abilities develop through iterative experiences across the curriculum Capstone Milestones Demonstrates awareness of context, audience, purpose, and to the assigned tasks(s) (e.g., begins to show awareness of audience's perceptions and assumptions) Benchmark Context of and Purpose for Writing Includes considerations of audience, purpose, and the circumstances surrounding the writing task(s) Demonstrates a thorough understanding of context, audience, and purpose that is responsive to the assigned task(s) and focuses all elements of the work Demonstrates adequate consideration of context, audience, and purpose and a clear focus on the assigned task(s) (e.g., the task aligns with audience, purpose, and context) Demonstrates minimal attention to context, audience, purpose, and to the assigned tasks(s) (e.g., expectation of instructor or self as audience) Content Development Uses appropriate, relevant, and compelling content to illustrate mastery of the subject, conveying the writer's understanding, and shaping the whole work Uses appropriate, relevant, and compelling content to Uses appropriate and relevant explore ideas within the context of the discipline and content to develop and explore shape the whole work ideas through most of the work Uses appropriate and relevant content to develop simple ideas in some parts of the work Genre and Disciplinary Conventions Formal and informal rules inherent in the expectations for writing in particular forms and/or academic fields (please see glossary) Demonstrates detailed attention to and successful execution of a wide range of conventions particular to a specific discipline and/or writing task including organization, content, presentation, formatting, and stylistic choices Demonstrates consistent use of important conventions particular to a specific discipline and/or writing task(s), including organization, content, presentation, and stylistic choices Follows expectations appropriate to a specific discipline and/or writing task(s) for basic organization, content, and presentation Attempts to use a consistent system for basic organization and presentation Sources and Evidence Demonstrates skillful use of high-quality, credible, relevant sources to develop ideas that are appropriate for the discipline and genre of the writing Demonstrates consistent use of credible, relevant sources to support ideas that are situated within the discipline and genre of the writing Demonstrates an attempt to use credible and/or relevant sources to support ideas that are appropriate for the discipline and genre of the writing Demonstrates an attempt to use sources to support ideas in the writing Uses straightforward language that generally conveys Uses language that generally Uses language that sometimes impedes Control of Syntax and Mechanics Uses graceful language that skillfully communicates meaning to readers with meaning to readers The language in the portfolio has conveys meaning to readers with meaning because of errors in usage clarity and fluency, and is virtually error-free few errors clarity, although writing may include some errors 186 Appendix D: Institutional Review Board Letter ... learning outcomes The nonexistence of common institutional learning outcome definitions and 13 criteria leads to difficulty in comparing data on the assessment of institutional learning outcomes. .. regarding the performance of students on institutional learning outcomes There were limitations in the data This study was based on research at one institution, selected for its participation in the... institution There may be a similar set of institutional learning outcomes across colleges and universities, but there is not a common definition or set of criterion assigned to institutional learning