Is Critical Thinking the All-purpose Outcome in Higher Education?1 John F Stevenson2 November 13, 2010 Introduction How can evaluators in higher education work with administrators and faculty to select, implement, and learn from institution-level measures of crucial learning outcomes? Critical thinking is a pervasive choice, and this paper explores issues in its definition and measurement, drawing on experiences of one university along with the published literature In definitions of learning outcomes in higher education, critical thinking is ubiquitous For evaluators who work with administrators and faculty, considering ways to define and measure critical thinking is a likely role At both the departmental level and the institution-wide level, academic skills have been a central feature in higher education renewal efforts across the country Allen (2006), for example, lists “a common core of broad expectations” including “written and oral communication, critical thinking, and information literacy” and characterizes them as “virtually universal” (p.34) Employers want them, state legislators want them … they are a part of the accountability movement in higher education that higher education evaluators know has been gaining momentum over the past 15 years or so (AAC&U, 2005) An emphasis on skills like critical thinking is also consistent with the effort to find ways to measure student learning outcomes; on the face of it skills lend themselves to operationalizable definitions In contrast to other intended outcomes that appear more abstract (e.g global awareness, selfreflective and ethical decision-making, commitment to lifelong learning), skills seem straightforward, measurable entities with an easy-to-grasp practical significance A survey of 433 Chief Academic Officers of higher education institutions conducted in early 2009 found that 74% of the institutions reported that critical thinking was one of the areas of intellectual skills addressed in their common learning goals for students, second only to writing skills in frequency (Hart Research Associates, 2009) This skill is recommended as part of AAC&U’s (2007) LEAP Initiative as one of six “Intellectual and Practical Skills” to be incorporated in a set of essential learning outcomes, and 73% of a sample of employers interviewed in 2006 recommended that colleges and universities place more emphasis on “critical thinking and analytical reasoning skills.” Reflecting this interest in critical thinking as an important learning outcome, national attention has been paid to its measurement For example, the Collegiate Learning Assessment (CLA) developed by the Council for Aid to Education (Shavelson, 2009) features a complex real-world problem context for assessing this skill, the Collegiate Assessment of Academic Proficiency (CAAP) developed by ACT uses real-world stimulus scenarios with a set of structured follow-up questions, and the VALUE Project sponsored by AAC&U (AAC&U, 2009) presents a “metarubric” for assessing it, distilling the common core from rubrics used by a large group of institutions across the country This paper will address alternative definitions and Paper presented at the annual meeting of the American Evaluation Association, San Antonio, TX Department of Psychology, University of Rhode Island, 10 Chafee Road, Kingston, RI 02881; 401-874-4240; jsteve@uri.edu measurement approaches with attention to ongoing definitional issues and the usefulness of alternate approaches in actual curricular change Defining and measuring “critical thinking” at the University of Rhode Island Table presents a summary of methods used at the University of Rhode Island to assess aspects of student learning related to critical thinking As the table makes clear, various methods, selected or developed by various entities within the institution, are employed At the program (departmental) level we not yet have a classification system that would allow us to count the number of departments that use some measure of critical thinking In Psychology this is one of ten identified learning outcome goals for our undergraduate program, and it has been given early priority Across the university, FSSE and NSSE results can be used to track the perceived availability of opportunities for learning that are associated with critical thinking, and compare with norms for similar institutions However, the data for these items are rarely made accessible to faculty, and faculty not generally seek them out How the psychometric strengths and available norms could affect internal refection on ways to enhance student learning is unknown, as the data have not been used in that way A faculty committee responsible for assessing the general education program devised a set of questions for students to complete along with end-of-semester course evaluations These are directly relevant for assessing the implementation of skill-focused aspects of the courses, and the results for a stratified sample of general education courses have been reported within the committee and summarized for external consumption CAAP results have not been shared beyond the small group of staff and faculty who have worked to implement the Wabash study at URI The first cohort to receive senior-year testing is due this spring Critical thinking as defined by a series of cognitive outcome objectives (similar to Bloom’s taxonomy of cognitive objectives) has been promoted as a means of assessing the impact of general education The faculty committee responsible for assessing general education anticipated that each major knowledge area (social sciences, natural sciences, etc.) would approach these learning outcomes in different ways, but that all would share commitment to them at a higher level Student assignments have been used to explore the value of the model, and rubrics have been drafted by the committee Local interest may be strongest when accreditation pressures rise; in the mean time internal differences of opinion about the utility of the model as applied across subject areas have slowed its application Defining critical thinking: generic definitions Historically there have been a few major threads in efforts to define the concept Scriven & Paul(1987) authored a definition that has much in common with Bloom’s taxonomy They stated, “Critical thinking is the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or generated by, observation, experience reflection, reasoning, or communication, as a guide to belief and action.” They went on to say it can and should transcend separate disciplines They also pointed to two basic aspects of the process: the skills for doing critical thinking and the inclination to use them Paul and Elder (2008) emphasized a humanistic theme in their definition: “Critical thinking is that mode of thinking … in which the thinker improves the quality of his or her thinking by skillfully taking charge of the structures inherent in thinking and imposing intellectual standards upon them.” Thus we have (1) a set of cognitive skills for analyzing information and addressing problems across a wide range of contexts; (2) the disposition to approach new situations from that critical perspective (thinking before acting); and (3) the habit of applying those same critical faculties to one’s own thought processes in a metacognitive way Halpern (1993), with a focus on the teaching of critical thinking, adds the important qualification that the skills should generalize beyond the context in which they were acquired, and be reflexively transferred to the “real world” contexts where they matter AAC&U’s VALUE rubric for critical thinking invokes the transdisciplinary nature of the activity, and defines a set of five important facets: (1) clear and comprehensive understanding of the problem; (2) careful development of information needed to address the problem; (3) critical analysis of assumptions and contexts; (4) creatively synthesized, clearly articulated position on the issue, with recognition of complexities and limits; and (5) logical derivation of consequences and implications with appropriate weighting of considerations Choices for measurement: Does a one-size-fits-all approach work? Both personal experience and the literature suggest that there are very different ways to define critical thinking, and issues in measurement extend from these definitional conundrums One dimension of difference concerns the need for domain-specific approaches to measurement In psychology several authors have empirically explored the question of how generic these skills really are Renaud and Murray (2008) provide an illustration of the discipline-linked ways of thinking about critical thinking, as they cite a definition in the psychology literature that emphasizes evaluation of evidence and logical links of the evidence to claims in order to conclude whether the claims have been supported That is certainly how I have always thought of it Renaud and Murray go on to demonstrate that subject-specific tests of critical thinking better than generic measures at demonstrating sensitivity to teaching effects – put another way, transfer to other contexts is not so easy to find Indeed, the published literature suggests that it has been difficult to show that course experiences transfer to generic skills In addition to the choice of generic vs context-related measurement, does it work better to use a test (whether general or course-related) or does it make more sense to use a rubric to evaluate the quality of actual work in classes? In the latter case, is it practical to use a generic rubric across many courses (as the AAC&U suggests) or such rubrics fail to capture the intended effects of particular courses? Here the conversation is partly about measurement sensitivity, partly about finding good comparison norms, and partly about the likelihood of impact of findings on local pedagogy Another dimension I have encountered is related to the content to which the critical thinking is applied Much of the definitional language seems aimed at the subject’s ability to examine the work of someone else (for example a published study or essay or opinion piece) and critically deconstruct it to arrive at defensible conclusions about its merit and better alternative conclusions A different context for critical thinking is presented when the subjects are responsible for asking and answering a question of their own, thus applying the critical perspective to their own evidence, reasoning, and conclusions In psychology, the design and conduct of research studies can be a part of the teaching/learning process from very early on In disciplines farther removed from the social sciences, such as the fine arts, I have encountered even more dramatically different ways of thinking about what constitutes critical thinking, and what problem contexts are addressed Think about how an actor takes on a new role, or an artist “sees” an object in a new way Perhaps the generic measures (e.g CLA, CAAP) are a bit too far removed from many of our disciplinary contexts to reflect much transfer from the educational experiences those contexts provide Who gets to decide what critical thinking is, and does that matter? As Table displays, at the University of Rhode Island many players have a bit part in assessing critical thinking The institutional research office, the assessment office, the Faculty Senate and its committees, and many departments – all are engaged in defining and measuring some version – or versions of this construct As a faculty member with long service on committees dealing with assessment and general education, as well service as a department chair and graduate program director, I have many reasons to be interested in how well we are doing at enhancing critical thinking, and what we could to improve Yet I have very little to show for it, despite all of the interesting approaches to measurement presented in Table My own view is that we need ongoing conversation across the disciplines about what we mean by critical thinking – and these conversations should be informed by the actual assignments we use to develop and measure it Indeed, our taxonomy of cognitive objectives is intended to lead to exactly that kind of conversation It may be less elegant than an externally normed measure, but it represents what faculty actually in their classes to promote “higher-order” information processing (Renaud & Murray, 2007), and how they think about student learning With a group of faculty fellows trying out new interdisciplinary first-year seminars, we are taking another look at how students on those cognitive learning objectives, and we hope to have a receptive audience to which we can report the results Our local conversation points to a larger issue: which of the several considerations in making measurement choices matters the most? For external accountability, and possible pride among peer institutions, the generic, nationally normed test has clear advantages These advantages may be most salient for administrators For an academic department, a context-specific test may provide the most efficient and reliable way to track improvement over time in student skills Validity and pedagogical impact are still open questions for this approach, and portfolios or samples of assignments reviewed by a faculty committee with a departmental rubric may have some advantages for those values However, for the general education context, a special challenge is that there is no faculty constituency readily at hand We are betting that the development of a locally owned rubric, applied to assignments from a variety of general education courses from across many disciplines, can have an internal impact on courses and on the structure of the requirements themselves The availability of a group of faculty teaching first year seminars and meeting to discuss how that is going may provide the constituency group necessary for “completing the loop” on the inside If that fails, administrators may have an easier time calling for generic testing As an evaluator I hope for a logic model linking process (i.e curricular elements and pedagogical methods) to short-term outcomes (i.e rubric-assessed assignments), medium-term outcomes (i.e department-wide tests) and to long-term outcomes (i.e generic tests like CAAP and CLA used to measure value-added across the institution) See Figure for a schematic representation of this logic That may just be a step too far, and too costly in money and person hours Still, it’s a pretty picture! Bibliography Adams, M.H., Whitlow, J.F., Stover, L.M., & Johnson, K.W (1996) Critical thinking as an educational outcome: An evaluation of current tools of measurement Nurse Educator, 21(3), 23-32 Allen, M.J (2006) Assessing General Education Programs San Francisco: Anker/Jossey-Bass Association of American Colleges (1994) Strong Foundations: Twelve Principles for Effective General Education Programs Author, 1818 R Street, NW, Washington, DC 20009-1604 Association of American Colleges and Universities (2004) Taking Responsibility for the Quality of the Baccalaureate Degree: A Report from the Greater Expectations Project on Accreditation and Assessment Author, 1818 R Street, NW, Washington, DC 20009-1604 Association of American Colleges and Universities (2005) Liberal Education Outcomes: A Preliminary Report on Student Achievement in College Author, 1818 R Street, NW, Washington, DC 20009-1604 Association of American Colleges and Universities (2007) College learning for the new global century: Executive summary Author, 1818 R Street, NW, Washington, DC 20009-1604 Association of American Colleges and Universities (2009) The VALUE Project overview Peer Review, Winter 2009, 4-7 Driscoll, A & Wood, S (2007) Developing Outcomes-based Assessment for Learner-centered Education: A Faculty Introduction Sterling, VA: Stylus Ewell, P (2004) General Education and the Assessment Reform Agenda Association of American Colleges and Universities, 1818 R Street, NW, Washington, DC 20009-1604 Ferguson, M (2005) Advancing Liberal Education: Assessment Practices on Campus Association of American Colleges and Universities, 1818 R Street, NW, Washington, DC 20009-1604 Gaff, J (2001) The Academy in Transition: General Education in an Age of Student Mobility Association of American Colleges and Universities, 1818 R Street, NW, Washington, DC 20009-1604 Gaston, P.L & Gaff, J.G (2009) Revising general education – And avoiding the potholes Washington, DC: American Association of Colleges and Universities Halpern, D.F (1993) Assessing the effectiveness of critical thinking instruction The Journal of General Education, 42(4), 238-254 Hicks, S.J & Hubbard, A.M (February, 2009) Using course-based assessment to transform general education Paper presented at the AAC&U Conference on General Education, Assessment, and the Learning Students Need Baltimore, MD Humphreys, D (2006) Making the Case for Liberal Education Association of American Colleges and Universities, 1818 R Street, NW, Washington, DC 20009-1604 Kanter, S.L., Gamson, Z.F., & London, H.B (1997) Revitalizing General Education in a Time of Scarcity Boston: Allyn & Bacon Kuh, G D (2008) High-impact Educational Practices: What they are, who has access to them, and why they matter Washington, DC: AAC&U Kuh, G.D., Kinzie, J., Schuh, J.H., & Whitt, E.J., & Associates (2005) Student Success in College: Creating Conditions that Matter San Francisco: Jossey-Bass Lawson, T.J (1999) Assessing psychological critical thinking as a learning outcome for psychology majors Teaching of Psychology, 26(3), 211-213 Leskes, A & Milller, R (2005) General Education: A Self-study Guide for Review and Assessment Association of American Colleges and Universities, 1818 R Street, NW, Washington, DC 20009-1604 Leskes, A & Wright, B.D (2005) The Art and Science of Assessing General Education Outcomes: A Practical Guide Association of American Colleges and Universities, 1818 R Street, NW, Washington, DC 20009-1604 McMillan, J.H (1987) Enhancing college students’ critical thinking: A review of studies Research in Higher Education, 26(1), 3-29 National Survey of Student Engagement (2008) Promoting engagement for all students: The imperative to look within 2008 results Bloomington, IN: Indiana University Center for Postsecondary Research Pace, D & Middendorf, J Eds (2004) Decoding the disciplines: Helping students learn disciplinary ways of thinking New Directions for Teaching and Learning 98: Summer Paul, R & Elder, L (2008) The miniature guide to critical thinking concepts and tools Dillon Beach, CA: Foundation for Critical Thinking Press Renaud, R.D & Murray, H.G (2007) The validity of higher-order questions as a process indicator of educational quality Research in Higher Education, 48(3), 319-351 Renaud, R.D & Murray, H.G (2008) A comparison of a subject-specific and a general measure of critical thinking Thinking Skills and Creativity, 3(2), 85-93 Schlesinger, M.A (1984) The road to teaching thinking: A rest stop The Journal of General Education, 36(3), 182-273 Scriven, M & Paul, R (1987) Critical thinking as defined by the National Council for Excellence in Critical Thinking (cited at www.criticalthinking.org/aboutCT ) Shavelson, R.J (2007) A Brief History of Student Learning Assessment: How We got Where We Are, and a Proposal for Where to Go Next Washington: Association of American Colleges and Universities Stevenson, J.F & Scarnati, B.S (February 2010) Comparing approaches to the critical thinking dilemma Workshop presented at the AAC&U Conference on General Education and Assessment, Seattle WA Stevenson, J.F., Grossman-Garber, D., & Peters, C.B (March 2007) Assessing the core: Learning outcome objectives for general education at the University of Rhode Island Roundtable presented at the AAC&U Conference on General Education and Assessment, Miami, FL Terenzini, P.T., Springer, L., Pascarella, E.T., & Nora, A (1995) Influences affecting the development of students’ critical thinking skills Research in Higher Education, 36(1), 23-39 Wagner, T.A & Harvey, R.J (2006) Psychological Assessment, 18(1), 100-105 Williams, R.L., Oliver, R., & Stockdale, S (2004) Psychological versus generic critical thinking as predictors and outcome measures in a large undergraduate human development course The Journal of General Education, 53(1), 37-58 Table Institution-Wide Assessment Methods for Critical Thinking at the University of Rhode Island Assessment type Assessment method Control mechanism Nationally normed and standardiz ed NSSE, FSSE (items on “thinking critically” and “solving complex real-world problems”) Selected by U administration, administered by institutional research office Locally developed Student course evaluation (items on chance to practice cognitive tasks) Faculty committee for assessment of general education Nationally normed and standardiz ed CAAP institution-wide for samples of freshmen and seniors (Wabash Study) Selected by U administration with faculty input; administered by assessment office Locally developed Taxonomy of cognitive skills with expectation of differences by discipline Developed and administered by faculty committee Advantages of the method Disadvantages of the method Indirect External comparisons/ accountability Reliable psychometrics Self-reported views have been shown to be predictive Targets locally identified concerns, Senate-approved outcomes Not directly connected to local intentions Rarely discussed with faculty May lack pedagogical relevance, credibility May prove poorly designed External comparisons/ accountability Reliable psychometrics Elegantly conceptualized outcomes Local control of definitions Direct link to pedagogy Not directly connected to local intentions May lack pedagogical relevance, credibility Direct May prove poorly designed Easy to contest on psychometric grounds Figure All Should Have Prizes! Process: Curricular elements Pedagogical methods (e.g NSSE; local survey) Short-term outcomes: Rubric-assessed assignments Medium-term outcomes: Department-wide tests Long-term outcomes Generic national tests: e.g CLA, CAAP ... doing critical thinking and the inclination to use them Paul and Elder (2008) emphasized a humanistic theme in their definition: ? ?Critical thinking is that mode of thinking … in which the thinker... stated, ? ?Critical thinking is the intellectually disciplined process of actively and skillfully conceptualizing, applying, analyzing, synthesizing, and/or evaluating information gathered from, or... to decide what critical thinking is, and does that matter? As Table displays, at the University of Rhode Island many players have a bit part in assessing critical thinking The institutional research