Best Practices in Student Learning Assessment 2011-02-17

Best Practices in Student Learning and Assessment: Creating and Implementing Effective Assessment for NYU Schools, Departments and Programs Michaela Rome, Ph.D Assistant Vice Provost for Assessment and Evaluation Last updated: February 17, 2011 Executive Summary In this report, you will find: •Answers to frequently asked questions about student learning assessment •Descriptions of the step process to developing a student learning assessment plan including: o Step 1: Develop Student Learning Goals o Step 2: Develop Course Objectives and Outline Educational Opportunities o Step 3: Choose Measures and Methods to Assess Student Learning o Step 4: Use Results and Review Assessment Activities •Sample materials including: o Syllabus o Sequenced Assignment o Scoring Rubrics o Assessment Plan/Report Summary •Examples of direct versus indirect measures (evidence) of student learning The goals of the report are for readers to: •Understand the links among teaching, learning and assessment •Understand the difference between direct and indirect measures of student learning •Understand why course or assignment grades are not sufficient measures of student learning •Develop a basic understanding of the parts of an assessment plan •Understand the development and use of assessment rubrics •Gain knowledge regarding the options available for measuring student learning •Understand assessment reporting and how to use assessment results to improve educational opportunities It is expected that these materials, in conjunction with one-on-one consultations regarding assessment practices, will provide faculty and administrators with the tools to accomplish the following: •Communicate the knowledge they have gained regarding assessment practices to their colleagues •Disseminate best practices information to their school, department or program •Effectively promote the implementation of assessment practices in their school, department or program •Develop and implement an assessment plan (including a feedback mechanism) for their school, department or program •Implement a reporting cycle for assessment in their school, department or program i Acknowledgments These materials were initially compiled for a subcommittee report for the NYU Task Force for Best Practices in Student Performance Assessment (Fall 2009) and have since been expanded and updated to meet the needs of the NYU community Many thanks to Matthew Mayhew (Steinhardt School of Culture, Education, and Human Development) and Robert Squillace (Liberal Studies Program) for their comments on an initial draft of this document ii Table of Contents I Assessment FAQs .1 II Developing an Assessment Plan A Define the Context: Program Mission Statement B Step 1: Develop Student Learning Goals C Step 2: Develop Course Objectives and Outline Educational Opportunities D Step 3: Choose Measures and Methods to Assess Student Learning Measures 10 Methods 10 E Step 4: Use Results and Review Assessment Activities 11 Appendix A: Assessment versus Grading 12 Appendix B: Syllabus Template and Sample Syllabus 14 Appendix C: Assignment Template and Sample Sequenced Assignment 22 Appendix D: Rubric Basics 24 Appendix E: Sample Rubrics (Art, Pre-Professional, Social Science, Science, Humanities) .26 Appendix F: Direct and Indirect Evidence of Student Learning 33 Direct (Clear and Compelling) Evidence of What Students Are Learning 33 Indirect Evidence of Student Learning 33 Evidence of Learning Processes that Promote Student Learning 34 Appendix G: Range of NYU Assessment Activities 35 Appendix H: Sample Assessment Plan Summary for Existing and New Programs* 46 Appendix I: Questions for Brainstorming Objectives and Outcomes 48 Appendix J: Sample Exam Blueprint 50 Appendix K: Sample Guiding Questions for Peer Review 51 References 53 Tables and Figures Figure Steps in the Assessment Process Table Standards for Assessment Practices iii This report presents an overview of some best practices in student learning assessment and should be considered representative rather than exhaustive This review is meant to acquaint faculty and administrators with the basics of assessment and can serve as an introduction to developing an assessment plan for their program, department or school The Assistant Vice Provost for Assessment, Michaela Rome, is available for consultation with faculty and administrators who are undertaking development, revision or implementation of assessment plans She can be reached at 212-998-4426 or at michaela.rome@nyu.edu I Assessment FAQs What is assessment? Broadly speaking, assessment is a process that involves: Studying activities (courses, co-curricular events like lecture series, fieldwork, advising, etc.) that are designed to meet specific goals (in this case, learning goals) • Determining if goals are being met • Adapting activities (and possibly goals) as appropriate if goals are not being met • But my school/program/department already assesses student learning How is this process different? In the vast majority of cases, assessment is occurring in academic units; however, what is often lacking is formal, systematic documentation of assessment activities, assessment results and use of assessment results Furthermore, assessment plans not always include direct measures of student learning (see section II.D.1 and Appendix F) The Middle States Commission on Higher Education, NYU’s accrediting agency, now requires that schools conduct assessment that provides systematic, formal, and explicit evidence of student learning in order to meet accreditation standards While Middle States does not prescribe what or how to assess, there are guidelines regarding the types of evidence that are acceptable, (e.g., Middle States does not consider that grades provide sufficient evidence of student learning, see Appendix A) Why assess? What are the benefits? In addition to fulfilling external accountability requirements, assessment can be an internally valuable process The list below presents a range of benefits to students, faculty and administrators Students benefit because: The clear expectations that good assessment requires help them understand where they should focus their time and energy • Assessment, especially the grading/scoring process, motivates them to their best • Assessment feedback helps them understand their strengths and weaknesses Assessment information documents what they’ve learned; this documentation is beneficial in applying for jobs, awards and programs of advanced study • Graduate students who have assessment experience will have an advantage when applying for jobs • • Faculty benefit because: Assessment activities bring faculty together to discuss important issues such as what they teach and why as well as their standards and expectations for student learning • Assessment activities help faculty see how their courses link together to form a coherent program and how the courses they teach contribute to student success in their subsequent pursuits • Assessment creates a common language that engages faculty spanning a variety of specializations and disciplines • Administrators benefit because: Assessment information documenting the success of a program or institution can be used to convince employers, donors, legislators, and other constituents of its quality and worth (This benefits faculty and students too!) • Assessment can help ensure that institutional resources are being spent in the most effective ways possible - where they’ll have the greatest impact on student learning • Suskie, 2004, pp 11-12 What is the process for assessing student learning? Develop and implement an assessment plan for your school, department or program The Assessment Plan is the formal, explicit statement of how you will systematically assess student learning, including how you will collect, compile, share and use assessment results with the goal of improving educational opportunities (curriculum, instruction, academic supports) and student learning This report will focus on developing an Assessment Plan for program/departmental majors; however, the process can be adapted for schools and other categories of students (minors, study abroad students, service learning programs, etc.) What are the components of an assessment plan? What is the process for assessment plan development? The steps for developing an assessment plan are listed below and are described in greater detail in section II of this report Assessment plan development is an ordered process, as illustrated in Figure The foundation of an assessment program consists of the student learning goals upon which the curriculum is built and against which assessment measures are aligned A feedback loop connects the final step back to the beginning of the process The assessment plan is developed within the context of the stated program mission and includes (1) development of broad program goals, (2) development of specific course objectives (including educational opportunities: curriculum, instruction, academic supports), (3) identification of measures and methods for assessing student learning, (4) use of assessment results to improve educational opportunities and, if needed, to revise program goals (feedback loop) Figure Steps in the Assessment Process Rome, 2009 The broader context of school-level mission and goals is not discussed here; however, it is expected that there will be alignment between a program’s mission/goals and, broadly speaking, those of the school How I know if my program is meeting the standards for assessment practices? Beyond meeting the minimum standards, what are the “best practices” in student learning assessment? Table presents a rubric which describes the minimum standards for student learning assessment practices In addition to the minimum standards, best practices are also described Schools, departments and programs are encouraged to move beyond the minimum standards to implement these best practices Table can be used to gauge the current status of assessment practices in your school, department or program in relation to minimum standards and best practices As changes are made to assessment practices, Table can be used to track your progress Table Standards for Assessment Practices Assessment Practices Best Practice: In addition to meeting the described standard… Learning Goals Learning goals are clearly and actively communicated to students (on program website, handbook, etc.) and to faculty in the program Course Objectives Course objectives are clearly and actively communicated to students (on syllabi) and faculty in the program • Educational Opportunities (Curriculum, Teaching, Learning, Academic Supports) • • It is clear that every student in the major has ample opportunity to master each learning goal, either through multiple courses or through intensive study in one course Courses are completed by students in a hierarchical sequence in which skills to be mastered are presented from basic to intermediate to advanced Earlier skills are reinforced in subsequent courses Similarly, this ordered and iterative approach to teaching and learning occurs within individual courses Learner-centered instructional practices are employed (e.g., sequenced assignments, detailed assignment guidelines, discussion of sample papers, multiple drafts of papers, dissemination of expectations (rubrics) to students, etc.) Meets Standard Learning goals are described in explicit, observable terms, using action words, how students will be able to use their knowledge, what thinking skills and disciplinary dispositions they will have, and/or what else they will be able to upon completion of the program Course objectives are described in explicit, observable terms, using action words, how students will be able to use their knowledge, what thinking skills and disciplinary dispositions they will have, and/or what else they will be able to upon completion of the course Every student has sufficient opportunity to master each learning goal through completing at least one course in which the learning goal(s) are addressed • Assessment Methods Evidence is provided that the assessment methods yield truthful, fair information that can be used with confidence • • Use of Results Each assessment method clearly matches the learning goal being assessed and multiple assessments are used systematically (repeatedly, on a schedule) over time Assessments are conducted for all students in the major (or a representative sample) Assessment practices include direct measures of student learning Needs Attention Learning goals not meet the described standard Course objectives not meet the described standard Educational opportunities not meet the standard Assessment methods not meet the standard Standards have been established that clearly describe performance levels considered minimally adequate for Assessment results are shared and discussed with faculty Use of results students completing the program, and positive assessment teaching in the program and are used to modify learning does not results are shared with faculty, students, academic goals, teaching methods, curriculum, and/or assessment meet the administrators, prospective students, and other audiences as strategies, as appropriate standard appropriate Adapted from Suskie, L (2004) Assessing student learning: A common sense guide (2nd ed.) San Francisco: Jossey-Bass (pp 68-69) II Developing an Assessment Plan Why develop an assessment plan? In general, most endeavors are more successful if they are clearly outlined and planned in advance Assessment is no different A department’s (or school’s) assessment plan and resulting assessment reports can serve a number of functions: External representation of institutional memory Valuable information can be lost as members of the department change roles, go on sabbatical, move to another university, retire or simply not recall the challenges, successes, explanations for decisions, solutions to problems, etc that have occurred through the assessment process Assessment plans and reports document these processes for future members and leaders of the department Shared departmental vision An assessment plan allows all departmental members to share an understanding of the department’s assessment vision Faculty can comment on and question the plan from an informed standpoint Faculty are aware of how their courses and educational practices fit in with the rest of the curriculum and what their roles are with regard to assessment Resource for new and adjunct faculty An assessment plan is an efficient means of communicating a department’s assessment activities and educational practices to new and adjunct faculty These faculty not need to wait for a committee meeting nor they need to rely on piecemeal information which may leave them with an incomplete or inaccurate depiction of the department’s assessment activities Sharing best practices Departments can share their assessment plans with each other and, in doing so, share successful approaches to assessment, creative solutions to overcoming obstacles to assessment, innovative changes made to curriculum and instruction to improve student learning, etc External audiences An assessment plan demonstrates to accrediting and funding agencies, parents, students and others that the department has thought through the assessment process and is committed to assessing student learning and to improving the teaching and learning process in the school or department Assessment reports document evidence of student learning as well as the improvements that have been made to educational opportunities A Define the Context: Program Mission Statement The mission is a broad statement of purpose that can guide faculty decision-making in designing a focused and coherent program of study and allows students to determine if the program is aligned with their educational and post-graduation goals The mission statement may articulate the purpose, philosophy, and values of the program, as well as identify the approaches used to fulfill the mission Mission statements should be succinct, accurate, clear, and realistic and should reflect input from program faculty Mission statements should be revisited for accuracy and relevance as often as appropriate for the discipline Below are several examples of statements that might be included as part of a mission statement: Faculty Review The faculty's performance is assessed annually by the faculty merit committee The committee judges each full-time faculty member's teaching effectiveness, as well as his/her service to the faculty and publication record during the past year Adjuncts' teaching effectiveness is judged at the end of every semester by the department's adjunct liaison The primary means of judging teaching effectiveness is via the departmental course evaluation forms New faculty members and adjuncts get extra scrutiny with a mid-semester evaluation and, typically, a class sit-in by a full-time faculty member Curriculum Reviews Curricula of all courses are regularly scrutinized and refined to enhance learning experiences, incorporate new concepts or teaching techniques, and to achieve the Department’s educational goals We have begun a process whereby each semester, the syllabi for undergraduate courses are collected and checked by the Directors of Undergraduate Studies Syllabi are examined to ensure that textbooks are up to date, that the prescribed meeting dates and times for the entire semester are scheduled, and that grading criteria are in place When deficiencies are seen, the faculty member is contacted, and changes are recommended We will be instituting standardized syllabi regulations for the Department in fall 2009 Curriculum Reviews During the previous year the department arranged for an ad hoc committee to be formed in order to evaluate the contents of individual courses, the required prerequisites for the courses, and the overall structure of the undergraduate program This internal review was supplemented by a critical assessment of our procedures by the Chair of Economics at Yale University Curriculum Reviews Curricula of all courses are regularly scrutinized and refined to enhance learning experiences, incorporate new concepts or teaching techniques, and to achieve the Department’s educational goals Each semester, the syllabi for undergraduate courses are collected and checked by the Director of Undergraduate Studies Syllabi are examined to ensure that textbooks are up to date, that the prescribed meeting dates and times for the entire semester are scheduled, and that grading criteria are in place When deficiencies are seen, the faculty member is contacted, and changes are recommended We will be instituting standardized syllabi regulations regarding the clarity of course objectives and grading for the Department in Fall 2010 External Comparisons/ Benchmarking The Undergraduate Program Committee periodically reviews requirements and curricula at peer institutions This information is used to revise courses and requirements, as needed Also, based on discussions with alumni, as well as professional, industrial, and academic contacts, we fine tune our courses to ensure that the students are learning about technology that is relevant and up to date External Comparisons/ Benchmarking The Undergraduate Program Committee periodically reviews requirements and curricula at peer institutions (most recently in 2006-2008, in a review of Literary Interpretation) This information is used to revise courses and requirements, as needed External Accreditation Review The ACEJMC requires the department to undergo accreditation review every six years The latest full review occurred in 2005 and resulted in renewed accreditation of the department Survey and Interview for Declared Majors and Seniors The department developed an exit survey and interview for graduating majors Each pending graduate is asked to fill out the survey and discuss it with his/her advisor prior to commencement For seniors in their final semester, we gather data, which includes information about acceptance to professional school and graduate school, job acquisition, future addresses and email contacts so that the department can keep in touch with the graduates, and other statistical information, including standardized test scores We also ask students to share their feelings about their undergraduate experience in the department, both positive and negative Because the exit interview and survey are optional, the Department is unable to collect data from all graduating seniors In an effort to obtain more comprehensive information to inform educational improvements, the Department has begun interviewing all declared majors These interviews take place each semester and occur during majors’ course registration meetings with their advisors Each semester we ask students to (1) provide information about current research they are doing and (2) to provide feedback to the department about the educational experiences in biology courses (see Appendix C) As students are free to decline to participate in this process, we were pleased that we obtained a 99% response rate for the fall 2008 survey administration Survey of Declared Majors and Seniors The department developed an exit survey for graduating majors Each pending graduate is asked to fill out the survey For seniors in their final semester, we gather data, which includes information about acceptance to professional school and graduate school, job acquisition, future addresses and email contacts so that the department can keep in touch with the graduates, and other statistical information, including standardized test scores We also ask students to share their feelings about their undergraduate experience in the department, both positive and negative Because the exit interview and survey are optional, the department is unable to collect data from all graduating seniors Student evaluation of courses/instruction Course evaluations prepared by the College of Arts and Science Student Council are used in each of our majors’ courses, and the results are reviewed each semester The departmental chair, the department’s director of undergraduate studies, and the specific course instructor review the results In addition, many of our faculty use their own evaluation forms in order to obtain feedback concerning issues specific to their course Some of these ask for ratings and opinions on a topic-by-topic basis within the class We implement course changes based on patterns that emerge from the student reviews Formative Course Assessment We use assessment at various points in our large courses, rather than merely at the end The Department has a standard form with questions that cover the lecture material and its accessibility, assessment of readings and text, and the quality of the lectures Formative assessment is used mostly in our large, introductory biology course, which is team taught Typically, we use scantron sheets so that the data can be rapidly collected Each instructor reviews results from his or her course and acts on them as warranted to improve students’ educational experiences Formative Course Assessment In addition to the end-of-semester evaluation, for all new instructors, or those teaching a course for the first time, we conduct a “pre-evaluation.” That is, we distribute the economics department course evaluation form in the third or fourth week of the semester These are reviewed by the Director of Undergraduate Studies (DUGS), and returned to the instructor Our goal is to catch problems in instruction as early as possible On numerous occasions, we have been able to detect a problem and taken steps to improve the quality of the instruction while the semester is still in progress Remedial steps include meetings with the DUGS to discuss teaching strategies and problems, having a more experienced instructor sit in on the class and provide feedback, and – on rare occasions – reassignment of the course Formative Course Assessment Our first goal in assessment is to verify that our courses are taught well Students evaluate instructors and Teaching Assistants (TAs) (who teach review sessions) at the fourth week of every Fall or Spring semester, and the Dean of Undergraduate Studies (DUS) and Associate DUS go over the evaluations, looking for problems with courses while there is still time to correct them The evaluations are returned to the instructors and TAs within a week or two Joint Student-Department Meetings Student feedback is collected during meetings of the department’s undergraduate program committee (which includes student members) The undergraduate program committee meets with student members times per year Student Club Meetings Students in the English and Dramatic Literature Organization (majors club) provide the department with feedback via discussions with the faculty advisor of the club several times per year Student Club Meetings We have a strong Psychology Club and an honors fraternity (Psi Chi), and both are in close contact with the DUS and Associate DUS The Psychology Club actively monitors and critiques undergraduate courses, selecting student representatives to evaluate teaching and learning Program Evaluation by Graduating Majors For the past three years, each graduating German major has been encouraged to fill out a Program Evaluation (see Appendix) The purpose of this evaluation is to collect feedback on students’ experiences of the major, points of satisfaction and dissatisfaction, and suggestions for improvement These anonymous evaluations are shared with the four core members of the Department – the Chair, the DUGS, the DGS, and the Director of Language Programs In the past, we have always discussed these evaluations, but we have not formalized a procedure for tabulating and potentially acting on student suggestions However, one of the results of our discussions of these evaluations was the recognition that single author courses, where an eminent author’s work is presented in the historical and artistic context of production is not only desired by the students but optimally fulfills the objectives of this more advanced level of scholarly inquiry in German We therefore now regularly offer such single author courses (in the past: Kafka, Nietzsche, Brecht, in the Fall Rilke) Beginning Spring 2011, we will form a committee, which will discuss these evaluations (including the ones from the past three years), draw up a report and discuss possible implementation of student suggestions if we feel they will enrich the program Independent Study Independent study is a research experience in which a student works in a faculty laboratory Students must have a GPA greater than 3.0 in the major to enroll Assessment is based on agreement between the faculty mentor and the student These criteria are then used by the faculty mentor to evaluate and assess student progress We are currently creating a specific evaluation form that will allow the student to provide feedback on his/her experience Performance on the MCAT The Department collects MCAT scores from the Office of pre-professional development and compare these to the national average Advanced Placement Credit A score of or on the AP Psychology or Statistics exam counts as fulfillment of Introductory Psychology or Statistics requirements, respectively Employment and Graduate School Placement The department tracks graduates to determine the number of students who have been employed in professional settings as well as those who have been accepted into master and doctoral programs via email survey Job Placement Rates According to our point person for alumni relations, we have some anecdotal data, and those data are stronger for our graduate programs than for our undergraduate program We have recently begun attempts to calculate job placement rates more formally In early May 2008, our internship coordinator administered an e-mail survey to graduates Unfortunately, the response rate has been insufficient to gauge job placement rates with reasonable confidence We are attempting to create a Microsoft Access database to make data collection and analysis much easier Advising All biology majors are required to meet at least once a semester with their advisors, who are members of the full-time faculty During that meeting, students have the opportunity to discuss their coursework, register for classes, and discuss career goals Advisors access the student’s electronic, on-line record, which provides a transcript, mid-term grades, and an audit that indicates courses that must yet be taken, GPAs, and other degree requirements The advisor reviews the student’s past and current performance, identifies strengths and weaknesses based on performance, and discusses future courses The advisor also monitors the additional requirements for graduation, including those for the general education program Finally, the advisor has a conversation with the student about any general concerns and career issues Faculty can elect to provide advice, send the student to the Director of Undergraduate Studies, or recommend that he or she visits the pre-professional advising office or other networks in the College Faculty try to be cognizant of emotional or health-related problems; in such cases, the advisor will recommend the Wellness Center or contact the appropriate office to seek assistance on behalf of the student Advising The department has a staff that is dedicated to advising economics majors and prospective economics majors Some of the advisors are members of the full-time faculty; others are nonfaculty advisors The non-faculty economics advisors are very well-versed in the rules and regulations of the major They regularly meet with the DUGS to discuss departmental requirements and student concerns The non-faculty advisors are very experienced in dealing with advising tasks that are relatively mechanical These advisors help students declare their major, register for courses, ensure that students are aware of the prerequisites for courses, provide very general advice about the requirements for the major, and review a student’s progress to ensure that they are on-track with completing their major on time If there is an advising situation that involves special circumstances or more academic or career counseling, then the non-faculty advisors refer the student to the appropriate faculty advisor Each faculty advisor specializes in a certain field For example, one deals with students interested in taking economics courses abroad, another deals with transfer students and students who wish to take economics courses within the United States, and a third advisor deals with students who are interested in taking courses at other schools and departments within NYU This division of labor ensures that students who receive an unfavorable decision from one advisor cannot then go to another advisor in the hopes of obtaining a more favorable decision However, students can always appeal to the DUGS The faculty advisors meet regularly to discuss student issues and to ensure that everyone is broadly applying the same rubric in making decisions Both the faculty-advisors or the non-faculty advisors can choose to send the student to the Director of Undergraduate Studies or recommend that the student visit other advising networks in the College, such as the pre-professional advising office Students are urged to inform the economics advisors and/or the DUGS about any problems they face with a particular instructor or course, so that the DUGS can deal promptly with the problem Advising All English majors are asked to meet at least once a semester with their advisors, who are members of the full-time faculty During that meeting, students have the opportunity to discuss their coursework, register for classes, and discuss career goals Advisors access the student’s electronic, on-line record, which provides a transcript, mid-term grades, and an audit that indicates courses that must yet be taken, GPAs, and other degree requirements The advisor reviews the student’s past and current performance, identifies strengths and weaknesses based on performance, and discusses future courses The advisor also monitors the additional requirements for graduation, including those for the general education program Finally, the advisor has a conversation with the student about any general concerns and career issues Faculty can elect to provide advice, send the student to the Director of Undergraduate Studies, or recommend that he or she visits the pre-professional advising office or other networks in the College Faculty try to be cognizant of emotional or health-related problems; in such cases, the advisor will recommend the Wellness Center or contact the appropriate office to seek assistance on behalf of the student Advising All majors are asked to meet at least once a semester with their advisors, who are members of the full-time faculty During that meeting, students have the opportunity to discuss their coursework, register for classes, and discuss career goals Advisors access the student’s electronic, on-line record, which provides a transcript, mid-term grades, and an audit that indicates courses that must yet be taken, GPAs, and other degree requirements The advisor reviews the student’s past and current performance, identifies strengths and weaknesses based on performance, and discusses future courses The advisor also monitors the additional requirements for graduation, including those for the general education program Finally, the advisor has a conversation with the student about any general concerns and career issues Faculty can elect to provide advice, send the student to the Director of Undergraduate Studies, or recommend that he or she visits the pre-professional advising office or other networks in the College Faculty try to be cognizant of emotional or health-related problems; in such cases, the advisor will recommend the Wellness Center or contact the appropriate office to seek assistance on behalf of the student Advising All international relations majors meet regularly with the department’s undergraduate advisor, Emily Mitchell-Marell, who closely monitors students’ progress through the program and helps with registration The advisor accesses the student’s electronic, on-line record, which provides a transcript, mid-term grades, and an audit that indicates courses that must yet be taken, GPAs, and other degree requirements The advisor reviews the student’s past and current performance, identifies strengths and weaknesses based on performance, and discusses future courses The advisor also monitors the additional requirements for graduation, including those for the general education program Ms Mitchell-Marell communicates regularly with the Director of the IR Program on students who are having difficulties meeting program requirements or who need help resolving specific issues The Director subsequently meets with some students on an as-needed basis Students are also encouraged to meet at least once a semester with other members of the full-time faculty During that meeting, students have the opportunity to discuss their coursework and discuss career goals Faculty can elect to provide advice, send the student to the Director of Undergraduate Studies, or recommend that he or she visits the pre-professional advising office or other networks in the College The advisor and faculty try to be cognizant of emotional or healthrelated problems; in such cases, they will recommend the Wellness Center or contact the appropriate office to seek assistance on behalf of the student Mentoring The small size of our major program and the correspondingly low ratio of students to faculty create the possibility for very close mentoring relationships between faculty members and individual students in the major Faculty in the undergraduate program are committed to making themselves fully available to students and to engaging in close advising and mentoring This approach is emphasized as policy in the department The extent to which students in our major take advantage of these opportunities is important to us and serves as an indicator of the overall effectiveness and appeal of our major program Required minimum Majors must complete required courses with a grade of C or higher and must attain a minimum GPA of 2.0 for all required courses and electives taken in the major Faculty advisors monitor student performance, and those who are in academic jeopardy are also referred to the Director or Assistant Director of Undergraduate Studies Students are permitted to re-take a course that they not successfully complete, but such attempts are only permitted once for each course If a grade of C or higher is earned when the course is re-taken, the student may remain in the major If the grade is below a C, the student is advised to change majors In these latter case, the student nearly always elects to change majors voluntarily However, in cases where academic performance continues to be poor, dismissal from the major can occur via a formal letter from the Director of Undergraduate Studies and the Departmental Chair Course grading criteria and required minimum grades Our program has a department-wide grading rubric, but it is a suggestion for professors rather than a requirement Students are assessed on their quality of work, attendance, class participation, assignment completion, progress, and ability to meet course objectives (see Table 2) Students must earn a minimum of a C to receive departmental credit for a course Many of our courses have prerequisites, and a student cannot take a course offering until all the prerequisites are satisfied with a minimum of a C Most notably, students cannot progress in the Foundations/Inquiry/Beat/Advanced Reporting sequence until each prior course in the sequence is passed with a C or above Given the importance of strong writing skills in this curriculum, students not achieving a B average are directed to see the department’s undergraduate adviser Appendix H: Sample Assessment Plan Summary for Existing and New Programs* I Goals* What will students learn? Think critically, creatively, and independently Conduct research and evaluate information by methods appropriate to the field Write correctly and clearly in forms and styles appropriate for the field, audiences, and purposes being addressed [Goal 4] [Goal 5] [Goal 6] II Objectives/Outcomes III Educational Opportunities What, specifically, will students know or be able to do? Courses with this outcome as primary objective Demonstrate originality and creativity in developing ideas and drawing conclusions V00.0001 Courses that reinforce this outcome V00.0012 V00.0004 V00.0016 Respond analytically and critically to ideas and texts, synthesize relevant materials, and formulate an original argument Analyze and draw evidence from both primary and secondary texts Effectively use library resources and document sources Produce academic writing that demonstrates ability to develop a thesis, use specific examples or evidence, and draw well-founded conclusions V00.0025 How will you measure each of the goals and objectives in columns I and II? Treat writing as a process of draft and revision; work extensively with individual students, both via written feedback and in person Facilitate collaborative work settings, including peerediting Offer tutoring for students who need or desire extra help Provide library workshops on using databases and other research tools * The exact number of goals will depend on your department, but 4-7 is a general guideline IV Assessment Measures How will you accomplish each goal? Assessment of student writing via rubric Student Course Evaluations Student focus groups Senior Survey Senior exit interviews Syllabi and curriculum review Sample Assessment Report Summary for Existing and New Programs* I Goals What will students learn? [This particular year, the department addressed of its goals] Think critically, creatively, and independently Conduct research and evaluate information by methods appropriate to the field Write correctly and clearly in forms and styles appropriate for the field, audiences, and purposes being addressed V Results What are the findings from Column IV? VI Possible Explanations Why might these results have occurred? 20% of students scored “unacceptable” and 30% scored “developing proficiency” on rubric criterion “coherence of argument” Students not being asked to produce sufficient writing to practice their skills Student evaluations of writing portion of classes indicates confusion and frustration Students not receiving enough “actionable” feedback on assignments Students not aware of standards Review of syllabi revealed broad variation in amount, type, and standards of writing Senior Survey reveals that students in our department feel that their writing skills have not been enhanced to as great a degree as in other departments (50% vs 70%, respectively) VII Action Taken How have you used the findings from the assessments? What improvements or changes have been made based on assessment findings? Revised common guidelines for writing standards Implemented faculty workshops to discuss writing assignments and agree upon common goals and share “best practices” strategies Discussed examples of poor, proficient, and superior writing in class Faculty/students engaged in question/answer session to help clarify expectation Created repository of writing assignments online for faculty Collaborated with NYU Libraries to provide additional on-site instruction on library resources Expanded writing tutorial support *Assessment plan, measurement tools, and full assessment report (if applicable) attached Appendix I: Questions for Brainstorming Objectives and Outcomes http://wac.colostate.edu/llad/v6n1/carter.pdf • • • • • • • • • • • • • • • • Imagine an ideal graduate from your program What kinds of skills, knowledge, or other attributes characterize that graduate? What is it that attracts students to this program? What value does this program offer a student? How you know whether your students possess the kinds of abilities, knowledge, skills, and attributes you expect of them? What kinds of assignments or other activities people in this program use to encourage the kinds of abilities, knowledge, and skills you have identified? What is it that distinguishes this program from related programs in the university? Is there anything about your program that makes it stand out from other similar programs? What kinds of research methodologies are people in this field [program] expected to perform? Oftentimes, disciplines [programs] are defined by ways of thinking What does it mean to think like a person in this discipline [program]? What kinds of jobs students in this field generally take? What kinds of skills are appropriate to jobs in this field? How you know whether students possess those skills? What advantages does a student in this program have on the job? What sorts of speaking and writing professionals in this field on the job? What sorts of speaking and writing students in their classes? Are there any particular types of communication that people this field [program] are expected to master? Drafting objectives and outcomes The next, and perhaps the most challenging, step is to use the notes to draft objectives and outcomes that the program faculty will readily see as reflective of their own program This means identifying the broader values or goals, which could become objectives, and the detailed information about each of those goals, which could become outcomes One way of doing this is to: a) b) c) d) e) f) type up and print the notes while the conversation is fresh and it is still possible to elaborate where the notes may be sketchy; read the printed notes several times, at first just to get a sense of the whole and then to search out superordinate ideas or themes: broad concepts that emerged from the conversation, ideas that are repeated, points that faculty members particularly emphasized, key words or phrases that keep coming up, etc.; mark the themes in the text of the notes and make a list of them, eliminating all but the ones that seem to be most important to the faculty; rearrange the electronic version of the notes to create a rough thematic outline consisting of the themes and under each theme the subordinate ideas that are attached to it and define it in more concrete terms; draft formal objectives by starting with a heuristic sentence opener such as, “Graduates of the Department of X should be able to demonstrate that they can: …” and rewriting each objective, i.e., each theme, as the completion of the sentence; draft the outcomes for each objective also by starting with a sentence opener such as, “Specifically, graduates should be able to demonstrate that they can: …” and completing the sentence by incorporating, wherever possible, concrete verbs used by the faculty to indicate what students should be able to do—to describe, to analyze, to critique, etc (when in doubt, Bloom’s taxonomy provides a good source for such verbs) Appendix J: Sample Exam Blueprint Departmental Goals Exam Items Student Learning Outcomes (based on course objectives): Calculus I Derivatives Antiderivatives 2 Etc integrals of functions of one real variable 10 Students will develop understanding of foundational concepts in calculus 11 Trigonometric functions 12 13 inverse trigonometric functions Logarithmic Exponential functions 14 15 16 17 18 Graphing 19 Maximizing and minimizing functions 20 21 Areas Volumes Possible Points 22 23 24 Total Points Percent of Exam 14 15% Appendix K: Sample Guiding Questions for Peer Review Review Questions: Please answer thoroughly and write clearly Reviewer name: _ Author’s name: Identify the program goals/ objectives Are they clear and realistic? Are they appropriate given the program activities? List the concepts/ variables that are being evaluated (e.g., for example, one of the KASA-B variety) and note the author’s operational definition of each variable Are the research questions SMART? If not, what is lacking? Is there a counterfactual for each research question? Identify the measure proposed by the author for each variable Is the scale appropriate for each? Is the alternative proposed measure appropriate? Does the author clearly describe how s/he will assess reliability and validity of one of the measures? If not, what is missing? List the strategies that the author proposes Has the problem been thoroughly described? If not, what’s missing from the description? How could the author expand the discussion (if needed)? Are you convinced (based on what is written in the memo) that there is a need? Based on what is described, you have a clear idea of what the program is? Be specific, describe it here or make notes on the author’s paper Write down any questions you have about the program components and/or functioning Does the author clearly state the implicit and explicit assumptions on which the program theory is based? References Grunert O’Brian, J, Millis, B.J & Cohen, M.W (2008) The course syllabus: A learningcentered approach San Francisco: Jossey-Bass Middle States Commission on Higher Education (2007) Student learning assessment: Options and resources http://www.msche.org/publications/SLA_Book_0808080728085320.pdf (retrieved November 12, 2009) Stevens, D.D & Levi, A.J (2005) Introduction to rubrics: An assessment too to save grading time, convey effective feedback and promote student learning Sterling, VA: Stylus Publishing Suskie, L (2004) Assessing student learning: A common sense guide (2nd ed.) San Francisco: Jossey-Bass ... Indirect Evidence of Student Learning 33 Direct (Clear and Compelling) Evidence of What Students Are Learning 33 Indirect Evidence of Student Learning 33 Evidence of Learning. .. speaking, those of the school How I know if my program is meeting the standards for assessment practices? Beyond meeting the minimum standards, what are the ? ?best practices? ?? in student learning assessment? ... Summary In this report, you will find: •Answers to frequently asked questions about student learning assessment •Descriptions of the step process to developing a student learning assessment plan including: