Vanderbilt Center for Teaching - Bryant and Rowe

Content Knowledge, Skills, and Attitudes Gained in the Teaching Certificate Program at Vanderbilt University - A Program Evaluation - Prepared for the Center for Teaching Vanderbilt University Mark Bryant and Christopher Rowe Peabody College for Education and Human Development May 2008 Table of Contents Executive Summary Introduction and Problem Statement Graduate Student Preparation 11 Trends Impacting Graduate Student Preparation Future Faculty Programs The Scholarship of Teaching Comparison of Similar Programs 24 University of Colorado – Boulder University of California – Santa Barbara University of Michigan Princeton University Michigan State University Summary of Programs Teaching Certificate Program Description 33 Assessment Methods 45 Document Analysis of Participant Knowledge, Skills and Attitudes54 Qualitative Assessment of Participant Knowledge, Skills and Attitudes 63 Basic Information Content Knowledge Program Perceptions Departmental Expectations Departmental Values Regarding Graduate Student Socialization 81 College of Arts & Science School of Engineering Peabody College for Education and Human Development Medical Center Divinity/Graduate School of Religion School of Nursing Observations High/Low Consensus Field Comparison Limitations to the Project 111 Conclusions 115 Recommendations 122 Closing Remarks on the Future of the Teaching Certificate Program 130 Bibliography 132 Appendices: A-H 135 List of Appendices Page A: Document Analysis Rubric 135 B: Participant Email Letter 136 C: Participant Interview Protocol 137 D: Participant Matrix 139 E: Department Listing of Directors of Graduate Studies Interviewed 148 F: Stakeholder Email Letter 149 G: Stakeholder Interview Protocol 150 H: Stakeholder Matrix 152 List of Figures Page Figure MSU Framework of Core Competencies 29 Figure Diagram of Teaching Certificate Program 37 Figure Program Enrollment by Stage 42 Figure Program Enrollment by School 42 Figure t-Test Results 55 Executive Summary The program outcomes However, one element that sets evaluation is to provide the Center for the Vanderbilt program apart from similar Teaching at Vanderbilt University with a programs is the required project that comprehensive evaluation of their Teaching highlights the Scholarship of Teaching and Certificate program, which is in its third year Learning (SoTL) The three cycles of inquiry, of existence As a joint project between the experimentation, and reflection increasingly Graduate School and the Center for emphasize teaching as a scholarly activity as Teaching, Vanderbilt’s Teaching Certificate defined by Boyer as one of four domains of program aims “to help graduate students, scholarship professional students, and post-doctoral seeks to answer three questions fellows develop and refine their teaching questions are: skills through three cycles of teaching What participants learn in the program, including knowledge, skills, and attitudes? How they apply what they learn when teaching at Vanderbilt or in faculty positions obtained after leaving Vanderbilt? What knowledge, skills, and attitudes regarding teaching Vanderbilt departments and programs want their doctoral students to possess upon graduation? activities, purpose each of this consisting of inquiry, experimentation, and reflection phases.” (Vanderbilt University) Much like other teacher preparation programs described in the body of this document, the Teaching Certificate program combines workshops, teaching observation and Specifically this evaluation The feedback experiences, reading groups, a literature The deliverable for this project is an review, and reflective essays to achieve its assessment of student learning and of the program’s strengths and weaknesses in information order to give the Center for Teaching useful perceptions information to improve the program and preparation for their students as well as the thereby improve the experience for the departments’ actual efforts or lack thereof participants This project consists of two in preparing their graduate students for phases: teaching responsibilities which they may participant analysis stakeholder analysis and The participant about stakeholders’ of the value of teaching encounter as a faculty member analysis stage primarily addresses the first For the participant phase, the two questions stated above and focuses on investigators created an evaluation rubric in documents and interviews with actual order to examine program documents This program participants, both those currently rubric operationalized four of six stated in the program as well as the few who had program objectives completed all requirements During the program objectives not evaluated rely on external “end-of-pipeline” analysis of participant stakeholder analysis phase, stakeholders defined as of performance once they have obtained full- were time employment after graduation Given interviewed in order to identify skills, that there are very few program finishers abilities, and attitudes that they deem as who have graduated from the university important for their graduate students This and moved into faculty roles, these two phase specifically program objectives were not assessed The addresses the third question stated above rubric created to assess the four program Questions objectives employed a 5-point scale and Graduate of Study the were at Directors The remaining two Vanderbilt evaluation used that elicited was used to gauge the acquisition of to identify expectations and attitudes of knowledge, skills and attitudes of teaching graduate preparation for teaching as a scholarly activity This quantitative It is important to note that this approach allowed the investigators to program evaluation contains important assess the magnitude of knowledge gained limitations that stem from the lack of regarding the four program objectives being program evaluated Based on the results of the operationalized objectives, and the open- document analysis a common interview ended electronic portfolio reporting system protocol was developed in consultation with In addition, there were inconsistencies in the Center for Teaching in order to extract documentation from one participant to more information than was obtained from another which probably impacted the level the document analysis of reliability with the outcomes This qualitative approach sampled participants from each of finishers, Three of the the four lack of objectives the three cycles and interviewed them in evaluated in the quantitative analysis order to establish how participant content suggest an increase in knowledge, skills, and knowledge increased attitudes of participants' learning with program The second phase of this evaluation, the throughout stakeholder the regard to the following: undergraduate analysis, learning, analysis of their own teaching, and sampled Directors of Graduate Study from engagement with their own teaching in a across the campus and interviewed them community of scholars using a common interview protocol in order objective, which is primarily a Cycle With the fourth activity, participants showed no significant difference at the end of Cycle in teaching at Vanderbilt approaching their own teaching as a activities on training graduate students to scholarly activity Based on these results, teach vary widely but conform somewhat to further information needed to be acquired trends in high and low consensus fields by way of qualitative analysis to determine The applied and natural sciences and some if the initial results were an accurate social sciences tend to focus more training representation of the participants change on research skills while many humanities Based on results from the participant areas devote more resources to teaching in interviews, graduate students appear to addition to that of research skills However, fulfill the four program objectives evaluated many departments base their success on and are able to approach their teaching as a student placement after graduation with a scholarly activity and learn from their own high desire to be at research-intensive teaching universities, and from others' teaching even Departmental though many Participants self-report that they have departments see their students at teaching- gained knowledge and skills from their intensive institutions or in industry participation in the program and the In conclusion, the investigators analysis of data demonstrates an increase in determined that there was a substantial knowledge, skills, and attitudes as they increase in knowledge, skills, and attitudes relate to teaching as a scholarly activity in the scholarship of teaching and learning From interviews with the by program participants Participant stakeholders, there is sufficient evidence of experience tends to vary widely depending tension that exists between research and on the department attitudes regarding teaching as evidenced from participant community at large A concern, however, is interviews and DGS interviews The role of that the DGS tends to be marginalized in many important departments as a service duty rather than a participant learning Using operationalized professional role dedicated to strengthening objectives, variables should be evaluated on graduate student education in research and the basis of how participants are gaining teaching As a result, it is important that knowledge, skills, and attitudes throughout the Center for Teaching be relied upon to the fulfill the need for training in pedagogy in management threatens the significance of order to fully prepare doctoral students for assessment professional employment Recommendations In addition, of data management, for quality program which evaluation Inadequate in this resulting is of data program from this teaching opportunities should be increased program evaluation are listed below in many departments in order to provide Program objectives should be operationalized in order to provide consistent evaluation of the increase in participant knowledge, skills, and attitudes When prompting participant reflection, the use of leading questions in the portfolio allows for more consistent reporting of outcomes in the various program cycles, which leads to greater validity when evaluating participant performance Possibly have participants revise their teaching statements more regularly than just at the end of the program, which should integrate their statements with every teaching activity undertaken graduate students with more substantive experiences in teaching to enhance their profile as employment they seek professional In comparison to other similar programs across the country, the Teaching Certificate program is on a positive trajectory to establish best practices in educating and evaluating teaching as a scholarly activity for the higher education Participants seem to enjoy a great deal of structure in the schedules, thus the use of soft deadlines or typical times to complete tasks can aid in efficient time management It is critical to effectively track the progress through the program in order to measure gains effectively, thus having students regularly self-report progress ensures accurate record keeping The portfolio system is clearly critical to evaluate participant progress, which justifies having a simple yet sophisticated system to handle selfreporting, tracking, and evaluation of participant activities Stakeholders are important to the continued success of the program and key faculty should be identified and approached as supporters of the program Additionally, DGS's deemed potential supporters should be welleducated on the program in an effort to continue to have a stream of applicants who become participants A possibility could be to modify the participants' academic transcripts to note this significant accomplishment and to add credibility to the program and its participants with regard to SoTL The CFT should spearhead a concerted effort in partnership with the Graduate School to integrate teaching into the overall graduate student experience considering that so many end up in teaching positions 10 The population of post-doctoral fellows is increasing and becomes an area of interest for gaining program participants, thus marketing efforts should be increased to this demographic 11 A major benefit of this program is the 'high touch' approach to participant activities and this high level of service should be continued 12 More consistent program evaluation is important to maintaining this important and critical program to the graduate student experience at Vanderbilt The next formal evaluation should occur when more participants finish the program and gain full-time faculty employment in order to evaluate the two program objectives not assessed in this study The program has a strong foundation on which to build, and the ongoing efforts of the Center for Teaching staff to improve the program will no doubt make it a leader in its field and a model program for other institutions to emulate Introduction and Problem Statement The at of the university community, and what Vanderbilt has designed a program for the participants are actually learning through institution’s graduate/professional students their and post-doctoral fellows to assist students Specifically, the Center for Teaching posed with developing their skills as teachers the following questions: Unlike traditional graduate professional What participants learn in the program, including knowledge, skills, and attitudes? How they apply what they learn when teaching at Vanderbilt or in faculty positions obtained after leaving Vanderbilt? What knowledge, skills, and attitudes regarding teaching Vanderbilt departments and programs want their doctoral students to possess upon graduation? preparation Center for programs, Teaching the Teaching Certificate program focuses on training in pedagogy Currently, the Center maintains data on participation and satisfaction; however, data on participants’ knowledge gains and skill development is minimal Information on what participants have of program program’s strengths and weaknesses in well as how well those developments match outcomes the assessment of student learning and of the knowledge and skills they have acquired, as intended in The deliverable for this project was an learned and how they are using the the participation order to give the Center for Teaching useful their information to improve the program and departments and programs is lacking The thereby improve the experience for the aim of this project was to assess how well participants This project consisted of two the Teaching Certificate program is meeting phases: its goals, how consistent it is with the needs participant analysis and Appendix E, Listing of Directors of Graduate Studies Interviewed College of Arts & Science School of Engineering Anthropology Economics English History Mathematics Philosophy Political Science Psychology Spanish & Portuguese Sociology Biomedical Engineering Chemical Engineering Environmental Engineering Mechanical Engineering Peabody College Medical Center Human & Organizational Development Leadership, Policy, and Organizations Special Education Teaching and Learning Psychology and Human Development Biochemistry Biomedical Informatics Cancer Biology Hearing and Speech Science Pharmacology Divinity/Graduate Department of Religion School of Nursing Appendix F, Stakeholder Email Letter Dear Professor _: We are writing to ask you to participate in a brief interview about the doctoral program in your department We are surveying Directors of Graduate Study across the university as part of a project that we are completing as a requirement for the Ed.D program at Peabody The project requires us to work with a client approved by the college faculty and to use analytical techniques to serve our client’s needs For our project, we are working as consultants to the university’s Center for Teaching staff as they assess and further enhance their programs for graduate students; they have identified Directors of Graduate Study as important constituents in this process We will be asking you questions about the desired and actual career outcomes for graduates of your program, and about the training and development students receive as they prepare for those careers We will make every effort to remove any identifiable information in the comments that you make or we will ask for your specific permission to use a comment that may be connected back to you and your department We know that you have a demanding schedule, and we will keep the interview to no more than 30 minutes We will work hard to accommodate your schedule and will be happy to come to your office or another campus location convenient to you One of us will be contacting you by phone soon to set up an interview time Thank you, in advance, for your consideration We look forward to speaking with you Sincerely, Mark Bryant Ed.D Candidate Christopher Rowe Ed.D Candidate Appendix G, Stakeholder Interview Protocol Stakeholder Interview Protocol General Stats How long has your doctoral program existed? On average, how many doctoral students you have in a cohort? What is the average length of time to degree in the program? What are the measures you use in order to gauge your students' successes in the marketplace? What, if anything, could your department to make your students more competitive? Department Preparation What skills and abilities are most important for your graduates to have? How you train and develop your graduate students to give them the skills and abilities that you value? What percent of your graduate students pursue academic vs non-academic careers? a For the academics, what percentage of your students pursue research intensive careers vs teaching intensive careers? Does what you just said about training and development differ for industry and academic paths? In what ways you prepare your students for careers that involve any level of teaching? a Are there specific departmental activities that you provide your students? On a scale of 1-5 with being not important and being very important, how important is your students' preparation for teaching in their chosen career? Teaching Center Information Do you know there is a CFT? What role does the CFT play in preparing your students for teaching after they graduate? What role should it play in your department in student teaching preparation? Have you heard of the TC program? If yes, what are your opinions of it? Appendix H, Stakeholder Matrix ... Graduate School and the Center for certificate from the Graduate School and Teaching, Vanderbilt? ??s Teaching Certificate the Center for Teaching program aims “to help graduate students, The Teaching. .. groups, and other programs Create and disseminate research-based best practices, models, and approaches to university teaching and learning -and facilitate access to resources that support them (Vanderbilt. .. preparation for teaching Teaching at Vanderbilt University is to research, trends, and student need, and in  an attempt to remain true to the CFT’s   Promote deep understanding of teaching and learning