April-2019-CAEP-8-Annual-Reporting-Measures

APRIL 30, 2019 ANNUAL REPORTING MEASURES COLLECTING INFORMATION ON PROGRAM IMPACT AND OUTCOMES Berea College Education Studies Department Teacher Education Program Table of Contents Section Purpose of Report The Eight CAEP Annual Reporting Measures Measure 1: Impact on P-12 Learning & Development Measure 2: Indicators of Teaching Effectiveness Measure 3: Employer Satisfaction and Completer Persistence Measure 4: Completer Satisfaction Measure 5: Completion/Graduation Rates Measure 6: Licensure/Certification Rates Measure 7: Employment Rate Measure 8: Consumer Information Page # 10 11 12 15 17 Purpose of Report The Council for the Accreditation of Teacher Preparation (CAEP) requires that Education Preparation Providers (EPPs) publicly share program information and data that is accessible to multiple stakeholders At Berea College, the EPP is known as the Teacher Education Program (TEP) within the Education Studies Department (EDS) There are eight annual reporting measures used to examine both program impact and program outcomes In the 2018 CAEP Handbook1, the measures are defined as “basic indicators” of EPP performance that are associated with “candidates as they complete preparation, and to completers once they are on the job” (p 26) Data for these measures were gathered from multiple sources including departmental surveys and focus groups; reports generated by the Berea College Office of Institutional Research and Assessment; and reports published by state agencies such as the Kentucky Center for Statistics (KYStats) and the Kentucky Council for Postsecondary Education At this time, the referenced (and most recent) handbook is the CAEP Handbook: Initial Level Programs 2018 The current CAEP Accreditation Handbook can be accessed at: http://caepnet.org/accreditation/caepaccreditation/caep-accreditation-handbook The Eight CAEP Annual Reporting Measures IMPACT: Performance of program completers once they are employed as P-12 teachers Impact on P-12 Learning & Development (Component 4.1) Indicators of Teaching Effectiveness (Component 4.2) Employer Satisfaction and Completer Persistence (Component 4.3) Completer Satisfaction (Component 4.4) OUTCOME: Outcomes of program preparation and consumer reporting information Completion/Graduation Rates Licensure/Certification Rates Employment Rate Consumer Information Measure 1: Impact on P-12 Learning & Development At this time, Kentucky no longer has a mandatory, state-funded process for obtaining data related to completers’ performance and P-12 student growth This is due to change in regulation that once required newly inducted teachers (program completers) to participate in the Kentucky Teacher Internship Program (KTIP) Participation in KTIP generated performance data related to completers’ performance in the classroom Additionally, new leadership at the Kentucky Department of Education has indicated that the state will continue its practice of not releasing P-12 student data Therefore, this impacts plans for collecting data on P-12 student learning and development With these changes in state leadership and mandated processes, EDS will focus on working with local school districts to obtain P-12 student data Relationship building with community partners has been a priority for the EDS thus far during the 2018-2019 academic year EDS has started a discussion with the Community of Teachers (a stakeholder group consisting of P-12 educators) on how to obtain P-12 student data in a way that respects school districts, completers, and P-12 students EDS will continue this collaboration and formulate plans for obtaining P-12 student data during summer retreats planned for the Community of Teachers and for EDS faculty and staff EDS anticipates that data collection will begin during the 2019-2020 academic year after receiving guidance and permission from school partners EDS will continue conducting completer focus groups so that the department can maintain strong relationships with completers and learn how they utilize program preparation to positively influence their students From analysis of focus group discussions, EDS has learned that completers apply program knowledge to measure student growth through the use of formative assessments Completers discussed how they analyze and interpret assessment results in order to better understand students’ abilities, to measure students’ progress, and to inform classroom interventions P-12 student growth data is available through the use of completers’ classroom assessments EDS must ensure that the processes for sharing assessment results and drawing conclusions based upon the data have mutual benefits for completers and EDS Measure 2: Indicators of Teaching Effectiveness To successfully meet this component, CAEP requires that teaching effectiveness be demonstrated “through structured and validated observation instruments and/or student surveys” Currently, the EDS has collected employer survey and completer survey data that asked about teaching effectiveness (data are presented below) However, since observation instruments and students surveys are the desired sources of evidence, EDS will collaborate with our partners to develop processes as well as observation tools and P-12 student perception surveys The measures need to be developmentally appropriate for users and must be approved by the school districts who are our collaborators Additionally, data from these measures should inform the professional growth of program completers and provide them with feedback that can be used to refine their practice Employer Survey Results The employer survey is a departmental survey that was administered for the first time in January 2019 to the principals of program completers Survey indicators are elements from the Charlotte Danielson Framework for Teaching2 that were selected by stakeholders; content validity for the indicators was established using the Lawshe (1975 method) Employers were selected based upon the responses from a survey of 2015-2016 and 2016-2017 completers who specified that they were currently employed as P-12 teachers Twelve out of 29 completers indicated that they are current P-12 teachers and provided their employers’ contact information Six of the 13 employers who were contacted replied to the survey resulting in a 46.2% response rate Employers were asked to rate their employees’ performance for each element using the following categories:     Exemplary (4): Meets the objective or standard with a high level of consistency Accomplished (3): Cleary developing the ability to meet the objective or standard with some degree of consistency Developing (2): Shows some development in addressing the objective or standard Ineffective (1): Shows little or no evidence of development in addressing the objective or standard The survey data reveal that employers perceive completers as effective teachers and are satisfied with their preparation from the TEP Overall, employers rated completers’ teaching effectiveness higher on elements related to planning and preparation and lower on elements related to instruction and management of instructional groups The Framework for Teaching that has been adapted for the Kentucky Department of Education can be accessed at: https://education.ky.gov/teachers/PGES/TPGES/Documents/Kentucky%20Framework%20for%20Teaching.pdf Employer Ratings of Completers’ Teaching Effectiveness Average Rating Framework for Teaching Element 1B Demonstrating Knowledge of Students – Knowledge of the Learning Process 3.67 1C Selecting Instructional Outcomes – Value, Sequence, and Alignment 3.67 1E Designing Coherent Instruction – Learning Activities 3.67 2B Establishing a Culture for Learning – Expectations for Learning and Achievement 3.67 1A Demonstrating Knowledge of Content and Pedagogy – Knowledge of Content-Related Pedagogy 3.50 1E Designing Coherent Instruction – Lesson and Unit Structure 3.50 1F Designing Student Assessment – Congruence with Instructional Outcomes 3.50 1F Designing Student Assessment – Design of Formative Assessments 3.50 2A Creating an Environment of Respect and Rapport – Teacher Interaction with Students 3.50 3A Communication with Students Expectations for Learning 3.50 3D Using Assessment in Instruction – Monitoring of Student Learning 3.50 2D Managing Student Behavior –Expectations 3.33 2D Managing Student Behavior –Monitoring of Student Behavior 3.33 2D Managing Student Behavior – Response to Student Misbehavior 3.33 2C Managing Classroom Procedures – Management of Instructional Groups 3.17 3C Engaging Students in Learning – Activities and Assignments 3.17 3D Using Assessment in Instruction – Feedback to Students 3.17 Completer Survey Results The completer survey is a departmental survey that was administered for the first time in January 2019 Survey indicators are elements from the Charlotte Danielson Framework for Teaching3 that were selected by stakeholders; content validity for the indicators was established using the Lawshe (1975 method) The survey was sent to completers who graduated in either 20152016 or 2016-2017 Of the 29 completers who were contacted, 21 responded to the survey resulting in a 72.4% response rate Thirteen completers indicated that they are currently employed full-time as P-12 teachers Completers represented the following specialty licensure areas: Elementary Education, Art, Physical Education and Health, Middle School Science, Middle School Math, English, Vocal Music, Instrumental Music, and Engineering and Technology The Framework for Teaching that has been adapted for the Kentucky Department of Education can be accessed at: https://education.ky.gov/teachers/PGES/TPGES/Documents/Kentucky%20Framework%20for%20Teaching.pdf Completers who are currently teaching were asked about their perceptions of their teaching effectiveness Overall, completers rated their teaching effectiveness at the acceptable level (scores of or above) Scores were highest for elements regarding lesson planning and using assessments Completers were asked to rate their own performance for each element using the following categories:     Exemplary (4): Meets the objective or standard with a high level of consistency Accomplished (3): Cleary developing the ability to meet the objective or standard with some degree of consistency Developing (2): Shows some development in addressing the objective or standard Ineffective (1): Shows little or no evidence of development in addressing the objective or standard Completers’ Ratings of Perceived Teaching Effectiveness Framework for Teaching Element Average Rating 2A Creating an Environment of Respect and Rapport – Teacher Interaction with Students 3.30 3E Demonstrating Flexibility and Responsiveness – Lesson Adjustment 3.30 1D Demonstrating Knowledge of Resource – Uses available technology to design, plan and implement instruction that facilitates student learning 3.17 3A Communication with Students – Expectations for Learning 3.10 3C Engaging Students in Learning – Activities and Assignments 3.10 1E Designing Coherent Instruction – Learning Activities 3.08 2B Establishing a Culture for Learning – Expectations for Learning and Achievement 3.00 2C Managing Classroom Procedures – Management of Transitions 3.00 2D Managing Student Behavior – Monitoring of Student Behavior 3.00 3D Using Assessment in Instruction – Monitoring of Student Learning 3.00 3D Using Assessment in Instruction – Feedback to Students 3.00 3E Demonstrating Flexibility and Responsiveness – Response to Students 3.00 1A Demonstrating Knowledge of Content and Pedagogy – Knowledge of Content-Related Pedagogy 2.92 1F Designing Student Assessment – Congruence with Instructional Outcomes 2.92 1F Designing Student Assessment – Design of Formative Assessments 2.92 2B Establishing a Culture for Learning – Importance of the Content 2.90 2C Managing Classroom Procedures – Management of Instructional Groups 2.90 2D Managing Student Behavior – Expectations 2.90 3D Using Assessment in Instruction – Assessment Criteria 2.90 Framework for Teaching Element Average Rating 1F Designing Student Assessment – Criteria and Standards 2.83 2D Managing Student Behavior – Response to Student Misbehavior 2.80 1E Designing Coherent Instruction – Lesson and Unit Structure 2.75 3B Using Questioning and Discussion Techniques – Quality of Questions 2.70 3B Using Questioning and Discussion Techniques – Discussion Techniques 2.70 1C Selecting Instructional Outcomes – Suitability for Diverse Learners 2.67 Measure 3: Employer Satisfaction and Completer Persistence Data from the Employer Survey demonstrate that employers perceive completers as effective teachers and are satisfied with their preparation from the TEP Although the data show areas for improvement regarding completers’ teaching effectiveness (available in the Employer Survey data table on page 6), employers indicated that they were highly satisfied with completers’ preparation for their assigned teaching responsibilities Additionally, employers rated completers’ preparation as “much better” when compared to their peers trained at different EPPs Employer Satisfaction Survey Items Overall, how satisfied are you with your employee's preparation for his/her assigned responsibilities during his/her first year of teaching? Response Values: = Highly Satisfied, = Satisfied, = Neutral, = Dissatisfied, = Highly Dissatisfied Average Rating = 4.83 How would you rate this employee's preparation compared to the preparation of other first- year teachers trained at different institutions? Response Values: = Much better, = Somewhat better, = About the Same, = Somewhat Worse, = Much Worse Average Rating = 5.00 Note: The data reflect the input from employers Employment Milestones Based on the responses from the Completer Survey (see Measure for more survey details), two out of the 13 completers who are currently employed as full-time P-12 teachers indicated a rank change from Rank III to Rank II According to the Education Professional Standards Board (EPSB) Division of Teacher Licensure and Quality4, a rank change to Rank II refers to those who have earned a “master's degree in a subject field approved by EPSB or equivalent continuing education.” Additionally, five completers indicated that they have been teaching for two years or more Information on the EPSB Rank System and Change can be accessed at: http://www.epsb.ky.gov/mod/page/view.php?id=101 Measure 4: Completer Satisfaction Feedback from the completer survey demonstrates that completers are satisfied with the quality of education they received from the TEP Completers also indicated that TEP experiences, particularly the Student Teaching Experience, were effective in preparing them for teaching positions Additional information regarding the survey items and sample can be found on page Completer Satisfaction Survey Items Overall, how satisfied are you with the quality of education that you received in the Teacher Education Program? Response Values: = Highly Satisfied, = Satisfied, = Neutral, = Dissatisfied, = Highly Dissatisfied Average Rating = 4.22 Note: The data reflect the input of 18 completers who responded to this item Please rate the effectiveness of the following experiences in preparing you for your teaching position? *This indicator was only presented to completers who indicated that they were currently employed as full-time teachers (N = 13) Response Values: = Highly Effective, = Effective, = Neutral, = Ineffective, = Highly Ineffective Teacher Education Program Experience Average Rating Student Teaching Experience 4.54 Clinical Field Experiences in Pre-Professional Terms 4.38 Education Studies Coursework 4.38 Content Area Coursework 3.92 Education Studies Internship 3.77 10 Measure 5: Completion/Graduation Rates Graduation rates for the Berea College TEP in comparison to EPPs statewide are provided in the tables below The data were retrieved from the 2017 Kentucky Teacher Preparation Feedback Report5 published in March 2018 by the Kentucky Center for Statistics6 It is noted in the report that the “cohort year includes students who started a Teacher Preparation Program (TPP) in that academic year, as reported by EPSB TPPs are designed to allow completion within three years” (pg on statewide version of the report) Only data for Year Three is provided because a majority of the students should have completed programs at their respective institutions by that point in time Compared to the rates for all Kentucky EPPs, Berea College has lower program completion rates and higher program exit rates These numbers reflect the Berea College TEP before the current Quality Assurance System was implemented to monitor admissions to the program, admission to the Student Teaching Term, and successful program completion The Education Studies Department anticipates that the program completion rates will increase due to new and updated assessments and process for program progression Cohort Completion Status Rates for Berea College Cohort Completion Status for Year Berea College TEP 2011 Cohort (n = 70) 2012 Cohort (n = 36) 2013 Cohort (n = 15) Program Exit (i.e., drop out, transfer) 38% 28% 33% Continuing Program (still enrolled at time of report) 15% 25% 40% Program Completion (graduated) 47% 47% 27% 2011 Cohort (n = 3,322) 2012 Cohort (n = 4,806) 2013 Cohort (n = 1,510) Program Exit (i.e., drop out, transfer) 15% 14% 14% Continuing Program (still enrolled at time of report) 9% 16% 14% Program Completion (graduated) 76% 70% 72% Cohort Completion Status Rates for All Kentucky EPPs Cohort Completion Status for Year All Kentucky EPPs This report can be viewed at the state level and at the institutional level Data for all Kentucky EPPs can be accessed at: https://kcews.ky.gov/Content/Reports/TPPFR_2017_Statewide.pdf Data for the Berea College TEP can be accessed at: https://kcews.ky.gov/Content/Reports/TPPFR_2017_INST_00195500.pdf This reporting agency is also known as KYStats Previously, it was the Kentucky Center for Education & Workforce Statistics (KCEWS) 11 Measure 6: Licensure/Certification Rates Licensure rates for the Berea College TEP in comparison to EPPs statewide are provided in the tables below The data were retrieved from the 2017 Kentucky Teacher Preparation Feedback Report7 Overall, there is a greater percentage of completers at Berea College who obtain the statement of eligibility compared to completers across all Kentucky EPPs However, there is a lower percentage of Berea College completers who obtain their certification compared to completers statewide Licensure/Certification Rates for Berea College Certification Area Arts & Humanities Career & Technical Education Career & Technical Education Elementary Education Elementary Education Elementary Education English/Language Arts English/Language Arts Mathematics Mathematics Mathematics Music Music Physical Health Physical Health Science Social Studies Social Studies Year of Completion 2012 2012 2013 2012 2013 2014 2013 2014 2012 2013 2014 2012 2013 2013 2014 2012 2012 2013 % of Completers who Obtained the Statement of Eligibility 100% 100% 0% 100% 100% 83% 100% 67% 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% % of Completers who Obtained Certification 100% 0% 0% 80% 33% 17% 33% 0% 67% 33% 60% 0% 0% 50% 0% 100% 0% 100% This report can be viewed at the state level and at the institutional level Data for all Kentucky EPPs can be accessed at: https://kcews.ky.gov/Content/Reports/TPPFR_2017_Statewide.pdf Data for the Berea College TEP can be accessed at: https://kcews.ky.gov/Content/Reports/TPPFR_2017_INST_00195500.pdf 12 Licensure/Certification Data for All Kentucky EPPs Certification Area Arts & Humanities Arts & Humanities Arts & Humanities Career & Technical Education Career & Technical Education Career & Technical Education Elementary Education Elementary Education Elementary Education English/Language Arts English/Language Arts English/Language Arts Mathematics Mathematics Mathematics Music Music Music Physical Health Physical Health Physical Health Science Science Science Social Studies Social Studies Social Studies Year of Completion 2012 2013 2014 2012 2013 2014 2012 2013 2014 2012 2013 2014 2012 2013 2014 2012 2013 2014 2012 2013 2014 2012 2013 2014 2012 2013 2014 % of Completers who Obtained the Statement of Eligibility 91% 71% 75% 89% 85% 91% 94% 90% 88% 92% 87% 91% 93% 91% 94% 82% 81% 79% 88% 80% 77% 97% 94% 93% 90% 86% 91% % of Completers who Obtained Certification 70% 46% 46% 67% 59% 72% 68% 69% 66% 73% 69% 74% 74% 76% 75% 46% 57% 47% 60% 54% 62% 76% 81% 77% 68% 68% 66% EPSB has changed the process for obtaining certification due to the suspension of KTIP Previously, in order to apply for certification, completers received their Statement of Eligibility upon successful program completion and participated in the KTIP8 Currently, completers no longer receive the statement of eligibility nor participate in KTIP They apply for their certification upon completion from their respective EPPs and completion of additional state requirements (e.g., passing the required Praxis examination for the certification area; submitting official transcripts, Information regarding the EPSB certification process for candidates trained in Kentucky can be accessed at: http://www.epsb.ky.gov/mod/page/view.php?id=341 13 etc.) Below are numbers for the Berea completers who applied for certification since the changes in the certification process Completion Term Spring 2018 Fall 2018 # of Program Completers 10 # of Completers Who Obtained Certification % of Completers who Obtained Certification 0%* 90%** *When compiling this report, EDS learned that there was a glitch in the processing of this completer’s certification paperwork This number may change **One of the completers successfully completed the Teacher Education Program but has not graduated from institution yet This number may change 14 Measure 7: Employment Rates Employment rates for the Berea College TEP in comparison to EPPs statewide are provided in the tables below The data were retrieved from the 2017 Kentucky Teacher Preparation Feedback Report9 There is ample evidence for the employment rates when reviewing the data at the state level, but there cannot be a direct comparison using data for that report because a majority of data at the institutional were redacted for privacy purposes Because the state wide and Berea College data cannot be directly compared, results from the EDS Completer Survey are provided so that some comparison between Berea College and all Kentucky EPPs can be made Program Completers One Year Post-Graduation from EPP All Kentucky EPPs 2012 Completers Employment Categories 2013 Completers 2014 Completers # % # % # % Public K-12 Teachers in KY 812 36.9% 786 35.5% 844 29.4% Employed in the Education Field (Excluding Public K-12) 667 30.3% 723 32.7% 522 18.2% Employed Outside of the Education Field 274 12.5% 309 14.0% 256 8.9% 28.6% 57.1% 406 5800.0% 443 20.2% 390 17.6% 844 29.4% 2,198 100.0% 2,212 100.0% 2,872 100.0% Non-Teacher Certified Employees in Public K-12 Not Employed in KY or Unemployed Total Program Completers One Year Post-Graduation from EPP Berea College TEP 2012 Completers Employment Categories Public K-12 Teachers in KY 2013 Completers 2014 Completers # % # % # % - - - Employed in the Education Field (Excluding Public K-12) Employed Outside of the Education Field Non-Teacher Certified Employees in Public K-12 Data for these categories were redacted, so percentages and totals could not be calculated Not Employed in KY or Unemployed Total This report can be viewed at the state level and at the institutional level Data for all Kentucky EPPs can be accessed at: https://kcews.ky.gov/Content/Reports/TPPFR_2017_Statewide.pdf Data for the Berea College TEP can be accessed at: https://kcews.ky.gov/Content/Reports/TPPFR_2017_INST_00195500.pdf 15 Because there are not comparable data for Berea College, data from the Completer Survey of 20152016 and 2016-2017 are displayed below in order to provide some context on the employment categories of Berea TEP completers When comparing these data to the statewide numbers, there is a higher percentage of Berea TEP completers who are employed as full-time teachers This may be partially explained by the fact that they survey did not limit employment as a full-time teacher to the state of Kentucky and this examined some completers 2-3 years post completion It should also be noted that different completion cohorts are represented in the EDS Completer Survey and the KYStats report EDS Completer Survey – Employment Data Employment Categories 2015-2016 Completers 2016-2017 Completers # % # % 44.4% 45.5% 11.1% 9.1% 11.1% 0.0% Employed in a setting not related to education 5.6% 18.2% Unemployed 0.0% 0.0% Unknown/Did Not Respond to Survey 27.8% 27.3% Total 18 100.0% 11 100.0% Employed as a full-time teacher Employed in a school setting in a part-time teaching and/or support role (e.g substitute teacher, teaching assistant, etc.) Employed in a related education setting (e.g college or university employee, Partners for Education, etc.) 16 Measure 8: Consumer Information For Consumer Reporting Information, EDS has reported the Cost of Attendance at Berea College, Student Debt Data for 2012 College Graduates, and Student Loan Default Rates for Kentucky 4Year Independent institutions All data are reported at the institutional level; data is not provided at the program level Tuition is paid by Berea College with funding from the endowment and gifts in addition to scholarships and grants brought by students; no student pays tuition These numbers are provided by the Financial Aid Office and are reported by the Office of Institutional Research and Assessment in the annual Fact Book10 2016-2017 Academic Year 2017-2018 Academic Year 2018-2019 Academic Year Tuition* $24,900 $25,200 $39,400 Registration Costs $7,042 $7,094 $7,354 Average Other Costs $3,100 $3,100 $3,300 Total Student Expense Budget $10,142 $10,194 $10,454 Berea College Cost of Attendance The Berea College Office of Institutional Research and Assessment published the “Student Financial Status and Debt Report” in August 201411 The report mainly consists of data specific to Berea College but does provide some state and national comparisons In the data table below, the percent of Berea graduates with zero debt is approximately the same as graduates nationwide and is 10% less than graduates statewide However, the average student debt of 2012 Berea graduates is approximately $15,000 less compared to the Kentucky average and $22,000 less than the national average Student Debt Data for 2012 College Graduates Average Debt of Borrowers Percent of Graduates with Zero Debt National Graduates Kentucky Graduates Berea College Graduates $29,400 $22,384 $7,224 29% 38% 28% The Kentucky Council on Postsecondary Education (CPE) released the report September 2017 Student Loan Default and Repayment in Kentucky12 In this report, Kentucky CPE defined CDR as “the percentage of borrowers who enter repayment on federal student loans within a given fiscal year and who subsequently default within three years of leaving the college” (pg 8) The data Access the Berea College 2018-2019 Fact Book at: https://www.berea.edu/ira/institutional-data-reports/ Access the Berea College Student Financial Status and Debt Report at: https://www.berea.edu/ira/institutional-datareports/ 12 Access the Kentucky CPE Student Loan Default and Repayment in Kentucky report at: http://cpe.ky.gov/data/reports/studentloanrepaymentreport.pdf 10 11 17 presented below represent the Cohort Default Rates (CDR) for 4-Year Independent Colleges that have EPSB approved EPPs The CDR reflects students from the entire institution and is not limited to those students enrolled in EPPs Institution 2011 CDR 2012 CDR 2013 CDR Alice Lloyd College 11.6% 14.0% 11.1% Asbury University 5.0% 3.8% 3.7% Bellarmine University 5.3% 4.1% 4.3% Berea College 12.8% 9.5% 10.3% Brescia University 12.8% 10.3% 10.3% Campbellsville University 16.8% 15.4% 12.7% Georgetown College 4.3% 5.5% 6.3% Kentucky Christian University 11.9% 13.1% 9.7% Kentucky Wesleyan College 10.5% 10.4% 11.4% Lindsey Wilson College 15.9% 12.0% 11.5% Midway University 10.3% 9.9% 10.1% Spalding University 8.7% 8.9% 7.2% Thomas More College 7.3% 5.9% 7.2% Transylvania University 4.1% 2.4% 2.6% Union College 13.2% 13.9% 11.6% University of Pikeville 12.5% 15.4% 13.0% University of the Cumberlands 8.7% 6.2% 6.4% 18