An examination of PBIS implementation fidelity and student outcom

University of Louisville ThinkIR: The University of Louisville's Institutional Repository Electronic Theses and Dissertations 8-2018 An examination of PBIS implementation fidelity and student outcomes in an urban school district C Angelique Scherer University of Louisville Follow this and additional works at: https://ir.library.louisville.edu/etd Part of the Disability and Equity in Education Commons, and the Secondary Education Commons Recommended Citation Scherer, C Angelique, "An examination of PBIS implementation fidelity and student outcomes in an urban school district." (2018) Electronic Theses and Dissertations Paper 3026 https://doi.org/10.18297/etd/3026 This Doctoral Dissertation is brought to you for free and open access by ThinkIR: The University of Louisville's Institutional Repository It has been accepted for inclusion in Electronic Theses and Dissertations by an authorized administrator of ThinkIR: The University of Louisville's Institutional Repository This title appears here courtesy of the author, who has retained all other copyrights For more information, please contact thinkir@louisville.edu AN EXAMINATION OF PBIS IMPLEMENTATION FIDELITY AND STUDENT OUTCOMES IN AN URBAN SCHOOL DISTRICT By C Angelique Scherer B.A., University of Kentucky, 1996 M.Ed., Cambridge College, 2004 Ed.S., Albany State University, 2006 A Dissertation Submitted to the Faculty of the College of Education and Human Development of the University of Louisville in Partial Fulfillment of the Requirements for the Degree of Doctor of Education in Educational Leadership and Organizational Development Department of Educational Leadership and Organizational Development University of Louisville Louisville, KY August 2018 Copyright 2018 by C Angelique Scherer All rights reserved AN EXAMINATION OF PBIS IMPLEMENTATION FIDELITY AND STUDENT OUTCOMES IN AN URBAN SCHOOL DISTRICT By C Angelique Scherer B.A., University of Kentucky, 1996 M.Ed., Cambridge College, 2004 Ed.S., Albany State University, 2006 A Dissertation Approved on July 30, 2018 by the following Dissertation Committee: _ Dissertation Director Dr W Kyle Ingle Dr Jason C Immekus Dr Marco Munoz Dr Terry Scott ii DEDICATION This is dedicated to all the students who I have had the honor of teaching or mentoring throughout my educational career I am a better person because of my students Either children will reveal your true character or they will encourage you to be a better person iii ACKNOWLEDGMENTS I am grateful for many people who have supported me during this doctoral process I appreciate my dream team committee who are experts in their respective fields I appreciate Dr Ingle’s attention to detail and for pushing me to be a better researcher I am glad that he agreed to be my committee chair because his feedback has been instrumental in my completion of this dissertation I am very grateful for Dr Immekus because he did not know me and shared PBIS research he had collected over the years Dr Immekus met with me to share research and information even though I had not been in any of his classes With just a common interest in PBIS, he gave me tons of information and made suggestions on possible research questions even before I was ready to write this dissertation Dr Scott is a nationally renowned expert in the field of PBIS and I am very appreciative to have been able to learn from him I appreciate the meetings we had to discuss the path my research was going Due to his work in the district of study, his insight was valuable for the steps I needed to take Finally, I want to acknowledge Dr Munoz for spending his valuable time to re-explain statistical procedures to me even after I had been through his stats classes He also encouraged me to keep pressing to completion I acknowledge that I would not have been able to complete this dissertation without this committee of scholars iv I cannot state how grateful I am to my mother, Rose Walton for encouraging and supporting me throughout my educational career As a first-generation college student, she supported me in countless ways Her belief in me is the main reason why I am able to complete this project I am so appreciative to my husband, Ron Scherer for walking this doctoral journey with me Without his support and encouragement, I would not have been able to finish this project Finally, I want to acknowledge the late Dr Charlton Forbes for inspiring me to continue against all odds His legacy reminds me that it does not matter where we begin, but where we finish Finally, I have to acknowledge my cohort members We are a unique group of individuals who challenged each other’s ideas and encouraged each other when times were difficult Dr Aaron Wisman gets a special acknowledgment for sharing his time to discuss statistical procedures I will not forget his love of numbers or his sense of humor during stressful moments I have learned so much from my cohort members and I am thankful to have shared this doctoral journey with a group of people who care about students as much as I v ABSTRACT AN EXAMINATION OF PBIS IMPLEMENTATION FIDELITY AND STUDENT OUTCOMES IN AN URBAN SCHOOL DISTRICT C Angelique Scherer July 30, 2018 The purpose of this study was to examine Positive Behavior Interventions and Supports (PBIS) implementation fidelity and student outcomes for secondary schools in a large, urban school district The state department of education placed the district under corrective action due to disproportionate suspension practices In response to the corrective action, district administration required 24 schools to participate in PBIS training This study is an analysis of PBIS implementation fidelity data, discipline referrals, suspensions, and standardized test scores to determine if there were significant differences after PBIS implementation I employed the repeated measures analysis of variance (ANOVA) to examine outcomes over a four-year period Results indicated increased PBIS implementation fidelity over time Discipline outcomes showed improvement in one year included in the study, but improved outcomes did not sustain over time Additionally, academic outcomes did not show any significant increases I conclude by discussing the results in terms of helping urban schools improve outcomes for students vi TABLE OF CONTENTS DEDICATION iii ACKNOWLEDGMENTS iv ABSTRACT vi LIST OF TABLES ix LIST OF FIGURES x INTRODUCTION Context of the Study Purpose of the Study 10 Significance of the Study 11 Research Design 12 Definitions of Key Terms 14 Summary 16 REVIEW OF LITERATURE 18 Disproportionate Suspension 20 Suspension and Academic Achievement 24 Suspension and Drop-out Rates 29 Positive Behavior Interventions and Supports (PBIS) 34 Tier I: Universal Interventions 39 Tier II: Targeted Interventions 42 Tier III: Intensive Interventions 43 PBIS Implementation 44 PBIS Implementation Fidelity Measures 48 PBIS Implementation in Urban Settings 54 vii 1067-1087 Pinkelman, S., McIntosh, K., Rasplica, C., Berg, T., & Strickland-Cohen, M (2015) Perceived enablers and barriers related to sustainability of school-wide positive behavioral interventions and supports Behavioral Disorders, 40(3), 171-183 Raffaele Mendez, L (2003) Predictors of suspension and negative school outcomes: A longitudinal investigation In J Wald & D J Losen (Eds.), New directions for youth development (no 99; Deconstructing the school-to-prison pipeline) (pp 17– 34) San Francisco, CA: Jossey-Bass Raffele Mendez, L., & Knoff, H (2003) Who gets suspended from school and why: A demographic analysis of schools and disciplinary infractions in a large school district Education and Treatment of Children, 26(1), 30-51 Raffaele Mendez, L., Knoff, H., & Ferron, J (2002) School demographic variables and out-of-school suspension rates: A quantitative and qualitative analysis of a large, ethnically diverse school district Psychology in the Schools, 39(3), 259-277 Rausch, M K., & Skiba, R (2004) Unplanned outcomes: Suspensions and expulsions in Indiana Bloomington, IN: Indiana University, Center for Evaluation and Education Policy Rausch, M K., & Skiba, R J (2005) The academic cost of discipline: The contribution of school discipline to achievement In annual meeting of the American Educational Research Association, Montreal, Quebec Rausch, M K., Skiba, R J., & Simmons, A B (2004) The academic cost of discipline: The relationship between suspension/expulsion and school achievement Center for Evaluation and Education Policy, School of Education, Indiana University 133 Retrieved from http://www.indiana.edu/~atlantic/wpcontent/uploads/2014/03/Academic-Cost-ofSchool-Discipline.pdf Ravitch, D (2013) Reign of error: The hoax of the privatization movement and the danger to America’s public schools New York, NY: Vintage Books Scott, T M., Anderson, C., Mancil, R., & Alter, P (2009) Function-based supports for individual students in school settings In W Sailor, G Dunlap, G Sugai, & R Horner (Eds.), Handbook of Positive Behavior Support (pp 421-444) New York: Springer Scott, T M., & Barrett, S B (2004) Using staff and student time engaged in disciplinary procedures to evaluate the impact of school-wide PBS Journal of Positive Behavior Interventions, 6(1), 21-27 Simsonen, B., Eber, L., Black, A., Sugai, G., Lewandowski, H., Sims, B., & Myers, D (2012) Illinois statewide positive behavioral interventions and supports: Evolution and impact on student outcomes across years Journal of Positive Behavior Interventions, 14(1), 5-16 Skiba, R (2013) Reaching a critical juncture for our kids: The need to reassess schooljustice practices Family Court Review, 51(3), 380–387 Skiba, R., Horner, R., Chung, C.G., Rausch, M., May, S., & Tobin, T (2011) Race is not neutral: A national investigation of African American and Latino disproportionality in school discipline School Psychology Review, 40(1), 85-107 Skiba, R., Michael, R., Nardo, A., & Peterson, R (2002) The color of discipline: sources of racial and gender disproportionality in school punishment Urban Review, 134 34(4), 317-342 Skiba, R., & Peterson, R (1999) The dark side of zero tolerance: Can punishment lead to safe schools? Phi Delta Kappan, 80(5), 372-382 Skiba, R., & Rausch, M (2006) Zero tolerance, suspension, and expulsion: Questions of equity and effectiveness In C.M Evertson & C.S Weinstein (Eds.), Handbook of classroom management: Research, practice, and contemporary issues (pp 1063-1089) Mahwah, NJ: Lawrence Erlbaum Associates Skiba, R., & Sprague, J (2008) Safety without suspension Educational Leadership, 66(1), 38-43 Smith, E J., & Harper, S R (2015) Disproportionate impact of K-12 school suspension and expulsion on Black students in southern states Philadelphia: University of Pennsylvania, Center for the Study of Race and Equity in Education Retrieved from https://static1.squarespace.com/static/56675bc2b204d55efa34e5c5/t/5696ccd2a12 f4471d7f5d704/1452723412202/Smith_Harper_Report.pdf Solomon, B G., Klein, S A., Hintze, J M., Cressey, J M., & Peller, S L (2012) A meta‐analysis of school‐wide positive behavior support: An exploratory study using single‐case synthesis Psychology in the Schools, 49(2), 105-121 Spaulding, S., Horner, R., May, S., & Vincent, C (2008) Implementation of school-wide PBIS across the United States Retrieved from http://www.pbis.org/blueprint/evaluation-briefs/implementation-across-us Stevens, J (2002) Applied Multivariate Statistics for the Social Sciences (4th ed.) Mahwah, NJ: Lawrence Erlbaum Associates, Inc 135 Stevens, J (2007) Intermediate Statistics: A Modern Approach New York, NY: Routledge Sugai, G., Flannery, K B., & Bohanon-Edmonson, H (2004) School-wide positive behavior support in high schools: What will it take? In Bohanon, H., Flannery, K B., Eber, L., & Sugai, G (Eds.), Positive behavior support in high schools: Monograph from the 2004 Illinois high school forum of positive behavioral interventions and supports, 5-15 Sugai, G., & Horner, R (2002) The evolution of discipline practices: School-wide Positive behavior supports Child and Family Behavior Therapy, 24(1-2), 23-50 Sugai, G., & Horner, R (2006) A promising approach for expanding and sustaining school-wide positive behavior support School Psychology Review, 35(2), 245259 Sugai, G., & Horner, R (2008) What we know and need to know about preventing Problem behavior in schools, Exceptionality, 16(2), 67-77 Sugai, G., & Horner, R (2009) Defining and describing school-wide positive behavior support In W Sailor, G Dunlap, G Sugai & R Horner (Eds.), Handbook of positive behavior support (pp 307-324) New York, New York: Springer Science & Business Media Sugai, G., Horner, R., Algozzine, R., Barrett, S., Lewis, T., Anderson, C., & Simonsen, B (2010) School-wide positive behavior support: Implementers’ blueprint and self-assessment Eugene: University of Oregon Sugai, G., Horner, R H., Dunlap, G Hieneman, M., Lewis, T J., Nelson, C M., Scott, T., Liaupsin, C., Sailor, W., Turnbull, A P., Turnbull, H R., III, Wickham, D 136 Reuf, M., & Wilcox, B (2000) Applying positive behavioral support and functional behavioral assessment in schools Journal of Positive Behavioral Interventions, 2(3), 131-143 Sugai, G., Horner, R., & Lewis-Palmer, T (2001) Team Implementation Checklist Eugene, OR: Educational and Community Supports Sugai, G., & Simsonsen, B (2012) Positive behavioral interventions and supports: History, defining features, and misconceptions Retrieved from http://www.pbis.org/common/cms/files/pbisresources/PBIS_revisited_June19r_20 12.pdf Suh, S., & Suh, J (2007) Risk factors and levels of risk for high school dropouts Professional School Counseling, 10(3), 297-306 Swain-Bradway, J., Swoszowski, N., Boden, L., & Sprague, J (2013) Voices from the field: Stakeholder perspectives on PBIS implementation in alternative educational settings Education and Treatment of Children, 36(3), 31-46 United States Census Bureau (2010) 2010 Census Urban and Rural Classification and Urban Area Criteria Retrieved from https://www.census.gov/geo/reference/ua/urban-rural-2010.html United States Departments of Education & Juvenile Justice (2014) Dear colleague letter on nondiscriminatory administration of school discipline Retrieved from http://www2.ed.gov/policy/gen/guid/school-discipline/fedefforts.html#guidance Vincent, C G., Swain-Bradway, J., Tobin, T J., & May, S (2011) Disciplinary referrals for culturally and linguistically diverse students with and without disabilities: Patterns resulting from school-wide positive behavior support Exceptionality, 137 19(3), 175-190 Wald, J., & Losen, D J (2003) Defining and redirecting a school‐to‐prison pipeline New directions for youth development, 2003(99), 9-15 Warren, J., Edmonson H., Griggs, P., Lassen, S., McCart, A Turbull, A., & Sailor, W (2003) Urban applications of school-wide positive behavior support: Critical issues and lessons learned Journal of Positive Behavior Interventions, 5(2), 8091 138 APPENDIX A: Benchmarks of Quality School-wide Benchmarks of Quality: SCORING FORM (Revised) School Name: _ District: Coach’s Name: _ Date: STEP 1: Each team member uses the Scoring Guide to determine appropriate point value Circle ONLY ONE response STEP 2: Coach: Indicate the most frequent response of you and each of your team members If there is a tie, report the higher score Critical Elements STEP PBIS Team Team has administrative support Team has regular meetings (at least monthly) 2 1 2 Team has established a clear mission/purpose Faculty Commitme nt Faculty are aware of behavior problems across campus through regular data sharing Faculty involved in establishing and reviewing goals 139 Effective Procedures for Dealing with Discipline Faculty feedback is obtained throughout year Discipline process described in narrative format or depicted in graphic format Discipline process includes documentation procedures Discipline referral form includes information useful in decision making 10 Problem behaviors are defined 1 2 1 1 2 3 19 Rules/procedures are developed and posted for specific settings (settings where data suggest they are needed) 20 Rules/procedures are linked to expectations 1 21 Staff are involved in development of expectations and rules/procedures 22 A system of rewards has elements that are implemented consistently across campus 23 A variety of methods are used to reward students 2 24 Rewards are linked to expectations 2 1 11 Major/minor behaviors are clearly differentiated Data Entry & Analysis Plan Established Expectatio ns & Rules Developed Reward/ Recognitio n Program Established 12 Suggested array of appropriate responses to major (office-managed) problem behaviors 13 Data system is used to collect and analyze ODR data 14 Additional data collected (attendance, grades, faculty attendance, surveys) and used by PBIS Leadership Team 15 Data analyzed by PBIS Leadership Team at least monthly 16 Data shared with team and faculty monthly (minimum) 17 3-5 positively stated school-wide expectations are posted around school 18 Expectations apply to both students and staff 25 Rewards are varied to maintain student interest 26 Ratios of acknowledgments to corrections are high 27 Students are involved in identifying/developing incentives 28 The system includes incentives for staff/faculty 140 2 Lesson Plans for Teaching Expectatio ns/ Rules Implementation Plan Classroom Systems 29 A behavioral curriculum includes teaching expectations and rules/procedures 30 Lessons include examples and non-examples 31 Lessons use a variety of teaching strategies 32 Lessons are embedded into subject area curriculum 33 Faculty/staff and students are involved in development & delivery of behavioral curriculum 34 Strategies to share key features of PBIS initiative with families/community are developed and implemented 35 A curriculum to teach the components of the discipline system to all staff is developed and used 36 Plans for training staff how to teach expectations/rules 37 A plan for teaching students expectations/rules and procedures/rewards is developed, scheduled & delivered 38 Booster sessions for students and staff are planned, scheduled, and implemented 39 Schedule for rewards/incentives for the year is planned 40 Plans for orienting incoming staff and students are developed and implemented 41 Plans for involving families/community are developed and implemented 42 Classroom rules are defined for each of the school-wide expectations and posted in classrooms 43 Classroom routines and procedures are explicitly identified for activities where problems often occur 44 Expected behavior routines in classroom are taught 45 Classroom teachers use immediate and specific praise 46 Acknowledgment of students demonstrating adherence to rules and routines occurs more frequently than acknowledgment of inappropriate behavior 141 1 2 1 2 2 1 1 2 2 2 Evaluation 47 Procedures exist for tracking classroom behavior problems 48 Classrooms have a range of consequences/interventions for problem behavior that are documented and consistently delivered 49 Students and staff are surveyed about PBIS 2 50 Students and staff can identify expectations and rules 51 Staff use referral process (i.e., office managed vs teacher managed) and forms appropriately 52 Staff use reward system appropriately 3 53 Outcomes (behavior problems, attendance, etc.) are documented and used to evaluate PBIS Action Plan / 107 = _ Benchmarks Score Benchmarks of Quality TEAM SUMMARY School _ Date _ Total Benchmarks Score _ Areas of Strength Critical Element Description of Areas of Strength 142 Areas in Need of Development Critical Element Description of Areas in Need of Development 143 APPENDIX B: Tiered Fidelity Inventory Action Plan Tier I Subscale Item Teams 1.1 Team Composition 1.2 Team Operating Procedures 1.3 Behavioral Expectations 1.4 Teaching Expectations 144 Implementation 1.5 Problem Behavior Definitions 1.6 Discipline Policies 1.7 Professional Development 1.8 Classroom Procedures 1.9 Feedback and Acknowledgment 1.10 Faculty Involvement Current Score Action(s) Person(s) Responsible Timeline Evaluation Eval 1.11 Student/Family/Community Involvement 1.12 Discipline Data 1.13 Data-based Decision Making 1.14 Fidelity Data 1.15 Annual Evaluation 145 CURRICULUM VITA NAME: Charlmeille Angelique Scherer ADDRESS: 1909 Carabiner Way Louisville, KY 40245 DOB: Louisville, Kentucky – March 3, 1973 EDUCATION & TRAINING: Ed.S., Educational Leadership Albany State University, Albany, GA 2005-06 M.Ed., School Administration Cambridge College, Cambridge, MA 2003-04 B.A., Special Education University of Kentucky, Lexington, KY 1991-96 AWARDS: 146 PROFESSIONAL SOCIETIES: Council for Exceptional Children (CEC) Council of Administrators of Special Education (CASE) Jefferson County Association of School Administrators (JCASA) PUBLICATIONS: NATIONAL MEETING PRESENTATIONS: Council for Exceptional Children (CEC) Fall Conference “Manifestation Determinations: To Manifest or not to Manifest” November 2016 REFEREED JOURNALS BOOKS AND SYMPOSIA INVITED PRESENTATIONS: 147 ... 2018 by C Angelique Scherer All rights reserved AN EXAMINATION OF PBIS IMPLEMENTATION FIDELITY AND STUDENT OUTCOMES IN AN URBAN SCHOOL DISTRICT By C Angelique Scherer B.A., University of Kentucky,... care about students as much as I v ABSTRACT AN EXAMINATION OF PBIS IMPLEMENTATION FIDELITY AND STUDENT OUTCOMES IN AN URBAN SCHOOL DISTRICT C Angelique Scherer July 30, 2018 The purpose of this... between PBIS implementation and improved student outcomes The study is significant because it provides information about PBIS implementation fidelity and the sustainability of PBIS in urban, secondary