Evaluation-of-TNTP-and-Solution-Tree-in-WCSD_Final-Report

Evaluation of TNTP and Solution Tree in Washoe County School District Prepared for: Laura Davidson Washoe County School District Prepared by: Jennifer Hogg, Marian Negoita, Emily McCaffrey, and Victoria Rodriguez Social Policy Research Associates September 30, 2020 This page intentionally left blank TABLE OF CONTENTS Executive Summary Introduction Section I: TNTP And Solution Tree Supports Section II: Commitment & Buy-In 13 Section III: Vendor Effectiveness 15 Section III A: Perceptions of Effectiveness 15 Section III B: Impact on School Climate and SEL Skills 20 Section IV: Implementation & Sustainability Considerations 29 Conclusion 33 Technical Appendices 34 Appendix A: Interview & Focus Group Samples 34 Appendix B: TNTP & Solution Tree Principal and Teacher Survey Response Rates 35 Appendix C: CITS Methodology and Detailed Results 38 Appendix D: Quantile Regression Methodology and Results 42 Appendix E: Description of School Climate Survey Measures 44 EXECUTIVE SUMMARY The New Teacher Project (TNTP) and Solution Tree are two research-based education vendors hired by Washoe County School District (WCSD) to support schools that have been designated in need of improvement Over the course of the 2019-2020 school year, these vendors worked with 25 schools to improve school leadership practices, instruction, and student achievement In fall 2019, WCSD commissioned a study to better understand the implementation (pre-COVID-19) and impact of these vendors Key Findings FIDELITY, COMMITMENT, & BUY-IN • Both vendors implemented the activities they proposed with fidelity • Principals and leadership teams think highly of their coaches, and especially appreciate that they empower school leaders to own the work • Awareness of the two vendors among non-leadership team teachers varied, with awareness somewhat higher among Solution Tree teachers VENDOR EFFECTIVENESS • School leaders overwhelmingly reported that vendors positively impacted their own instructional leadership skills • Principals and teachers reported increased effectiveness of collaborative team meeting time, greater teacher buy-in for using student data to drive instruction, and strengthened beliefs in the ability of all students to meet high standards, particularly among staff working with TNTP • Impact analyses reveal that working with TNTP significantly increased staff perceptions of school climate, whereas working with Solution Tree did not Additional statistical analysis suggests that TNTP participation yielded the greatest impacts in schools with relatively lower school climate scores • The evidence on the overall impact of the intervention on student SEL skills or perceptions of school climate is inconclusive IMPLEMENTATION & SUSTAINABILITY CONSIDERATIONS • Consistency in school leadership is essential for sustainability • School staff are committed to continuing the work they started this year and agree it will take time for results to materialize • New principals described a challenging start to their work with the vendor and suggested more communication from the district prior to implementation would have been helpful • Messaging should be as cohesive as possible (across state, district, schools, and vendors) to reduce confusion for teachers • School structures that provide the time and support for teachers to meaningfully collaborate are important contributors to progress INTRODUCTION Washoe County School District (WCSD) is committed to investing in evidence-backed strategies to support its highest-need schools and students As such, it has invested federal Title I 1003(a) School Performance Support and state-funded Turnaround grant funding in two evidence-based providers for its Acceleration Zone, Comprehensive Support and Improvement (CSI), and Targeted Support and Improvement (TSI/ATSI) schools: The New Teacher Project (TNTP) and Solution Tree WCSD hired Social Policy Research Associates (SPR) to lead an evaluation of these two vendors’ supports on key outcomes of interest including school leadership practices, instruction, and student achievement in the district Building upon the knowledge gained in a previous study of Solution Tree support for WCSD schools, SPR conducted an implementation study that provides essential information on how these services were delivered, what influence they had on key outcomes of interest, and formative considerations for future school support activities, and an impact study that endeavors to estimate the impact, quantitatively, of these vendors on school climate and student academic and social-emotional learning (SEL) outcomes The implementation study primarily drew from two data collection strategies: interviews with key staff from a representative sample of schools, and surveys of principals and teachers at all TNTP and Solution Tree-supported schools in the district SPR planned to conduct site visits to WCSD schools in the spring of 2020, but due to school closures resulting from COVID-19, opted for virtual interviews instead Key research questions, developed with thought partnership from WCSD’s Research and Evaluation department, Title I District Coordinators, and TNTP and Solution Tree trainers, guided the implementation evaluation (Exhibit 1) Exhibit 1: Implementation Evaluation Research Questions Research Questions #1: What supports are TNTP/Solution Tree coaches providing to schools? #2: What is the level of buy-in and commitment among school leaders and staff to implementing the essential activities in the TNTP/Solution Tree models? #3: How and to what extent has TNTP/Solution Tree’s support affected school leaders’ capacity to serve as strong instructional leaders? #4: How and to what extent has TNTP/Solution Tree’s support affected teachers’ skills and capacity to implement strong, datadriven instruction? #5: What challenges WCSD schools face in adopting the essential activities in the TNTP/Solution Tree models? #6: What are the enabling conditions for high fidelity, successful implementation, and sustainability of the TNTP/Solution Tree models? Data Sources Document review, interviews, focus groups Interviews, focus groups, principal and teacher surveys SPR’s initial plan for the impact study was to conduct a quasi-experimental analysis of the impact of supports provided by TNTP and Solution Tree on student achievement, attendance rates, and disciplinary outcomes However, due to the closure of schools in WCSD between March and June 2020, students did not complete end-of-year assessments, which would have provided the data necessary for this analysis Instead, SPR leveraged existing School Climate Survey data to conduct a quasiexperimental analysis of the impact of working with a vendor on school climate indicators and student SEL skills Key research questions guided this analysis (Exhibit 2) Exhibit Quasi-experimental Analysis Research Questions Research Questions #7: Have TNTP/Solution Tree’s services contributed to changes in staff attitudes and school climate? #8: Have TNTP/Solution Tree’s services contributed to changes in student academic learning, engagement, attendance, and/or disciplinary outcomes? #9: To what extent have TNTP/Solution Tree’s schools met their goals for students’ growth and achievement? Data Sources School Climate Surveys School Climate Surveys (for student SEL skills only) N/A Overview of Data and Methodology As described above, this evaluation draws on both qualitative and quantitative data to understand the implementation of TNTP and Solution Tree supports, as well as assess progress toward WCSD’s intended outcomes These data provide information about the individual schools as well as progress overall and are described in more detail in Exhibit Exhibit 3: Data Sources and Methodology Data Source Document Review Interviews and Focus Groups (n=48) Description SPR reviewed documentation on TNTP and Solution Tree models of support for school improvement, as well as WCSD documentation on each vendors’ proposed scope of work for the 2019-2020 school year SPR reviewed progress updates from the district on which activities and supports had been implemented between September 2019 and December 2020 SPR conducted interviews with Title I District Coordinators (3), TNTP coaches (2), and Solution Tree coaches (6)1 in January 2020 to gain a deeper understanding of the ways in which trainers work with WCSD districts, to learn about the enabling conditions and key challenges schools face when working to improve, and to develop familiarity with the context and progress of the schools receiving support Then, in May and June 2020, SPR conducted interviews with principals (8) and school staff (29), selected from a sample of four TNTP schools and four Solution Tree schools as approximate There were 11 total vendor coaches working with WCSD schools in SY2019-2020 (two TNTP coaches and nine Solution Tree coaches) representatives of the schools working with the vendors, based on the following characteristics: • Level of supports (enhanced or regular) • Time working with vendor (one year or multiple) • Level (elementary or secondary) • School type (traditional public, alternative, or charter) • Baseline performance and progress (as determined by start-up interviews with vendor coaches and district staff) • Location & student demographics Consideration was also made for school staff capacity to participate, particularly in light of the constraints introduced by distance learning See Appendix A for a full list of schools selected and interviewees TNTP and Solution Tree Principal and Teacher Survey (n=301) Student and Staff School Climate Survey SPR summarized findings from this data collection activity in a brief memo, which can be found here SPR administered surveys to principals and teachers at WCSD schools that worked with TNTP or Solution Tree during the 2019-20 school year Surveys were tailored to the specific supports and practices used by each vendor but were aligned to enable analysis across vendors Four unique surveys were administered to the following groups: • Principals at TNTP-supported schools • Teachers at TNTP-supported schools • Principals at Solution Tree-supported schools • Teachers at Solution Tree-supported schools Surveys were administered via email in late May through early June 2020, prior to the conclusion of the school year The surveys instructed school staff to reflect on activities that occurred during the school year prior to distance learning, and as such, the findings reflect activities and experiences up until March 2020 A total of 727 teachers and principals were invited to participate in the survey, of which 41% responded See Appendix B for a full description of survey response rates SPR summarized findings from this data collection activity in a brief memo, which can be found here WCSD provided SPR with aggregate results from the annual School Climate Survey that the district has administered since 2011 The survey intends to provide schools with data that reflect components of school climate that support a positive learning and working environment and that promote academic success among all students The survey battery consists of three primary instruments: (1) Student Climate and Safety Survey; (2) Teacher and Staff Climate and Safety Survey; and (3) Family Climate and Safety Survey For this study, we analyzed a subset of pre-selected domains from the Student and Staff Surveys aligned to the vendors’ supports Responses to the surveys are anonymous and aggregated at the school level Guide to this Report The purpose of this report is to describe our findings from both the implementation study and impact study conducted over the course of the 2019-2020 school year Sections I, II, IIIA, and IV present themes from our interviews, focus groups, and TNTP and Solution Tree Principal and Teacher Survey (implementation data), while Section IIIB presents findings from our impact analysis • • • • Section I describes the models of support provided by TNTP and Solution Tree to selected schools during the 2019-2020 school year Section II explores the level of commitment and buy-in among school staff for the TNTP and Solution Tree models of school improvement Section III presents findings from our comprehensive analysis of vendor effectiveness at influencing intended outcomes o Part A focuses on perceptions of vendor effectiveness as assessed through our implementation study o Part B describes findings of vendor impact on school climate and student SEL skills Section IV describes common challenges, enabling conditions for successful implementation, and considerations for sustainability Across our data sources, many of our findings were consistent across the two vendors Unless otherwise noted, the findings we present apply to both TNTP and Solution Tree Where findings diverge, we highlight these differences using a green checkered text box SECTION I: TNTP AND SOLUTION TREE SUPPORTS TNTP and Solution Tree are education vendors dedicated to school improvement at the school and classroom level Both vendors provide school leaders with tailored supports to promote progress toward high-quality instruction and high student achievement In this section, we provide an overview of TNTP and Solution Tree’s models and the specific supports they provided to schools in WCSD during the 20192020 school year prior to distance learning (March 2020), and conclude with observations of differences in vendors’ approaches to school improvement TNTP TNTP is a national technical assistance and professional development provider that works at multiple levels of school systems to “advance policies and practices that ensure effective teaching in every classroom.”2 TNTP’s model focuses on providing school-level support in the following key areas:3 Rigorous Academics TNTP coaches help school staff examine if students are studying relevant, challenging, and engaging content Talented People TNTP works with school leaders to uncover if educators have the right skills in the appropriate role to help students succeed Supportive Environments TNTP coaches help schools assess whether schools have the supportive environments—defined as policies, systems, and communities—to support all students TNTP’s approach is grounded in solving problems that get in the way of equity, and tailors its model of support to the unique needs and goals of districts and schools WCSD hired TNTP in 2019 to begin working with nine of its CSI elementary schools Based on the results of school-level needs assessments, and through conversations with district and school leaders, TNTP designed a set of supports that focused on “improving teachers’ instructional practices and developing and supporting leaders to be strong instructional leaders who are equipped to develop their teachers and sustain instructional improvements on their campus.”4 TNTP SUPPORTS IN WCSD Consistent with the district-wide focus on developing strong professional learning communities (PLCs), TNTP’s support for WCSD schools included a particular focus on school-based collaborative teams as a vehicle toward school improvement The work began with leadership teams from each of the nine schools, comprised of administrator(s), instructional coaches, specialists, and teacher leaders, participating in a Summer Academy which served as the “launching off point” for the partnership Over the course of the eight-day Academy, leadership teams prepared for the new school year while also building instructional expertise Teams identified goals and priorities for the year that aligned with their School Performance Plans (SPP) around the following topics: (1) adopting the new ELA curriculum, (2) understanding demands, (3) TNTP-Supported Elementary Schools (SY2019-2020) Anderson Booth Canaan Desert Heights Duncan Echo Loder Mariposa Charter Matthews Natchez TNTP’s mission statement (https://tntp.org/about-tntp) As described on their website (https://tntp.org/what-we-do) TNTP Scope of Support & Proposed Calendar (2019-2020), internal document provided by WCSD using data, (4) developing foundational skills, (5) aligning questions and tasks, and (6) mapping instructional plans.5 School leaders were also provided with professional development to better support teachers around instructional priorities during the school year During the school year, principals participated in virtual and onsite coaching In addition, TNTP provided group learning experiences for school leadership teams as a follow up to their Summer Academy The school-year support focused on “building principal and site-based leadership team’s ability to be the instructionally-focused leaders needed for their school improvement efforts,” including developing leaders’ ability to: (1) observe classrooms to identify actionable feedback and norm instructional practices, (2) use data to make instructional decisions and analyze student work, and (3) develop an instructional vision.6 TNTP Key Priority Areas for WCSD Schools • • • High-quality Tier instruction, with particular focus on accessibility for English Learners Professional learning opportunities for teachers Implementing collaborative teams TNTP coaches also supported on-site data coaches at eight schools, focusing on building understanding of using English Language Arts (ELA) curriculum materials in service of “a strong vision of standards aligned literacy instruction.”7 Survey and interview data confirm that TNTP provided the full range of their supports to each of their participating school sites with few exceptions.8 Solution Tree Solution Tree is a global education publisher and professional development provider that aims to “transform education worldwide to ensure learning for all.”9 For the past several years, WCSD has adopted Solution Tree’s trademarked PLC at Work® model as a guide for teacher collaboration districtwide, and partnered with Solution Tree to provide school-specific coaching and support in a subset of its CSI and TSI/ATSI schools Solution Tree’s model of transforming a school into a professional learning community (PLC) requires a shift in adult mindsets and capabilities, and the structures to facilitate continuous improvement Solution Tree coaches provide support to selected school leaders in facilitating shifts in mindsets regarding student potential – that all students can learn at high levels – and towards collective responsibility for all students – from a focus on “my students” to “our students.” Coaches also provide support on using student data to adapt and improve educator practice, which is the vehicle for shifts in student outcomes To work toward these goals, Solution Tree coaches help organize schools into collaborative teams which hold regular, structured meetings organized around the following four questions:10 10 TNTP Scope of Support & Proposed Calendar (2019-2020), internal document provided by WCSD TNTP Scope of Support & Proposed Calendar (2019-2020), internal document provided by WCSD TNTP Update 121819, internal document provided by WCSD One principal indicated their school did not participate in group learning experiences on the TNTP Principal Survey Solution Tree’s Vision (https://www.solutiontree.com/about/overview) DuFour, R., DuFour, R., Eaker, R., Many, T W., & Mattos, M (2016) Learning by Doing: A Handbook for Professional Learning Communities at WorkTM • • • true for staff that began work with their vendor more recently, and because of the loss of the spring quarter due to the pandemic The amount of dedicated collaboration time and competing priorities have a large influence on teacher perceptions of the effectiveness of collaborative teams Teachers believe more uninterrupted time for collaborative meetings is necessary for progress of essential activities, and while many felt that they needed more time for meetings, almost as many felt that the time they have needs to be more focused District policies and resources play a role in supporting schools to redesign schedules that prioritize this collaboration time, and provide the needed supports (e.g., additional staff, aligned initiatives and training) Consistency in school leadership is important to carry the vision for the work forward and make necessary changes to support school improvement While new principals often bring with them the opportunity to make changes to school culture and structures, it takes time to cultivate the supportive environment that promotes trust and vulnerability that is the foundation for successful implementation of collaborative teams and other positive changes The data suggests that in this respect, principal tenure is more important than teacher and other staff tenure, as school leaders are the conduit for developing effective instruction in their staff, as opposed to training that comes directly from vendors or the district Interviewees suggest streamlining school improvement efforts could increase the effectiveness of vendor-supported efforts The structure of employing two vendors and three district liaisons caused some confusion, and interviewees suggest that streamlining by selecting one vendor and one district liaison would ensure consistency across schools and provide a more coordinated and sustainable effort Several school staff noted that in district meetings with their colleagues, it was challenging to collaborate because the focus of the work varied across schools, limiting their ability to discuss or share relevant best practices Consistent districtwide efforts would also ease transitions for school staff between different sites; for example, collaborative team norms and structures and frameworks for thinking about high-quality instruction would be consistent across schools CONCLUSION The findings from this evaluation point to strong implementation and early benefits of working with TNTP and Solution Tree School leaders (including both principals and school leadership teams) overwhelmingly reported that vendors positively impacted their own instructional leadership skills, and that their work together had benefits at the teacher level as well These teacher-level benefits include increased effectiveness of collaborative team meeting time, greater teacher buy-in for using student data to drive instruction, and strengthened beliefs in the ability of all students to meet high standards Among TNTP schools, work with the vendor translated into statistically significant positive impacts on school climate as measured by the Staff Climate Survey While the evaluation did not include student-level academic indicators of progress for schools served in SY2019-2020 due to COVID-19, the quantitative and qualitative results reinforce the quality of these vendors and their potential to positively influence school, classroom, and student outcomes Our hope is that this report equips district leaders with the data and considerations needed to continuously improve its approach and support of these vendor-school partnerships, including how to help schools sustain the work they began this school year into the future 33 TECHNICAL APPENDICES Appendix A: Interview & Focus Group Samples Exhibit A-1: Schools Selected for Qualitative Data Collection Vendor School TNTP Anderson Elementary Echo Loder Elementary School Mariposa Academy Natchez Elementary Dilworth STEM Academy Palmer Elementary Veterans Elementary Washoe Inspire Solution Tree Count of Interviewees 5 Exhibit A-2: Roles of Interview & Focus Group Participants Number of Participants Principals Teachers Other staff* 18 11 *Other staff include deans, coaches and specialists, and one social worker 34 Appendix B: TNTP & Solution Tree Principal and Teacher Survey Response Rates Exhibit B-2: Survey Response Rates Vendor School/Role Number of Staff Invited Number of Staff Responded Response Rate Solution Tree ALLEN ELEMENTARY SCHOOL 27 15 56% Principal 0% Teacher 26 15 58% BENNETT ELEMENTARY SCHOOL 31 13 42% Principal 0% Teacher 30 13 43% DILWORTH MIDDLE SCHOOL 43 13 30% Principal 1 100% Teacher 42 12 29% DRAKE ELEMENTARY SCHOOL 21 10 48% Principal 0% Teacher 20 10 50% LEMELSON K-8 S.T.E.M ACADEMY 22 23% Principal 1 100% Teacher 21 19% LINCOLN PARK ELEMENTARY SCHOOL 28 32% Principal 1 100% Teacher 27 30% O'BRIEN S.T.E.M ACADEMY 43 18 42% Principal 1 100% Teacher 42 17 40% PALMER ELEMENTARY SCHOOL 32 17 53% Principal 1 100% 35 TNTP Teacher 31 16 52% SPARKS MIDDLE SCHOOL 43 15 35% Principal 0% Teacher 42 15 36% STEAD ELEMENTARY SCHOOL 43 23 53% Principal 1 100% Teacher 42 22 52% SUN VALLEY ELEMENTARY SCHOOL 32 15 47% Principal 0% Teacher 31 15 48% TRANER MIDDLE SCHOOL 40 17 43% Principal 1 100% Teacher 39 16 41% TURNING POINT/PASS 11 64% Principal 1 100% Teacher 10 60% VETERANS S.T.E.M ACADEMY 28 32% Principal 1 100% Teacher 27 30% WARNER ELEMENTARY SCHOOL 22 41% Principal 0% Teacher 21 43% WASHOE INSPIRE ACADEMY 15 53% Principal 1 100% Teacher 14 50% ANDERSON ELEMENTARY SCHOOL 32 13 41% Principal 1 100% Teacher 31 12 39% 36 Total BOOTH ELEMENTARY SCHOOL 26 31% Principal 0% Teacher 25 32% CANNAN ELEMENTARY SCHOOL 31 13 42% Principal 1 100% Teacher 30 12 40% DESERT HEIGHTS ES 31 18 58% Principal 1 100% Teacher 30 17 57% DUNCAN S.T.E.M ACADEMY 31 14 45% Principal 0% Teacher 30 14 47% ECHO LODER ELEMENTARY SCHOOL 33 13 39% Principal 1 100% Teacher 32 12 38% MARIPOSA 11% Principal 1 100% Teacher 0% MATHEWS ELEMENTARY SCHOOL 40 12 30% Principal 1 100% Teacher 39 11 28% NATCHEZ ELEMENTARY SCHOOL 13 46% Principal 1 100% Teacher 12 42% 727 301 41% 37 Appendix C: CITS Methodology and Detailed Results To help ensure that the effect estimates reflect the effect of the intervention alone, CITS models have a comparison group to control for potentially confounding events The model calculates how the observed outcome deviates from the predicted outcome (based on the pre-period trend), but it does this separately for the group affected by the intervention (the treatment group) and for the group not affected by the intervention (the comparison group) The model then calculates the effect estimate by subtracting the comparison group deviation from the treatment group deviation The underlying assumption of a CITS model is that the confounding event (which occurred when the intervention began) affected the treatment and comparison groups similarly Under that assumption, subtracting the comparison group deviation from the treatment group deviation removes the effect of the confounding event The CITS regression model is: (2) 𝑌𝑎𝑡 = 𝛼 + 𝛽𝑡 + λ𝑇𝑎𝑡 + γ(𝑡∗𝑇𝑎𝑡) + Σ𝛿POSTk + Σθ(𝑇𝑎𝑡∗POST) + πX𝑎𝑡 + ΣζnSn + ε𝑎𝑡, where 𝑌𝑎𝑡 is the outcome for group 𝑎 (treatment or comparison group) at time 𝑡; 𝑡 is the time period centered at the last pre-period month; 𝑇𝑎𝑡 is a binary variable that equals for the treatment group and for the comparison group; POST is a binary variable that equals for the SY2019-2020 and otherwise; 𝛼 and 𝛽 equal the intercept and slope of the pre-intervention trend for the comparison group; (𝛼 + λ) and (𝛽 + 𝛾) equal the intercept and slope of the pre-intervention trend for the treatment group; 𝛿 represents the deviation from the trend for the comparison group in the post-period; and θ represents the estimated effect in the post-period—that is, the deviation from the trend for the treatment group minus the deviation from the trend for the comparison group X𝑎𝑡 is a vector of other time-varying school-level characteristics (school size, the percentage of students eligible for Free and Reduced-Price Lunch, the percentage of students with Individualized Education Plan, the percentage of students who are limited English proficient, the percentage of students who are female, and the percentage of students by racial group); ΣSn is a vector of school fixed effects (that is, a set of 0/1 dummy variables indicating which school the observation came from, where n is the number of schools included in the model; and ε𝑎𝑡 is an error term All the CITS models were estimated with clustered robust standard errors, which take into account the nested structure of the data.22 MULTIPLE COMPARISONS ADJUSTMENT Our CITS approach involved the estimation of 45 different models (for each survey domain, separate models were estimated for overall impact, TNTP impact, and Solution Tree impact) This raises the likelihood of estimates appearing as statistically significant only by chance, even if there is in fact no significant relationship This is known in the literature as a Type I error (by contrast, a Type II error is not rejecting the null hypothesis of no association even if, in fact, an association exists) A common way to deal with Type I errors is to adjust the p-values associated with each estimate We opted for a 22 Clustered robust standard errors are calculated in a way that takes into account that measurements for individual schools over time are highly correlated, which tends to artificially lower standard errors Clustered standard errors are less biased than regular standard errors 38 methodology that replaces the original p-values with sharpened False Discovery Rate (FDR) q-values.23 The reason to opt for this methodology was its flexibility—it allows a small number of type I errors to still occur in exchange for greater statistical power compared to familywise error rate (FWER) methods (such as Bonferroni adjustments) which prioritize the rejection of Type I errors at the expense of Type II errors COMPLETE CITS RESULTS Below we present the full model results from the CITS regressions The coefficients and associated adjusted p-values were already presented in the body of the report The full model results include additional information that allows readers and researchers to examine the strength of the evidence and include the findings in other research (for example, meta-analyses) Exhibit C-1: Staff Outcomes, Overall VARIABLES Impact estimate (sharpened q value) School-level covariates included Observations R-squared Number of schools (1) Expectations of Success (2) Fairness and Respect (3) Staff Collaboration (4) Staff-Student Relationships (5) Work Stress 3.214 3.939 -4.063 -0.387 -1.911 1.000 1.000 1.000 1.000 1.000 Yes Yes Yes Yes Yes 368 368 368 368 368 0.064 0.049 0.031 0.064 0.121 92 92 92 92 92 (5) *** p