NGC Final Evaluation Report Feb 8_2010

CENTER FOR STUDENT SUCCESS Ninth Grade Counts Evaluation Report Portland Schools Foundation Summer 2009 Acknowledgments This report has been made possible with the assistance of Nate Waas Shull representing Portland Schools Foundation, and Reese Lord representing the Office of the Mayor of the City of Portland (referred to throughout the report as project directors) They have worked diligently to ensure that all available data, materials, and records were expeditiously provided to the Center staff, and they were always available to provide additional explanation and insight into this vast collection of materials In instances when surveys, interviews, and/or focus group data required additional clarification, the communication and organization for these was always thorough and timely Without this level of support and follow through, this report could not have successfully been completed The extensiveness of this report is an indication of their willingness to commit their time in support of providing precision and thoroughness to this evaluation process Also, it should be noted that this Ninth Grade Counts project was successfully planned and implemented in an abbreviated timeline As decisions were made at the highest levels, individuals charged with the responsibility for setting these plans in motion immediately proceeded with the design and implementation required for the successful launch of the Ninth Grade Counts initiative Ultimately their work resulted in the successful implementation of this broad array of summer transition programs Again, careful follow-through and clear communication was pervasive throughout implementation Recognition of this careful, thoughtful, and thorough effort is requisite and fundamental to its continued successful implementation Center for Student Success Continuing Education/Graduate School of Education Portland State University PO Box 751 Portland, Oregon 97207-0751 503-725-9519 www.centerforsuccess@pdx.edu Report completed: February 2010 Table of Contents Section 1: Introduction 1.1 1.2 1.3 1.4 2-4 2-3 4-5 Introduction Executive Summary Methodology Evaluation Team Biographies Section 2: Program Background 2.1 2.2 2.3 2.4 Overview Key Materials Data Sources “Connected by 25” Research Synopsis Section 3: Impact of Program on Students 3.1 2009 Youth Corps Student Survey Results 3.2 Student Focus Group Results Section 4: Impact of Program on Services and Resources 6-15 6-9 10 11-13 13-15 16-30 16-26 27-30 4.1 Debrief Session – Program Providers 4.2 2009 Ninth Grade Counts Partner Program Survey Results 4.3 Project Directors Focus Group Results 31-50 31-34 35-46 47-50 Section 5: Conclusions/Recommendations 51-53 Appendix A Table: Cohort Graduation Rate by District (Class of 2008) Appendix B Total number of completing students by program and district Appendix C Student Focus Group Protocols Appendix D Ninth Grade Counts and Summer Youth Corps Debriefing Session Agenda Appendix E Project Directors’ Focus Group Protocols Center for Student Success Portland State University Section 1: Introduction 1.1 Introduction In October 2009, representatives from the Portland Schools Foundation and the City of Portland Mayor’s Office began meeting with representatives from Portland State University’s Center for Student Success (Graduate School of Education) to begin discussions leading to an agreement to provide third-party evaluation services related to the Ninth Grade Counts summer transition project The scope of the work became more clearly defined to include a review and analysis of existing data, surveys, archived documents, and available qualitative and quantitative data Further discussion of the requisite components of the Center’s evaluation process included data to be collected through student and program staff focus groups and interviews From the compilation and analysis of this extensive information and substantial set of materials, a composite of conclusions, insights, and recommendations for programmatic modifications and thoughtful next steps was to be prepared It should also be noted that the project directors retained the services of the Northwest Evaluation Association for quantitative analysis of participant demographics and performance data That analysis is on-going and results will be reported separately from this report What follows is the summary of findings related to program evaluation work completed primarily in 2009 The report is divided into the following sections for ease of accessing specific findings: overview of the project, impact of program on students, impact of program on services and resources, conclusions and recommendations, and appendices 1.2 Executive Summary This report examines Ninth Grade Counts, a network of 22 distinct summer transition programs targeting at-risk incoming 9th graders from Multnomah County’s six major school districts during the summer of 2009 The initiative, coordinated by the Portland Schools Foundation (PSF), represents a broad-based effort to increase high school graduation rates by helping more students start high school prepared to succeed The report also includes examination of Youth Corps – Exploration, an initiative of the Education Cabinet led by the Portland Mayor and the Multnomah County Chair, which provided college and career exposure opportunities for a subset of Ninth Grade Counts participants Detailed information on findings, recommendations, and methodology is included in this report Several key items from the report are highlighted below: • Nearly 800 students completed summer transition programs operated by 22 community-based organizations and school districts as part of the Ninth Grade Counts initiative; 420 of those students also took part in weekly career and college exploration trips through the Youth Corps – Exploration Center for Student Success Portland State University Section 1: Introduction • Approximately 500 participating students received high school credit through their participation in the program Both students and partner program staff indicated that the opportunity to earn credit is a key motivator for students to participate in the program • Leaders of the initiative worked with local school districts to identify the students most at-risk of dropping out by applying a common set of academic indicators to the incoming 9th grade class; programs worked to focus recruitment on these students • The initiative provided partner programs with significant in-kind resources to help increase their capacity These included 36 full-time AmeriCorps staff, 28 Masters of Arts in Teaching students from Marylhurst University, transit passes, student gift card incentives, and coordination and transportation for weekly career site visits – all totaling more than $550,000 of in-kind support • Students who participated in the career and college exploration experience reported overwhelmingly positive perceptions and impacts of the program Students agreed or strongly agreed that the program made them think more about their futures (83%); made them feel better prepared for 9th grade (70%); motivated them to graduate high school (78%); helped them think about new career and postsecondary options (66%); and made them think college was possible for them (82%) • Survey and focus group findings show that students were consistently motivated to participate in the program due to personal factors including the desire to improve themselves, have fun, and gain new experiences; encouragement from a family member or school advisor was also crucial; other motivating factors included to possibility of earning credit, and the promise of a transit pass or gift card incentive • Representatives of the 22 partner programs that form the Ninth Grade Counts network reported very positive perceptions of their participation in the initiative Staff from these partner programs agreed or strongly agreed that participating in the Ninth Grade Counts network had a positive impact on their programming (95%); increased their focus on incoming 9th graders (75%) and students at high risk of dropping out (60%); enhanced their services to youth (90%); and allowed them to serve more students (65%) 21 of the 22 programs indicated an intention to participate in the initiative again, and 100% of programs agreed that the effort was worthwhile and should be continued These represent a selection of high-level findings from this evaluation More detailed findings and recommendations for further development of the initiative are included in the following report Center for Student Success Portland State University Section 1: Introduction 1.3 Methodology For purposes of this evaluation, our methodology included a study of all evidence and associated materials that were provided; focus groups of students and project staff; in-depth analysis of student and program provider surveys; review of meeting notes and minutes; and the development of an understanding of the project’s historical background to include the work of key initiators and providers With an emphasis on qualitative research, this study focuses on the accomplishments of the Ninth Grade Counts 2009 project, while endeavoring to answer questions related to program meaning and relevance, program effects and impact, and accomplishment of stated goals and objectives Following the necessary protocols and design work associated with qualitative research has been important in this evaluation process, and the result is a valid and credible set of conclusions and recommendations intended for program improvement Again, this level of work was made possible with extensive assistance from project directors from the Mayor’s Office and the Portland Schools Foundation 1.4 Evaluation Team Biographies Linda Jessell, MA Center for Student Success, Director Portland State University Linda Jessell’s extensive educational background is both varied and comprehensive and includes 33 years in K-12 education combined with years of work in higher education In addition to her work in the Center for Student Success, Linda has most recently taught courses at both Portland State University and Lewis and Clark College Previous to joining the Center, Linda held several administrative positions for the Gresham-Barlow School District; Director of Curriculum K-12, Director of Secondary Schools, and Principal of Gresham High School Linda’s expertise lies in the areas of staff development, and curriculum and instruction Deborah Sommer, EdD Center for Student Success, Assistant Director Portland State University Deborah Sommer has a rich background in education of over 38-years in public education; as a staff development specialist, vice-principal, principal, curriculum director, school superintendent and university faculty Having worked in large and small school districts, suburban and rural schools, Center for Student Success Portland State University Section 1: Introduction and at an ESD, she has extensive experience in curriculum development, including power standards, curriculum mapping and program audits A specialist in team building and conflict resolution, contract negotiations and building parent and community support for schools, she was the recipient of the Oregon Superintendent of the Year award in 2006-Canby School District Richard Dills, EdD Educational Consultant Rick Dills has 32 years of experience as an educational practitioner Dr Dills was the director of student achievement for the Gresham-Barlow School District in Gresham, Oregon, a district serving 12,000 students in 20 schools that has received national and international recognition for its leadership in using Rasch-based assessment information to study and promote student growth His more recent assignments focused on curriculum, assessment, instructional technology and professional development Dr Dills has taught classes on performance assessment and educational measurement for Portland State University, and previously worked for the Oregon University System to develop its Proficiency-Based Admission System (PASS) As coordinator for standards and assessment for PASS, Dr Dills collaborated with university faculty and teachers from Oregon high schools to develop a set of educational standards that have been lauded as among the most comprehensive and meaningful in the country Center for Student Success Portland State University Section 2: Program Background 2.1 Overview As a foundation for the Ninth Grade Counts project in Multnomah County, the “Connected by 25” research base inspired community leaders to reach consensus around a commitment to focus on high risk teenagers in jeopardy of not completing their high school experience A deeper understanding of the issues and challenges associated with disaffected youth provided the motivation for this group of city, county, and school district leaders to collectively commit their political will around a program targeted toward supporting high risk teenagers as they transition from 8th to 9th grade The “Connected by 25” research (available at www.connectedby25.org ) drove the initiatives that emerged from the Mayor’s Office and the Portland Schools Foundation (PSF) during 2008 and 2009 PSF funds were committed to support high school interventions and a pilot summer program was in place during the summer of 2008 Lessons learned from the pilot included a need to provide clearer communication and to have agreements about expectations between PSF and program partners Most importantly, these lessons served to inform the design and processes that shaped the work on the development of the 2009 programs Simultaneously, the Mayor’s Education Cabinet committed to a multi-year initiative around three strategies These included: Ninth Grade Counts (students transitioning from 8th to 9th grade), Youth Corps - Exploration (incoming 9th graders participating in Ninth Grade Counts), Youth CorpsInternships (incoming 10th graders), and Youth Corps - Work Experience (incoming 11th and 12th graders) The implementation of the Ninth Grade Counts and Youth Corps – Exploration summer programs for incoming 9th graders launched alongside a pilot version of the 10th grade internship program, as well as a large-scale work experience program for older students funded through federal stimulus dollars and managed by Worksystems, Inc This evaluation focuses exclusively on the interventions for incoming 9th graders More specifically, the Ninth Grade Counts initiative emerged from PSF staff with the intent of connecting identified youth with the necessary support to begin high school on the right track The target number of students for participation in this program was 1,000 The Youth CorpsExploration, an initiative out of the offices of Mayor Adams and County Chair Wheeler, focused on 9th graders and was intended to provide college and career exploration to approximately 500 of those students participating in the Ninth Grade Counts program PSF, the Mayor and Chair’s offices, and Worksystems are currently working to further align this series of targeted summer programs Center for Student Success Portland State University Section 2: Program Background As these efforts progressed, involved parties developed a deepened knowledge and understanding of the consequences related to young people who not complete their high school education And, as a next critical step these leaders agreed upon a process for early identification of these high risk students The ensuing development of a common set of criteria set the stage for a coordinated effort on the part of six Multnomah County school districts to identify students as “academic priority” and then to begin the process of securing a commitment from them to participate in a summer transition program Noteworthy at this point in the project’s development was the work done by project directors (from the Portland Schools Foundation and the Mayor’s Office) to secure agreement across the county’s six major school districts to develop these common indicators of high risk eighth grade students, and then to commit to a concerted effort to secure their participation in one of the supportive summer transition programs These indicators, originally conceived and used in the Portland Public School District, correlated with the “Connected by 25” research data and provided the metric for early identification of eighth graders potentially at risk of dropping out of high school As each school district’s education strategy team moved forward with the process of student identification, the interpretation of these criteria varied to some degree, and some modification of indicators occurred Early in this process, the eighth grade students considered to be highly at-risk of dropping accounted for roughly 40% of the entire class This figure is consistent with some national studies of the high school dropout rate, particularly as it relates to some minority populations Given pervasive inconsistency and inaccuracy of reporting of dropout data nationally and locally, we recommend use of Oregon Department of Education (ODE) data for the “Cohort Graduation Rate” (see Appendix A) for the purpose of monitoring the progress of this first group of students participating in summer transition programs The data provides the necessary information related to each of the six districts’ graduation rates over time as well as a comparison to the statewide average More specific information related to progress over time for students participating in each of these programs should be targeted for monitoring within an on-going student data tracking system (e.g credits by grade level, attendance, GPA, etc.) Recruitment of students into summer transition programs was conducted either by school staff or by staff working in nonprofit community programs It became apparent during this recruitment effort that those adults who had the closest relationships with a particular student had the greatest impact on securing their commitment to participate At the school sites, counselors were critical to successful recruitment, and in the nonprofit partner programs adults who had developed close relationships with students and their families were most able to successfully recruit them into these summer programs Center for Student Success Portland State University Section 2: Program Background Although these community nonprofit programs were not legally able to access the school’s student information base (FERPA regulations) regarding academic priority students, a great deal of successful recruitment resulted from their knowledge of the students in the program and the relationships that they had developed with students’ families These trusting relationships served them extremely well as they proceeded with the recruitment process As consideration is given to future recruiting processes, other factors that to a somewhat lesser degree contributed to securing student participation included: incentives such as gift cards and TriMet passes, the Ninth Grade Counts guidebook, and opportunities for college and career visitations Of particular significance to this project’s success was the extensive level of community support and associated resources committed to the summer 2009 programs (Ninth Grade Counts and Youth Corps – Exploration) School districts and community-based organizations used their existing resources and programs to join the broader effort to support students through this key transition PSF and the Mayor/Chair’s offices also worked together to leverage additional resources to support these programs With an estimate of in-kind resources available to support students and families at a half-million dollar level, there can be no question as to the resolve of Multnomah County constituents around successful program implementation And given this level of commitment, it is also critical to the continued evolution of these programs that accountability and reporting systems are firmly in place to provide explicit and comprehensive data to program supporters related to student achievement and growth over time Review of intended outcomes provides for another means of analyzing the summer programs’ accomplishments Reaching as many of the county’s academic priority students (year targeted incoming 9th graders) as possible and involving them in one or more of the summer support programs was one targeted outcome The project came close to meeting the original goal of 1,000 students participating in a summer program Early indicators from program self-reports indicate that 830 students completed one program Further analysis showed that several students participated in more than one program A total of 773 unduplicated students completed the program (see Appendix B) The program component that includes career and college visitations reached 420 students Given the commitment to develop a student data tracking system, determining student attrition rates must be an important feature of the system in order to provide validation and accuracy to student participation numbers Another intended outcome was to increase the capacity and enrich the quality of summer transition programs Specific examples indicative of successful accomplishment of this outcome included: the staffing commitment from AmeriCorps, and the participation of Marylhurst MAT students, the provision of TriMet passes and student incentives, and the coordination and expense of the career and college site visits These enhancements to the original program design added viable enrichment and expansion opportunities for students Program providers in their survey completed at the conclusion of the summer work (2009) indicated that they had the capacity to involve more students and they also reported that they could become more creative with interventions Overall success related to this outcome is apparent given this increase in the Center for Student Success Portland State University Section 4: Impact of Program on Services and Resources decisions to participate in a summer program A further discussion of expanding course credit options could be important as future considerations are reviewed Critiquing the partner surveys and student surveys revealed some conclusions that will drive future survey design and implementation The student survey was actually an “after-thought” and did not provide all of the information project directors were looking for This led to more clarity regarding the next student survey which will include a pre/post survey format that evaluates attitudinal shifts and includes survey questions related to college and career site visitations The partner surveys provided valuable actionable data, and open-ended questions were very helpful Examples provided in support of this conclusion included: the importance of AmeriCorps, partners’ support for continuation of the Ninth Grade Counts program, and the reality that inclusion of the Marylhurst MAT students was not sufficiently structured As the Ninth Grade Counts program solicited and secured supporting resources, those strategies they found to be most successful used the “weight” of the Mayor’s Office and the Office of the County Chair for outreach They recruited 65 businesses for site visits in a very short period of time, and support from the Oregon Business Association also contributed to this success TriMet bus passes for student participants was another incentive added to the resource contributions and beneficially addressed student transportation challenges up-front AmeriCorps provided 36 full-time staff at no cost and these volunteers were both well trained and supported Although this resource may not be available for next summer’s program, it should be carefully considered as it provided a significant contribution to the project’s overall success Resources cited by the project directors as needing improvement or future review include: clarification of the role of the coach, better matching coaches to the site they visit with the student(s), providing business site personnel with a deeper understanding of the student population and the program goals, placement parameters for Marylhurst MAT students, gift cards for students, and transportation to and from the career and college sites Careful consideration of public relations strategies provides another area for review Clarification around the project and its “branded identity” will promote better communication from all directions Suggested strategies for enhanced public relations include: E-blasts from the Mayor and Portland Schools Foundation, the Mayor’s Facebook page, websites for Ninth Grade Counts and Youth Corps (to include a mechanism for tracking availability of program slots), public relations through Partner Programs, and grassroots/community newspapers and newsletters Materials that supported the project’s implementation included the Ninth Grade Counts Guidebook and the Coaches Handbook The Ninth Grade Counts Guidebook was used mainly for school counselors and parents It is perceived as an important resource to have available for constituents as it unifies the program and its goals and communicates these quite effectively Center for Student Success Portland State University 49 Section 4: Impact of Program on Services and Resources The Coaches Handbook was well received by program staff but appeared to have been used by coaches only sporadically Program staff reported that upon occasion they did not feel that the handbook content was reflected in the behavior of the coaches As a future consideration, this document should be more intentional and practical In conclusion, it must be noted that the speed at which the Ninth Grade Counts program was designed and implemented (complete with all the above-described components) is deserving of commendation Its contributions to the larger Portland community, specifically in relation to this young population of high-risk teenagers, should be recognized and celebrated and over time as this work evolves those adults who are primary contributors should be congratulated for their efforts Center for Student Success Portland State University 50 Section 5: Conclusions and Recommendations Conclusions and Recommendations To begin, it is of particular significance to note the extensive efforts that were in place in order to successfully implement the summer 2009, Ninth Grade Counts program The abbreviated timeline and its associated challenges were effectively overcome, and the program’s launch and implementation by all accounts made positive contributions to participating students Through the qualitative research study of this program conducted by PSU’s Center for Student Success, we are able to articulate a set of credible conclusions and recommendations specifically targeted toward future program improvement As the primary purpose of this project is to provide support for high-risk students as they transition from 8th to 9th grade programs, it is particularly important to have systems in place to track their progress over time and throughout their high school years We highly recommend that project directors continue working with local school districts to access student data, and that they use Oregon Department of Education data from the “Cohort Graduation Rate” as this most accurately tracks the same students over time as they participate in a four-year high school program By specifically using this data, program providers will be able to more precisely assess the effectiveness of their work and its ultimate impact on a cohort of students while intentionally excluding results related to students who have not had longer-term involvement And, to more accurately track students over time, project directors should continue working to develop a plan to expand the data tracking system to include on-going progress reports regarding student growth and achievement (e.g GPA, attendance, test data, etc.) This data directly contributes to program accountability with the community and with those who make significant contributions to the project’s resources Both student participation numbers as well as attrition numbers should be included in this expanded database A review of the recruitment process, particularly as it relates to those who most significantly impact student decision-making, indicates that parents and families have the single greatest impact on individual student commitment and decisions Also, trusting relationships with school counselors seemed to seriously impact students as they made their decisions regarding participation in summer transition programs Understanding that these two contributing factors have the greatest influence on student decisions, future recruiting efforts should more directly involve these key stakeholders Additional communication and public relations tools yet to be developed will require a more clearly defined brand and program identification process In particular, a focus on program goals, student results, and clarity about the “cost” of not working effectively with the community’s highrisk teenagers should all be carefully articulated in all supporting program documents Center for Student Success Portland State University 51 Section 5: Conclusions and Recommendations The opportunity to earn credit through summer program participation was consistently identified by students as providing an incentive for them to participate Knowing this, careful consideration should be given to expanding the availability of academic credit for students “Academic priority” students earning this advantage as they enter their ninth grade year could potentially experience more success As we know, credit deficiency is a key to students deciding to not complete their high school program And, by providing high-risk students with earlier credit opportunities the likelihood is increased that 9th grade credit deficiencies could be overcome The student survey process is another component that should be re-examined particularly given that its results revealed important insights into students’ summer program experiences In the future it will be important to move to a pre/post survey format (using a “forced choice” Likert format) in order to track student attitudinal shifts, and to provide more in-depth feedback related to specific programmatic components (e.g college and career visits, curriculum, activities, etc.) As future planning takes place, a process for elicitation of feedback from parents/family members through interviews (or focus groups) must be included In order to more broadly and deeply understand the contributions of the various programs to successes that students may be experiencing (or not experiencing), parental input will provide a critical piece Knowing the importance of parental and family contact, efforts should be in place to communicate directly with families during the recruitment process and throughout the summer program, in addition to soliciting their feedback at the conclusion of their student’s summer experience Another avenue for feedback that will be important to future program development is the forum provided for the program providers on August 20th, 2009 This organized session included opportunities for these participants to comment, react, and express opinions related to their experiences over the course of the summer The results revealed substantive information that will directly contribute to long-term program improvements A forum for program providers’ feedback should be continued and possibly expanded in the future The other feedback loop that contributes valuable information for project directors is the Partner Program Survey Emerging from this survey data was the unanimous conclusion that Ninth Grade Counts and Youth Corps made a difference for participating students and included a recommendation that these programs be continued This fact is of such significance that it should be included in all future program communication tools In particular, it should be included in those that solicit resources, those that recruit students, and those that account for program progress over time Modifications to this survey in the future should include a careful examination of partners’ needs and perceptions, and a method for ensuring survey consistency in administration and in tallying results Center for Student Success Portland State University 52 Section 5: Conclusions and Recommendations A timeline of data gathering activities and expectations for all program providers relative to them should be developed as soon as possible and well in advance of summer 2010 In addition, at the orientation meetings involving program providers to take place this spring, it is recommended that a review of all the evaluation components be included so that all participants have a clear picture of what data is to be collected How changes were made for 2010, based on 2009 feedback from providers and participants, should also be highlighted so that all stakeholders begin to develop a sense of ownership for program outcomes Final conclusions related to the overall program reveal significant successes and commendation is warranted in relation to all contributors to this project Those worthy of commendation should include: project directors from the Portland Schools Foundation and the Mayor’s Office, all individuals and agencies who contributed actual and/or in-kind resources in support of program implementation, program providers who delivered direct services to participating “academic priority” students, AmeriCorps and Marylhurst MAT students who provided additional program oversight and support, and all other coaches and/or volunteers contributing to this success Of equal importance to the successful implementation of the Ninth Grade Counts program are the significant contributions made by families and their students who made a serious commitment to participating in the summer transition program experience in order to build for success as they embark upon their high school career Again, on behalf of Portland State University’s Center for Student Success we wish to extend our thanks to project directors for this opportunity to work in partnership with them on this program evaluation We are pleased to be associated with such a quality effort and such significant work on behalf of Portland-area youth Center for Student Success Portland State University 53 Appendix A Table: Cohort Graduation Rate by District (Class of 2008) Appendix B Total number of completing students by program and school district Appendix C Student Focus Group Protocols Appendix D Ninth Grade Counts and Summer Youth Corps Debriefing Session Agenda Appendix E Project Directors’ Focus Group Protocols Center for Student Success Portland State University Appendix APPENDIX A Cohort Graduation Rate by District (Class of 2008) District Cohort Graduation Rate (Class of 2008) Centennial 65% David Douglas 62% Gresham Barlow 72% Parkrose 63% Portland Public 54% Reynolds 51% Oregon statewide average 68% - Data from the Oregon Department of Education Center for Student Success Portland State University Appendix APPENDIX B Ninth Grade Counts 2009 - Total number of completing students by program and school district Total students Program Name completing* Big Brothers Big Sisters Campfire USA - Xploregon 23 David Douglas School District - Ninth Grade Counts 49 El Programa Hispano - Puentes 60 I Have A Dream Foundation 15 IRCO - ASPIRE-SST 19 IRCO - MCSI IRCO - Sabin SUN (plus Brothers & Sisters Keepers) 28 IRCO - SSSES 22 Metropolitan Family Service 12 NAYA - Ninth Grade Summer Leaders 14 Neighborhood House 14 Oregon Building Congress - Pre-ACE Academy Parkrose School District - ELL Summer Program 22 Parkrose School District - Summer Stampede Academy 98 Portland Public Schools - 8th-9th Grade Summer Program 77 REAP, Inc - Challenge Camp 22 Reynolds School District - Ninth Grade Summer Transition 62 Self-Enhancement, Inc 64 Step Up 184 Straightway Services 20 Urban League 11 TOTAL 830 District Name Centennial David Douglas Gresham-Barlow Parkrose Portland Reynolds Non-participating district or private school Total students completing a Ninth Grade Counts partner program* 13 70 55 133 453 83 23 TOTAL 830 * NOTE - these figures include duplication of students who particiated in more than one program Number served in district-sponsored programs: 308 Number served in CBO-sponsored programs: 522 Center for Student Success Portland State University Appendix APPENDIX C Ninth Grade Counts Student Focus Group – Protocols December 9th and 11th, 2009 12:00 – 1:00 Madison High School and David Douglas High School I Getting started  Introductions (who we are, what we are doing and why, who they are and which program(s) did they participate in last summer, etc.)  Describe the focus group process, length, participation and expectations, how this information will be used  Review confidentiality II One of the main purposes of your summer program was to help you to be better prepared for 9th grade What thoughts you now have about that? What happened in the program that helped you to be better prepared? What you wish would have happened in the program to have better prepared you? III Why did you choose to participate (what was your motivation)? Did someone you know encourage you to become involved? Who was this and what did they do/say to convince you to participate? IV How is your 9th grade year going so far? What grades you think that you will be getting first semester? Talk about your attendance (are you having trouble getting to school, staying there and attending all classes) What would have helped you during the summer program with your grades and attendance? V One of the goals of this summer program was to help you to be on track toward graduation at the end of your 9th grade year This means that you will have to have passed all your required courses So, you’ve talked about your progress first semester, how you think second semester will go for you? If you’re on track, how will you stay on track? If you’re not on track, how are you planning to get back on track (what will you differently)? VI Who is (are) the adult (adults) who you think have helped you the most with staying on track in high school? Did anyone from your summer program this for you? Are you continuing to stay in touch with them? If so, how? If not, why not? VII At this point in your high school program, you plan to complete high school? What things might get in the way of this goal? Center for Student Success Portland State University Appendix VIII At this point in time what are your thoughts about continuing with your education after high school (college, career training, etc.)? Talk briefly about adults who are actively contributing to helping you think about these things IX Your summer program included some college and career site visits How did you feel about these – were they helpful to your planning around your future? If so, in what ways? If not, why not? X Are there other thoughts that you have now that we have had this conversation that you would like to have others be aware of as they continue to plan for future summer programs? Center for Student Success Portland State University Appendix APPENDIX D Ninth Grade Counts and Summer Youth Corps Debriefing Session & Celebration August 20, 2009 – 9:30-12:30 NAYA Family Center Meeting goals: - Provide an opportunity to share successes and lessons learned from our shared effort this summer - Gain information to improve future efforts - Recognize the contribution of volunteers Attendees Community-based and school district-based summer program staff and administrators, Youth Corps coaches, AmeriCorps members, Marylhurst students, staff from the Youth Corps and Ninth Grade Counts initiatives, and other interested individuals Time 9:30 9:40 9:45 Activity Welcome & Introductions Include initial welcome from a NAYA rep Overview of meeting goals Overview of the summer initiatives - Purpose of the initiatives (brief) - Review visual diagram from their packets - Briefly take general questions 9:55 BREAKOUT GROUPS – concurrent groups Facilitator Todd & Nate Todd Nate FACILITATOR NOTE: Before ending your session, request one person from your group to provide a 3-5 minute report out to the large group on your group’s 1) successes, 2) challenges, and 3) hopes for the future Group 1: Youth program providers Facilitator: Nate FIRST – Take time to complete the provider survey (15 minutes max) Recruitment: 1) How successful you feel your program was in recruiting students (numbers, targeted groups, etc.)? 2) Were Ninth Grade Counts and Youth Corps resources helpful in your recruitment (staff support, Guidebook, incentives, field trips, etc.) 3) How might Ninth Grade Counts and Youth Corps be more helpful in your future recruitment/engagement of students? Center for Student Success Portland State University Notes: Rebecca Appendix AmeriCorps/Marylhurst resource: 1) Describe the impact on your program of having this support 2) Describe any challenges you faced in working with an AmeriCorps associate or Marylhurst student 3) How might the process be improved for the future? 4) Include time for any additional feedback Youth Corps 1) Describe your experience of working with the volunteer coaches What was their value? What changes would you recommend? 2) How would you describe the communication you received about Youth Corps? What could be improved? 3) What are the qualities/elements of the site visit that added value to your youth? To your program? (please consider both career and colleges separately) 4) What are the qualities/elements of the site visit that were least effective? 5) How you think the site visits could be improved? Group 2: AmeriCorps associates and Marylhurst students 1) In groups of two, describe your overall experience in working at a Ninth Grade Counts/Youth Corp program site this year 2) In large group, describe how prepared you felt you were for your experience, and whether there is something you wish you had known or done prior to starting your placement 3) Describe how supported you felt in your work by your host site 4) Describe your experience of interacting with students who may have been facing significant barriers 5) Describe anything that surprised you during this experience 6) What additional feedback could you provide to Ninth Grade Counts/Youth Corps to help make this experience even better for future participants 7) Include time for any additional feedback Facilitator: Todd D Notes: Carol D Facilitator: Emily G Notes: Samantha L Group 3: Youth Corps Coaches 1) How did you come to be a coach? 2) Tell us about your experience with the program providers 3) Describe your experience interacting with the students 4) What impact you think you had on the youth? 5) What tools/training did you receive that you benefited most from? 6) What tools/training would have been more helpful? 7) What are the qualities/elements of the site visit that added value to Facilitator: Todd D the youth? What didn’t work? Center for Student Success Portland State University Appendix 8) How would you describe the communication you received? Notes: 9) What is the biggest thing you are going to take away from this experience? 10) Is there anything you are going to change or differently as a result? 11) Do you intend on staying connected with the program and the youth? 11:05 11:15 11:40 12:00 12:30 BREAK – Reconvene in main room (to start back up right at 11:15) Brief report outs from each group (Include a few highlights of the experience, challenges, and hopes for the future – about minutes for each of groups) Volunteer recognition (certificates for AmeriCorps, Marylhurst, and coaches) Lunch and Celebration END Center for Student Success Portland State University Todd & Nate Dan R & Sam A Appendix APPENDIX E Ninth Grade Counts Project Directors Focus Group Protocols November 20, 2009 Purpose of interview: to gather information for the purpose of writing a history of the 2009 network of summer transition programs ***(Talk about the confidentiality protections) What are your thoughts (reflectively) in relation to the process used to develop common criteria in order to identify for/with the districts their “academic priority” students? Were the criteria successful and/or accurate in terms of the students identified for the program? Was the process used successful and useful – would you use a different process in the future? How successful you think that the recruiting process was – why? Are there changes that you think would be important to make in this process? What were the common set of student data tracking practices that were used? Were they common across all districts/schools? How did this all work? One of your identified “outcomes” was to enroll as many as possible of the academic priority students into summer support programs In retrospect, did this occur; to what degree did this occur; did you actually reach your specified goal of 1,000 students (on your “Resource Description and Request Form” document you indicate the target number as 2,500 – which was your actual goal? How close did you actually get to this? Another identified outcome was to increase the capacity and enrich the quality of summer transition programs From your perspective, did this occur? If so, specify why and provide some specific examples If not, why not – and what might you have done to improve on this issue? Ninth Grade Counts had a year objective to have 500 ninth graders participating in weekly career and college visits Did this occur? If so, was it successful – why you think this was the case? If not, why not? How did the effort work related to students securing credits with the 9th grade counts partners? Were there obstacles with this – why? Center for Student Success Portland State University Appendix What comments might you want to make regarding the partner surveys and the student surveys? You incorporated a variety of strategies to solicit and secure resources in support of this program Which strategies you think were the most successful – least successful? Why? Which you think will be the most important to continue using? 10 As you think about the various public relations strategies that you used, which you think were the most successful – the least? Why? Which you intend to replicate and use again? 11 You produced and distributed guidebooks – the Ninth Grade Counts guidebook that appears to be for students (is this correct? Could it also be used by counselors/school staff?), and the Coaches handbook? Talk about how effective people found these to be, and from your perspective how you might be thinking about proceeding with these types of materials In other words, what changes (if any) you have in mind for these products? Center for Student Success Portland State University Appendix ... www.centerforsuccess@pdx.edu Report completed: February 2010 Table of Contents Section 1: Introduction 1.1 1.2 1.3 1.4 2-4 2-3 4-5 Introduction Executive Summary Methodology Evaluation Team Biographies... Northwest Evaluation Association for quantitative analysis of participant demographics and performance data That analysis is on-going and results will be reported separately from this report What... with items related to NGC? ??s positive impact on their programs, as follows:    95% agreed that NGC had a positive impact on their programming (item 3) 75% agreed that NGC increased their focus