Evaluating-PFA-Effectiveness-Research-Brief

Research Brief EVALUATING PRESCHOOL FOR ALL EFFECTIVENESS This brief summarizes findings of the effect of Preschool for All participation on school readiness skills August 2013 1871 The Alameda, Suite 180 San Jose, CA 95126 Tel: 408.247.8319 Fax: 408.260.7749 TABLE OF CONTENTS Introduction Methods Sampling and Recruitment Schools and Classrooms .3 Participant Consent .3 Instruments .4 Child Assessment Measures Self-Regulation .5 Parent-Completed Measures Administration of Child Assessments Completion Metrics .10 Languages Used by Participating Children .11 Key Findings: Approaches to Evaluating PFA Effectiveness 12 Method 1: Comparing PFA Graduates and Non-Graduates 13 Home Language 13 Parents’ English Language Proficiency .14 Family Income 14 Parental Education 15 Job Loss 15 Kindergarten sample differences between PFA and Non-PFA families .15 Method 2: Longitudinal Examination between Preschool and Kindergarten 16 Method 3: Regression Discontinuity Design 17 Sample 18 Findings 18 Results by Language of Assessment 25 Summary 27 Appendix A Letter-Word Recognition RDD Scatterplot .28 Appendix B Applied Problems RDD Scatterplot 29 Appendix C Head-Toes-Knees-Shoulders RDD Scatterplot 30 EVALUATING PFA EFFECTIVENESS R EPORT B RIEF This report describes three methodological approaches employed in examining the effect of PFA on children’s readiness for school and reports on results of findings INTRODUCTION In March 2004, San Francisco voters passed Proposition H, which stated that, “It shall be the policy of the City and County of San Francisco to provide all four-year-old children who are City residents the opportunity to attend preschool,” and First San Francisco was designated as the implementing agency First San Francisco first began to offer Preschool for All (PFA) in four pioneering neighborhoods in 2005 and has since expanded to reach every corner of the city According to the First San Francisco Strategic Plan and the Evaluation Framework, primary objectives in San Francisco include: • The provision of high-quality preschool is accessible, available, and affordable for all, and that • Children are socially, emotionally, and academically prepared to succeed in school PFA is approaching the end of its ten-year funding authorization and First San Francisco has partnered with Applied Survey Research (ASR) to conduct an outcome evaluation to help PFA determine how effective it has been in achieving these objectives In an early phase of the evaluation, ASR conducted a series of interviews and focus groups with various PFA stakeholders, policy leaders and advocates in the field of Early Childhood Education in San Francisco to identify the primary outcomes that stakeholders believed PFA should demonstrate Consistent with the PFA Logic Model and First San Francisco’s Strategic Plan, stakeholders identified child outcomes and ECE system outcomes as being the critical for determining how successful PFA has been in San Francisco Through a series of individual child-level assessments conducted in fall 2011 and fall 2012, this report examines the effectiveness of Preschool for All in preparing children for school, as guided by the following research questions: • • How does PFA participation influence children’s readiness for school? How does PFA participation influence the language and literacy development, particularly of dual language learners? San Francisco City Charter SEC 16.123-2 Public Education Enrichment Fund First San Francisco 2010 Request for Proposals: Preschool for All Outcome Evaluation Planning and Design Contractor METHODS SAMPLING AND RECRUITMENT The study sample was drawn at the classroom level in 2011 and 2012 Pre-K classroom sampling was stratified by auspice to ensure accurate representation of classrooms from San Francisco Unified School District, San Francisco State University, family child care homes In 2011, the sampling target was to include 500 PFA preschool students in the assessments All Preschool for All (PFA) classrooms were included in the sampling frame and a random sample of 42 classrooms stratified by auspice was selected to obtain a targeted sample of 500 students The 2012 target for the regression discontinuity design sample was to include 100 classrooms to reach a total of 2,000 children to assess, which would include an oversample of kindergartners in an attempt to increase the odds of including a PFA graduate in the sample All PFA classrooms were included in the sampling frame and a stratified random sample of 46 classrooms was selected by auspice to achieve a targeted 500 students All Early Education Elementary Schools were included in the kindergarten classroom sampling frame and kindergarten classrooms were randomly selected for inclusion in the sample SCHOOLS AND CLASSROOMS As part of their contractual agreement with First San Francisco, the preschool classrooms sampled are obliged to participate in evaluation activities These included 42 preschool classroom sampled in 2011 and the 46 preschool classrooms that were sampled in 2012 In the San Francisco Unified School District, recruitment efforts in late summer into early fall of 2012 included an initial email to principals from the First San Francisco Preschool for All Program Coordinator introducing them to the study and inviting them to participate by replying to the email request with the names and contact information of their kindergarten and/or transitional kindergarten classroom teachers Through the support of the Executive Director of Instructional Support at SFUSD, a list of teacher names and estimated class enrollment was provided in late August 2012 Using this list, the principal emails were followed up by letters sent to each individual kindergarten and transitional kindergarten teacher All Early Education Elementary schools were called to obtain kindergarten and transitional kindergarten teachers’ email addresses, to which introductory email messages were also sent A total of 48 kindergarten and transitional kindergarten classrooms agreed to participate in the study PARTICIPANT CONSENT As kindergarten teachers agreed to participate, enrollment numbers were confirmed and packets of consent forms were prepared for teachers to distribute to the parents of the students in their classrooms Each child was sent home with a packet of materials for their parents which included a letter describing the study and an active consent form attached to a self-addressed, postage-paid envelope for parents to return forms directly to ASR Up to three rounds of forms were sent to classrooms in attempts to increase consent rates and children’s books were sent along with consent forms to thank parents and families for their time and consideration At the PFA preschools, because evaluation activities are a part of each site’s obligation, passive consent was implemented such that teachers of sampled classrooms distributed packets to parents that included a letter describing the study and providing an opt-out option, and two forms that parents were requested to complete and return (described in the next section) via the self-addressed, postage-paid envelopes included Children’s books were included in preschool parent packets as well INSTRUMENTS The same battery of assessment instruments was administered in 2011 and 2012, with the exception of the ROWPVT-Spanish-Bilingual Edition, which underwent a version change between the 2011 and 2012 administrations This is described in greater detail below CHILD ASSESSMENT MEASURES In identifying appropriate methods of assessing children’s school readiness, several researchers and policymakers nationwide rely on the guidelines outlined in the National Education Goals Panel (NEGP) Given these guidelines, four tools were selected for administration with students to measure the impact of PFA on school readiness outcomes In addition, a recent study adjudicating the relative importance of various early education efforts finds that of the myriad measures examined in their analysis of population-based datasets involving over 16,000 children, only three of the school-entry measures predicted subsequent academic success: early reading, early math, and attention skills Thus, the selected measures encompass skills related to language development and early literacy, early math skills, and self-regulation L ANGUAGE D EVELOPMENT AND E ARLY L ITERACY Early language and literacy skills that develop during the preschool years are important foundations for later literacy, and research has demonstrated links between early literacy and academic achievement, graduation rates, and enhanced productivity in adult life The Receptive One-Word Picture Vocabulary Test-4 (ROWPVT) is a 15-20 minute test of receptive vocabulary that is available in English and Spanish-Bilingual (SBE) editions The ROWPVT and ROWPVT-SBE consist of a series of test plates, each with four pictures on them The examinee is shown a test plate and asked to identify which of the four pictures describes a stimulus word The instruction given to the examinee is: “I am going to say a word and show you some pictures, and I want you to point to or tell me the number of the picture that matches the word.” Testing is stopped when the examinee responds incorrectly to six out of eight consecutive items The ROWPVT Spanish-Bilingual Edition is administered initially in the child’s preferred language The assessor says to the examinee in English or Spanish: “I am going to show you some pictures, and I want you to point to (or tell me the number of) the picture that is the same as the word I say Sometimes I may say the words in Spanish and sometimes I may say the words in English.” If the child misses an item in his/her dominant language, the item is administered again in the non-dominant language If the child chooses the same image again, the child is prompted to try again If the child responds correctly in either language, the item is counted as correct The language the child responds correctly in is logged on the scoring sheet Testing is stopped when the examinee responds incorrectly to four out of six consecutively administered items A raw score is derived by counting the total number of correct responses up to the last item administered Conversion charts in the ROWPVT and ROWPVT-SBE manuals provide standard scores, age-equivalent scores, and A complete report of findings from the 2011 assessments may be found in a separate report: Preschool for All Outcome Evaluation: Child Outcomes Preliminary Data Memo Lorrie Shepard, Sharon Lynn Kagan and Emily Wurtz, Eds., Principles and Recommendations for Early Childhood Assessments (Washington, D.C.: National Education Goals Panel, 1998) Duncan, G., C.J Dowsett, A Claessens, K Magnuson, A.C Huston, P Klebanov, L S Pagain, L Feinstein, M Engel, J Brooks-Gunn, H Sexton, & K Duckworth 2007 School Readiness and Later Achievement Developmental Psychology, 43(6): 1428-1446 Dickinson, D.K., and P.O Tabors (Eds.) 2001 Beginning Literacy with Language: Young Children Learning at Home and School Baltimore, MD: Paul D Brookes; Missall, K., A Reschley, & J Betts 2007 Examination of the predictive validity of preschool early literacy skills School Psychology Review, 36, 433-452 Martin, Nancy and Rick Brownell 2011 Receptive One-Word Picture Vocabulary Test – Novato, CA: Academic Therapy Publications Brownell, Rick 2001 Receptive One-Word Picture Vocabulary Test – Spanish Bilingual Edition Novato, CA: Academic Therapy Publications percentile rank Standard scores are based on a normal population distribution having a mean and median of 100 and standard deviation of 15 Thus, a standard score of 100 indicates performance is at the midpoint for a particular age group Percentile ranks reflect one’s performance relative to that of the normative population For example, a percentile rank of 53 indicates the student scored as well as or better than 53 percent of same-aged students Conversely, this also means that 47 percent of students that age scored higher Because the procedures for administration differ for the two instruments, the raw score conversions to the other types of scores are calculated separately for each of the instruments Moreover, in 2011, the ROWPVT-SBE version was still in beta testing As such, in fall 2012, the older, norm-tested version was used, but changes between the two Spanish-Bilingual versions used render direct comparisons across tools unreliable As such, reports of findings will include only descriptive results for the ROWPVT-SBE across years and mean comparison will not be made The Letter-Word Identification subscale of the Woodcock-Johnson III Tests of Achievement and Bateria 10 III Woodcock-Muňoz Pruebas de Aprovechamiento were used to measure early literacy (pre-reading and word decoding) skills in English- and Spanish-speaking students, respectively The test consists of a series of test plates in which examinees are asked to accomplish a task using the pictures or symbols on the test plate Children are assessed in identifying letters, words, and distinguishing letters from pictures For example, the first plate consists of the letters “P”, “O”, “B”, and “K” and the examinee is asked to identify the “P” The test typically takes about five minutes for a child to complete E ARLY M ATHEMATICS In the widely-cited Duncan, et al study cited above, early math skills were shown to be the single strongest predictor of later academic success In addition, a follow-up study found that K-5 students with persistently low 11 math skills were less likely to finish high school and less likely to go to college A second Woodcock-Johnson subtest, the Applied Problems subscale of the Woodcock-Johnson III Tests of Achievement/ Bateria III Woodcock-Muňoz Preuebas de Aprovechamiento was used to assess mathematical skills The test is administered in English or Spanish, and it also takes approximately five minutes to complete Items include tasks such as showing two fingers, counting objects, and adding or subtracting small numbers For example, the first illustrated test plate item shows a picture of a dog and a mug and the examinee is asked, “How many dogs are there in this picture?” Testing on Woodcock Johnson tools is completed when consecutive items are answered incorrectly Raw scores are derived by counting the total number of correct responses Conversion charts on the testing instrument provide age- and grade-equivalent estimates based on the raw score SELF-REGULATION While specific definitions vary, behavioral self-regulation generally refers to the ability to concurrently inhibit impulses while executing intention, and is widely referred to as the behavioral manifestation of attention, working 12 memory, and inhibitory control which derive from executive functions Self-regulation skills become particularly important in the transition to kindergarten, as they enable children’s successful adaptation to the demands of the Woodcock, Richard W., Kevin S McGrew, and Nancy Mather 2001 Woodcock Johnson-III Tests of Achievement Itasca, IL: Riverside Publishing Muñoz-Sandoval, Ana F., Richard W Woodcock, Kevin S McGrew, and Nancy Mather 2005 Batería III Woodcock-Moz: Pruebas de habilidades cognitivas Itasca, IL: Riverside Publishing 11 http://phys.org/news/2011-04-early-math-skills-academic-success.html 12 von Suchodoletz, et al 2013; Wanless, S.B., M.M McClelland, S.L Tominey, & A.C Acock 2011 The influence of demographic risk factors on children’s behavioral regulation in prekindergarten and kindergarten Early Education and Development, 22: 461-488 10 classroom environment Indeed, a growing body of international literature finds that self-regulation skills provide a 13 foundation for positive school experiences and predict later academic achievement 14 The Head-Toes-Knees-Shoulders Task (HTKS) is a structured, five-minute interactive assessment that assesses a child’s ability to self-regulate by asking children to perform the opposite of a response to four different commands The test was administered in English, Spanish, or Chinese The test is administered with the child standing and, for example, when the examiner says, “Touch your toes,” the child is meant to the opposite, which in this case, is to touch his/her head When the examiner says, “Touch your head,” the child is meant to touch his/her toes The four possible commands and correct opposite responses are: • Touch your head (correct response is child touching his/her toes) • Touch your toes (correct response is child touching his/her head) • Touch your knees (correct response is child touching his/her shoulders) • Touch your shoulders (correct response is child touching his/her knees) A child receives two points for correctly executing each command, and one point for a self-correction That is, a child may receive one point if, for example, when instructed to “Touch your toes,” the child initially reaches for her toes, but then stops and corrects herself and touches her head Total scores range from to 40 points possible PARENT-COMPLETED MEASURES In addition to the readiness assessments conducted with the children, there were two parent- completed supplemental instruments, described below P ARENT S URVEY In order to provide greater context within which school readiness develops, the Parent Survey was sent home to parents with the initial letter seeking consent for their child’s participation in the evaluation This survey was designed to collect background information on the child’s family, including some demographic information, economic indicators, general health and well-being, parenting practices and attitudes, and daily family and household routines P RESCHOOL AND K INDERGARTEN B EHAVIOR S CALES (PKBS) Designed for children ages 3-6, the Preschool and Kindergarten Behavior Scales (PKBS) was enclosed with the letters sent to parents The PKBS is designed to be completed by parents or caregivers and provides an appraisal of children’s social skills as well as problematic behavior The tool is comprised of the Social Skills scale, which includes measures of Social Cooperation, Social Interaction, and Social Independence, and the Problem Behavior scale, which includes measures of problematic behaviors characterized as Externalizing Problems (includes measures of self-centeredness, anti-social behavior, and attention difficulties) and Internalizing Problems (includes measures of social withdrawal and anxiety problems) A summary of all of the data collection instruments used in the PFA outcomes study is shown in the following table Blair, C., & R.P Razza 2007 Relating effortful control, executive function, and false belief understanding to emerging math and literacy ability in kindergarten Child Development, 78: 647-663; McClelland, M.M., A.C Acock, & F.J Morrison 2006 The impact of kindergarten learningrelated skills on academic trajectories at the end of elementary school Early Childhood Research Quarterly, 21: 471-490; McClelland, M.M., C.E Connor, C.L Farris, A.M Jewkes, & F.J Morrison 2007 Links between behavioral regulation and preschoolers’ literacy, vocabulary and math skills Developmental Psychology, 43: 947-959; von Suchodoletz, A., S Gestsdottir, S.B Wanless, M.M McClelland, F Birgisdottir, C Gunzenhauser, & H Ragnarsdottir 2013 Behavioral self-regulation and relations to emergent academic skills among children in Germany and Iceland Early Childhood Research Quarterly, 28: 62-73; Wanless, S.B., M.M McClelland, A.C Acock, C.C Ponitz, S.H Son, X Lan, F.J Morrison, J.L Chen, F.M Chen, K Lee, M Sung, S Li 2011 Measuring behavioral regulation in four societies Psychological Assessment, 23(2): 364-378 14 Ponitz, Claire E Cameron, Megan McClelland, J.S Matthews, and Frederick J Morrison 2009 Head-Toes-Knees-Shoulders Task 13 Figure Summary of instruments used in the PFA outcome evaluation Instrument What it measures ROWPVT WJ/WM Letter-Word Recognition WJ/WM Applied Problems HTKS Parent survey PKBS Available languages Administration time Receptive vocabulary English, Spanish 15-20 mins Early literacy English, Spanish mins Early math English, Spanish mins Self-regulation English, Spanish, Chinese English, Spanish, Chinese English, Spanish, Chinese mins Family background characteristics Social skills & problem behavior 10-15 mins 10-15 mins ADMINISTRATION OF CHILD ASSESSMENTS The four child assessments were conducted with each participating child individually in a quiet corner of the classroom or in a separate supervised room provided by the site The assessment battery began with the Receptive One Word Picture Vocabulary Test, and was followed by the Woodcock-Johnson III Tests of Achievement Applied Problems subscale, the Head-Toes-Knees-Shoulders Task, and ended with the Letter-Word Identification subscale of the Woodcock-Johnson III Tests of Achievement If a child refused to take part in the assessment upon initial invitation, the assessor would invite the child again on a separate occasion (either at a later time the same day or on a different day), up to two times If the child refused all three invitations to participate, s/he was not assessed Children also had the right to refuse to complete any individual assessment in the battery, or any individual item in the assessment If a child needed a break, one was offered, and if s/he wanted to stop, her/his wishes were respected and the child would be invited back to complete the remainder of the assessment battery at another time Each child’s assessment language preference (English or Spanish for all measures; Chinese available for HTKS) was assessed by first asking teachers to identify language needs of each participating student in their class in order that assessors could be assigned classrooms according to the language needs of the class In the few cases in which this information was unable to be obtained prior to the assessment date, the language routing protocol on the Receptive One-Word Picture Vocabulary Test Spanish Bilingual Version was used, in which the assessor asks the child a series of questions regarding their language use to determine language dominance These questions included: • • Which language you know better, Spanish or English? (Spanish, English, or both) In which language would you like me to speak with you today? (Spanish or English) The assessor would proceed accordingly For Chinese-speaking students, in the rare event that the preferred language was not able to be determined by conferring with the classroom teacher in advance of the assessment date, language needs were assessed by the assessor on a case-by-case basis Upon completion of the Applied Problems subscale, assessors were instructed to ask the child if they preferred to continue in Chinese or in English for the HTKS, and the HTKS was administered accordingly Assessors were instructed that mono/bilingual students could be conversationally communicated with in their preferred language to help facilitate the testing process, but the English-language assessment instruments must be administered in English when specified by the assessment battery flowchart A summary of the language routing protocol is displayed in the Assessment Battery Flowchart in Figure on the following page All Chinese-bilingual assessors were native Chinese speakers and all but one Spanish-bilingual assessors were native Spanish-speakers Figure Assessment Battery Flowchart Classroom Language Form English Chinese bilingual Other Chinese only English ROWPVT For Chinese bilingual and monolingual children, the HTKS task was conducted in the child’s preferred language Chinese HeadToes-KneesShoulder Task Spanish bilingual Spanish only Spanish Bilingual ROWPVT English WJ Applied Problems For Spanish bilingual children, the remainder of the assessment was conducted in the child’s preferred language Spanish WM Applied Problems English HeadToes-KneesShoulder Task Spanish HeadToes-KneesShoulder Task English WJ Letter-Word Identification Spanish WM Letter-Word Identification Assessment complete: Child receives stickers/silly bands METHOD 2: LONGITUDINAL EXAMINATION BETWEEN PRESCHOOL AND KINDERGARTEN Because the 2012 kindergarten sample was not universally sampled across the San Francisco Unified School District kindergarten classrooms, the net used to potentially catch previously-assessed PFA grads was limited, as the sample was limited to Early Education Elementary schools From the sample of students assessed in the fall of 2012, 25 kindergarten students were identified who were also assessed when they were in preschool in the fall of 2011 This section examines the changes in assessment outcomes for this small subgroup of PFA preschool graduates Due to the small sample size, results are presented here for descriptive purposes and are not intended to draw generalizable statements from The average age of this subgroup at kindergarten was years months and the average time between the 2011 and 2012 dates of assessments was 12.5 months Across all measures, the students’ skill levels were either above the 5-yr 5-moh age-equivalent, or they demonstrated accelerated growth that closed the gap somewhat between their actual age and their age-equivalent scores For example, the table below shows results from the Receptive One-Word Picture Vocabulary Test The last two columns show an age-equivalent score that indicates that PFA graduates are on track for their age at kindergarten However, the final column demonstrates a 15-month difference between the 2011 and 2012 age-equivalent scores, which suggests that the year of preschool accelerated growth on this dimension beyond that which would be expected to occur over a 12.5 month period due simply to developmental maturation Figure 14 shows a similar pattern observed in early mathematical skills Figure 12 Change in receptive vocabulary skills over time among PFA graduates ROWPVT English raw score ROWPVT English standard score Matched n 2011 mean score 2012 mean score Change in score Age-equivalent at beginning of kindergarten Change in age equivalent after a year of PFA 16 52.94 70.38 17.44*** yrs mo 15 months 16 97.19 100.75 3.56 Source: Receptive One-Word Picture Vocabulary Test-4 (2012) Note: Spanish Bilingual ROWPVT results are not compared across years as the instrument changed versions between years In early literacy skills as measured by the Woodcock-Johnson Letter-Word Recognition subscale, results suggest that although growth was not accelerated, PFA graduates are performing at a level of approximately months above their age at kindergarten When results are examined separately for assessments conducted in English and Spanish, students who were assessed in Spanish are found to demonstrate accelerated growth in early literacy skills, advancing by approximately 2.5 months over and above that which would be expected by maturation alone Figure 13 Change in early literacy skills over time among PFA graduates Letter-Word Rec – all cases English only Spanish only Matched n 2011 mean score 2012 mean score Change in score Age-equivalent at beginning of kindergarten Change in age equivalent after a year of PFA 21 16 8.81 10.38 3.80 15.67 17.44 10.00 6.86*** 7.06*** 6.20* yrs mo yrs mo yrs mo 11-12 months 10-11 months 15 months Source: Woodcock Johnson III Tests of Achievement/Woodcock-Muňoz Pruebas de Aprovechamiento (2012) Note: To reduce error due to administration differences, this includes only cases where language of administration was the same in both years 16 Figure 14 Change in early math skills over time among PFA graduates Applied Probs – all cases English only Spanish only Matched n 2011 mean score 2012 mean score Change in score Age-equivalent at beginning of kindergarten Change in age equivalent after a year of PFA 21 16 10.00 10.88 7.20 16.00 17.00 12.80 6.00*** 6.12** 5.60* yrs mo yrs mo yrs 10 mo 14 months 14 months 13 months Source: Woodcock Johnson III Tests of Achievement/Woodcock-Muňoz Pruebas de Aprovechamiento (2012) Note: To reduce error due to administration differences, this includes only cases where language of administration was the same in both years Figure 15 Change in self-regulation skills over time among PFA graduates HTKS – all cases Matched n 2011 mean score 2012 mean score Change in sco 21 9.95 27.14 17.19*** Source: Head-Toes-Knees-Shoulders Task (2012) Note: Because there was a great deal of shifting in HTKS administration language between 2011 and 2012, all cases were used for year-to-year comparisons While this section describes promising findings from a small sub-set of PFA graduates over the final year of preschool, these data are limited in that they cannot inform as to how these changes might compare against children who did not attend a PFA-funded preschool, and so not provide us with an estimate of the impact of PFA Instead, the regression discontinuity design provides the most rigorous and accurate option for measuring the contribution of PFA to children’s readiness The results of these analyses are described next METHOD 3: REGRESSION DISCONTINUITY DESIGN To what extent does PFA participation lead to enhanced student readiness for school? This section brings the regression discontinuity design to bear on the question of the impact of PFA on child-level school readiness outcomes While the Preschool for All program has expanded access to free and/or subsidized preschool to fouryear-olds in San Francisco over the years, participation in PFA preschools remains completely voluntary As such, direct comparisons of assessment results between students who have and have not attended PFA preschools is subject to potential selection bias That is, certain families are more likely to select into sending their children to PFA preschools, and those families likely differ in systematic and unobservable ways from families who not select into PFA, and those differences could lead to biases in results Evidence of systematic differences between PFA graduates and kindergartners who did not participate in a PFA preschool are presented above The differences between groups suggest that cross-sectional analyses would lead to biased estimates of the impact of PFA on school readiness outcomes The regression discontinuity approach taken in this section uses assessment data from a “treatment group” of beginning kindergartners who are PFA graduates and comparing their means against the “control group” of current PFA preschoolers who are in the beginning of their last year of preschool Comparing these two groups reduces potential selection bias, as both the treatment and control groups have already selected into PFA Assignment into the treatment of control group is dependent upon each child’s date of birth In the San Francisco Unified School District, children are eligible to attend kindergarten if they are years old by November of each academic year Those who make that “cutoff” birthdate will be in kindergarten and comprise the treatment group; those who not make the cutoff are younger, in preschool, and comprise the control group This clear cutoff point allowed us to examine the effect of PFA by restricting our comparison of the means of treated children to means of children in the control group to those whose birthdates fall close to the cutoff 17 SAMPLE The regression discontinuity design requires two samples of students: (1) a control group of students currently enrolled in a PFA program, who were born on or after the specific cutoff date for 2012 SFUSD kindergarten eligibility, which was November 1, 2007; and (2) a treatment group of older students who graduated from a PFA preschool, who are currently in kindergarten, who were born before November 1, 2007 The sample obtained for the regression discontinuity design was drawn as follows: • The full sample of 1,034 completed assessments was pruned to remove the following cases: o Students who were neither a current or previous PFA preschoolers (n=199) o Students with identified special needs (n=11); o Students who were currently attending their second year of a PFA preschool (n=21 ) • This resulted in a pruned sample of 803, which included 547 control cases and 256 treatment cases • A random sample of 350 was drawn from the much larger control group to more equally balance the 17 numbers in each group, resulting in 606 cases overall • These 606 cases included 306 boys (50.5%) and 300 girls (49.5%) The age distribution of the sample is identical to the overall preschool and kindergarten sample ages Figure 16 Age at date of testing by cut-off Control group of current PFA students (Born after cut-off) Treatment group of former PFA students (Born before cut-off) Total Mean Age N yrs mo 350 yrs mo 256 606 FINDINGS In Figure 17 below, the mean test scores for each assessment is presented for the control and the treatment group As expected, mean scores for the older group of students born before the cutoff are higher than those of their younger counterparts The following analyses in effect control for age to estimate the impact of PFA on each of the assessed measures Figure 17 Mean comparisons of test scores before and after cutoff Measure Entering… PreK ROWPVT WJ Letter-Word Recognition WJ Applied Problems HTKS Mean 52.42 8.89 10.45 9.20 SD 16.60 5.62 3.95 12.35 Mean 62.18 13.89 14.96 21.44 Kindergarten SD 16.84 7.18 5.12 13.78 Sample Size 606 606 584 600 Standard deviations are in parentheses The first columns in the series of tables in Figure 18 on the following page contains estimates of the full model (Model 1), including squared terms and interactions to account for possible curvatures in slope and to estimate slope differences between control and treatment groups The coefficients for the quadratic terms and interactions are mostly not statistically significantly different from zero, which indicates that the relationship between age and 17 Jewell, Nicholas P 2009 Statistics for Epidemiology The author notes that in case-control designs, tests of independence are most powerful when the sample sizes are equal In a series of prior studies employing RDD in evaluating preschool effectiveness (e.g., W Steven Barnett, et al.), treatment and control groups were approximately equal 18 assessment scores is primarily linear The second set of columns in Model in the tables below present estimates that assume a linear relationship between age and assessment scores and excludes the squared terms Because the interaction terms remain non-significant, indicating non-significant differences between treatment and control group slopes, the final set of columns in Model drop the interaction term to provide estimates of the effect of 18 PFA in the most efficient, parsimonious model The following series of tables in Figure 19 shows estimates of the effect of PFA using different widths of windows around the cutoff, restricting analyses to children whose birthdates fall within 12 months of the cutoff, months, and months, based on the most parsimonious model estimating the effect of PFA on each readiness measure As illustrated in the tables, narrowing the window decreases the observed effect of PFA, likely due to the decreased effect of age Narrowing the window in effect reduces potential bias due to unobserved differences However, as the window narrows, standard errors tend to increase as the sample size falls substantially, thereby decreasing the accuracy of the estimates As such, discussion of results will focus on the 12-month window, on children whose birthdates fall within 12 months of the cutoff date 18 Note the gradual decline in standard errors as the model iterates to the final, most efficient form represented in the third column 19 Figure 18 The effect of PFA on school readiness outcomes within 12 months of cutoff birthdate ROWPVT-raw English ROWPVT-raw English Edition Model Born before cutoff (treated) Days to cutoff Days to cutoff squared Days to cutoff * cutoff Days to cutoff squared * cutoff ROWPVT-raw Spanish Bilingual Born before cutoff (treated) Days to cutoff Days to cutoff squared Days to cutoff * cutoff Days to cutoff squared * cutoff Letter-Word Recognition Born before cutoff (treated) Days to cutoff Days to cutoff squared Days to cutoff * cutoff Days to cutoff squared * cutoff English assessment Applied Problems Born before cutoff (treated) Days to cutoff Days to cutoff squared Days to cutoff * cutoff Days to cutoff squared * cutoff English assessment HTKS Born before cutoff (treated) Days to cutoff Days to cutoff squared Days to cutoff * cutoff Days to cutoff squared * cutoff Chinese assessment Spanish assessment Model Beta -3.84 0.02 0.00 0.07 0.00 SE 3.93 0.03 0.00 0.05 0.00 Sig 0.33 0.38 0.81 0.22 0.22 Beta 12.43 -0.01 0.00 -0.03 0.00 SE 6.63 0.03 0.00 0.08 0.00 Sig 0.06 0.83 0.16 0.72 0.18 Beta 1.54 0.01 0.00 0.02 0.00 5.10 SE 1.24 0.01 0.00 0.02 0.00 0.64 Sig 0.21 0.31 0.86 0.32 0.47 0.00 Beta 0.88 0.01 0.00 0.02 0.00 1.71 SE 0.91 0.01 0.00 0.01 0.00 0.47 Sig 0.34 0.31 0.57 0.09 0.27 0.00 Beta 5.06 0.02 0.00 0.01 0.00 -0.07 -3.40 SE 2.68 0.01 0.00 0.03 0.00 1.86 1.32 Sig 0.06 0.11 0.72 0.69 0.45 0.97 0.01 Model Beta -0.72 0.03 SE 2.88 0.01 Sig 0.80 0.00 -0.01 0.01 0.56 ROWPVT-raw Spanish Bilingual Edition Beta SE Sig 4.45 4.45 0.32 0.04 0.01 0.00 0.00 0.02 0.00 5.10 SE 2.79 0.01 Sig 0.69 0.00 Beta 4.74 0.04 SE 4.19 0.01 Sig 0.26 0.00 Beta 2.24 0.01 SE 0.87 0.00 Sig 0.01 0.00 0.84 Letter-Word Recognition Beta SE Sig 2.08 0.90 0.02 0.01 0.00 0.00 0.00 Beta -1.12 0.03 0.71 0.64 Applied Problems Beta SE 1.43 0.66 0.01 0.00 0.00 5.07 0.64 0.00 Sig 0.03 0.00 Beta 1.59 0.01 SE 0.65 0.00 Sig 0.01 0.00 0.00 0.00 0.30 1.73 0.00 1.70 0.47 0.00 Beta 6.48 0.02 0.47 HTKS SE 1.94 0.01 Sig 0.00 0.00 Beta 6.34 0.02 SE 1.88 0.00 Sig 0.00 0.00 0.00 0.01 0.79 -0.17 -3.33 1.85 1.31 0.93 0.01 -0.15 -3.35 1.85 1.30 0.93 0.01 Figure 19 The effect of PFA on school readiness outcomes by cutoff window ROWPVT-raw English Born before cutoff (treated) Days to cutoff within year of cutoff (N=480) Beta SE Sig -1.12 2.79 0.69 0.03 0.01 0.00 RWOPVT-raw English Edition within months of cutoff (N=254) Beta SE Sig -3.00 3.91 0.44 0.04 0.02 0.06 within months of cutoff (N=150) Beta SE Sig -5.50 5.12 0.28 0.06 0.05 0.29 within year of cutoff (N=124) Beta SE Sig 4.74 4.19 0.26 0.04 0.01 0.00 RWOPVT-raw Spanish Bilingual Edition within months of cutoff (N=62) Beta SE Sig 9.44 6.92 0.18 0.02 0.04 0.50 within months of cutoff (N=32) Beta SE Sig 4.38 9.54 0.65 0.07 0.09 0.49 within year of cutoff (N=605) Beta SE Sig 2.24 0.87 0.01 0.01 0.00 0.00 5.07 0.64 0.00 Letter-Word Recognition within months of cutoff (N=317) Beta SE Sig 1.42 1.14 0.21 0.01 0.01 0.02 4.72 0.82 0.00 within months of cutoff (N=183) Beta SE Sig 0.75 1.49 0.62 0.03 0.02 0.09 4.76 1.14 0.00 within year of cutoff (N=583) Beta SE Sig 1.59 0.65 0.01 0.01 0.00 0.00 1.70 0.47 0.00 Applied Problems within months of cutoff (N=305) Beta SE Sig -0.95 1.27 0.46 0.04 0.03 0.11 1.38 0.65 0.03 within months of cutoff (N=175) Beta SE Sig -0.20 1.14 0.86 0.03 0.01 0.02 1.36 0.86 0.12 within year of cutoff (N=599) Beta SE Sig 6.34 1.88 0.00 0.02 0.00 0.00 -0.15 1.85 0.93 -3.35 1.30 0.01 HTKS within months of cutoff (N=314) Beta SE Sig 5.98 2.80 0.03 0.02 0.01 0.15 -0.43 2.82 0.88 -4.42 1.91 0.02 within months of cutoff (N=182) Beta SE Sig 3.41 3.79 0.37 0.05 0.04 0.19 -0.70 3.38 0.84 -0.31 2.70 0.25 ROWPVT-raw Spanish Bilingual Born before cutoff (treated) Days to cutoff Letter-Word Recognition Born before cutoff (treated) Days to cutoff English assessment Applied Problems Born before cutoff (treated) Days to cutoff English assessment HTKS Born before cutoff (treated) Days to cutoff Chinese assessment Spanish assessment 21 R ECEPTIVE L ANGUAGE The tables above show no significant effect of PFA on receptive vocabulary for students assessed on the English language version of the test For those assessed on the Spanish-Bilingual Edition, a strong effect of treatment is found in the full regression discontinuity model that amounts to an advantage of nearly one year in age-equivalent scores However, in more parsimonious models that drop non-significant quadratic and interaction terms, the effect is reduced to non-significance Due to the small sample size of the Spanish-only analysis, these results should be interpreted cautiously and as only suggestive that the impact of PFA on receptive language development may be greater for Spanish speakers relative to English-only speakers E ARLY L ITERACY As shown in Figure 19 above, the effect of PFA on early literacy is statistically significant and it is estimated that a PFA preschool experience improves Letter-Word Recognition by an average of 2.24 points, which corresponds to an approximate 3-month gain in early literacy skills This finding is comparable to prior studies 19 using this method and measure The figure below depicts the relationship between scores on the Letter-Word Recognition subtest and the number of days from the November 1, 2007 cutoff date children in the sample were born Separate regression 20 lines are fitted for the treatment and control group to depict the discontinuity Figure 20 Effect of PFA on early literacy skills Letter-Word Recognition Score TREATMENT GROUP 2.24 CONTROL GROUP -365 365 Days to Cut-off Source: Woodcock Johnson III Tests of Achievement/Woodcock-Muňoz Pruebas de Aprovechamiento (2012) N=605 19 Gormley, William T Jr., Ted Gayer, Deborah Phillips, and Brittany Dawson 2005 “The Effects of Universal Pre-K on Cognitive Development” Developmental Psychology, 41(6):872-884 This study finds an effect size of 2.99 20 The scatterplot and regression lines fitting the actual data are presented in Appendix A 22 E ARLY M ATHEMATICS 21 Similar to results found in prior studies using the Woodcock-Johnson Applied Problems subscale , estimates obtained indicate a 1.59-point increase in Applied Problems scores at the discontinuity, which corresponds to an approximate 3-4-month gain in early mathematical skills associated with PFA preschool attendance (p