SUPERVISOR CORE MANAGING FOR RESULTS - Trainer’s Guide -

SUPERVISOR CORE: MANAGING FOR RESULTS - Trainer’s Guide - TRAINING TIPS AND CONTENT PART 1 Introduction Take several minutes for welcome, introduction of trainer, and trainees (depending on size of group), and brief overview of session agenda Child welfare services agencies are required by the C-CFSR to monitor the impact of their programming efforts for children, youth, and families through performance on a series of outcome measures This training module provides a basic understanding of: outcome evaluation concepts, specific measures tracked by the C-CFSR process, and useful approaches for applying outcome information to monitor, assess, and improve practice Learning Objectives Knowledge K1 The trainee will be able to recognize key concepts and terminology for outcome measurements (e.g., median, point in time, entry cohort, measures, rolling year) K2 The trainee will be familiar with the outcome measures on the quarterly, countyspecific reports created as part of the C-CFSR process (i.e., the difference between the federal and state-enhanced measures) K3 The trainee will be able to explain how to navigate the Child Welfare Dynamic Report System website of the UC Berkeley Center for Social Services Research, and understand that all data on the C-CFSR quarterly, county-specific reports (as well as much more detailed information) can be located there K4 The trainee will be able to explain the interrelated nature of the outcome measures for safety, permanency, and well-being that are tracked on the quarterly county-specific outcome reports Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 Skills S1 The trainee will be able to demonstrate the ability to ask policy and practice questions related to trends identified in the C-CFSR outcome measures S2 The trainee will be able to develop a plan to improve the delivery of child welfare services in his or her agency through monitoring outcomes with the CCFSR process Values V1 The trainee will value the need to supervise and monitor in ways to achieve ASFA outcomes V2 The trainee will value ongoing recognition of trends identified in the outcome measures that enhance culturally sensitive and responsive practice by helping target services appropriately to ethnic, age, and gender groups V3 The trainee will value incorporating various practice issues into supervision: A Outcomes vs process B Fairness and equity C Incorporating data Background and Concepts for Outcomes in Child Welfare (Trainer's Slides PowerPoint [PPT] Slide Show) Slide Data-Informed Practice—What I Care? The child welfare system has traditionally not done a very good job of monitoring and evaluating its performance Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 Slide Slide Current Placement System—Highly Simplified Child welfare policy and practices need to be guided by clear and specific goals, and progress toward those goals requires good performance data Yet—in spite of the large quantity of information often collected about children in outof-home care—child welfare agencies are often unable to provide quick and reliable responses to questions posed by policymakers, administrators, and the public Slide Data Skepticism ACTIVITY: Ask trainees to call out reactions they have to the idea of using data in their work, reasons that they may not like data, etc Use flip chart to record answers Sentiment among child welfare agency staff is that, while they may be inundated by data, this information is not useful for planning, evaluation, or other decisions Their experience has often been that data systems used by the agency were designed for other purposes and cannot provide answers to questions that are pertinent to their practice Further, they likely feel that entering data into CWS/CMS is a waste of time and takes away valuable time that they could be doing actual casework This attitude is largely due to a legacy of state data systems like CWS/CMS as being “roach motels” of data—workers enter information in, but virtually nothing ever comes out Or, when data does come out, it is not Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 Slide (continued) presented in a way that is easily accessible or useful A first step, then, in overcoming this aversion to data is to make the connection for workers between the information they input into CWS/CMS about their cases, and how successful their agency is helping its clients To so requires analyzing data that has been put into CWS/CMS and sharing it—on an ongoing basis —in a format that workers can understand When workers hear the word “data,” they immediately think of “MATH” or “STATISTICS” and can be intimidated or turned off A common reaction is “I hate math! I became a social worker so that I could help people—not work with numbers!” Another likely reaction is, “I've been a social worker for 25 years, and I know very well what area most needs work in our agency—it’s placement disruptions (or reunifying kids more quickly, or cutting down on the number of kids reentering care, etc.).” These typical responses are not unreasonable— given that workers have probably never seen examples of how data can be useful If supervisors can be shown that information drawn regularly from CWS/CMS is a form of feedback on the impact of their efforts (as evidenced by performance on important goals such as preventing recurrence of maltreatment, or achieving reunification for children, etc.), their distrust of the usefulness of data will decrease It is important to recognize that each worker’s practice experience provides a very real and valid perspective of their agency’s areas of success and failure This validation can be used to point out to supervisors that each of them is actually a data person whether they realize it or not That is, their experience is their data source from which they draw conclusions and make decisions about what areas have greater or lesser importance Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 Slide (continued) Extrapolating from each worker's experience with their specific caseload to the information on all workers’ caseloads that is captured in CWS/CMS, provides a more comprehensive picture of where a child welfare agency needs to focus its efforts Clearly, improvements could be made in all parts of the child welfare system—but limited time and resources demand that efforts be made in the areas of greatest need As a supervisor, it is critical to know where your attention would best be focused—and to keep on top of trends because the area of focus changes over time In the ER unit—are referrals and/or substantiations going up or down? For which age or ethnic groups? Are these trends reflected in entries to care? In the permanency unit—are reunifications increasing or not? If they are, is our re-entry rate the same or are many of the kids we send home later coming back into care? Drawing on information in CWS/CMS will help supervisors to identify these trends and keep track of progress toward addressing them Regularly updating staff in unit meetings—and celebrating any progress or success—can positively reinforce and motivate staff Fancy statistics are not necessary Simple line or pie charts can easily point out those areas where the agency is succeeding best and where it most needs to focus its efforts to help families Supervisors will begin to understand that access to timely data and the ability to analyze this information can help agency staff to identify and respond to changing population needs—and enable workers to monitor the results of their work Overcoming skepticism regarding the usefulness of data is therefore a crucial first step toward better outcomes—and thus for improved lives of children and families Ask for ways that might make using data and Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 outcomes measures more accessible for supervisors and workers Record suggestions on flip chart If training staff from only one or two counties, ask them to tell you general trends in child welfare in their county (e.g., how many children enter care each year, how long children generally stay in care, what proportion of children reunify after 12 months in care, etc.) Record these answers and check them against actual trends for the counties during the live demo of the UC Berkeley website Then underscore the necessity to make these efforts to be aware of key trends, keep staff informed about areas of need, and to track progress, because… monitoring child welfare agency performance is no longer simply a good idea—it's the law Slide California’s Child Welfare Outcomes & Accountability System The Children and Family Services Review (CFSR) is an ongoing nationwide process conducted by the U.S Department of Health and Human Services The first stage of the CFSR examined service delivery by child protection, foster care, adoption, family preservation, family support, and independent living programs in all states The CFSR examines seven outcomes for children and families as well as seven systemic factors States were measured against national standards During the first round of the CFSR, no state achieved substantial conformity on all measures Thus, all states are implementing a Program Improvement Plan (PIP) The CFSR reviews all states periodically to follow progress and may impose fiscal penalties on those states that fail to improve In 2003, California did not achieve conformity on any of the seven outcomes, and passed only one Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 of seven systemic factors The state faced $18 million in penalties if it had failed to meet federal requirements by 2005 In January 2008, the federal government levied a nearly $9 million penalty against the state for failing to meet the outcome goals regarding re-entry and placement stability Loss of this funding during times of fiscal hardship will likely result in loss of services to children and families Slide (continued) Assembly Bill 636, the Child Welfare Outcomes and Accountability Act of 2001, was developed as a means to carry out California's PIP The bill provides a legislative framework for monitoring and assessing county performance to ensure the safety, health and well-being of the state’s children Since the passing of this legislation, California was instrumental in influencing the federal government to review and subsequently revise their federal outcome measures in 2007 Although many of the revised federal measures mirror the previously enhanced California measures, there are still a few vital outcome areas that California chooses to track in the absence of federal mandates Those measures include the “Timely Response” of social workers regarding Immediate and 10-day referrals and “Timely Social Worker Visits.” Once again, California is leading the way in the creation of well-being measures for children who are in foster care As of June 2009, the measures for health, dental, and educational outcomes for children in care were under development Each county has created a self-assessment that identifies strengths and areas for improvement, as well as a system improvement plan (SIP) to outline actions needed to improve county performance with respect to the outcomes Key Concepts to Understand Child Welfare Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 Outcomes Before examining the specific outcomes tracked under The California Child Welfare Outcomes and Accountability System, it is important to know the background about the reasons (i.e., sound methodological basis) why many of the measures were chosen Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 Slide Trainee Handout # Longitudinal Data Analysis—The Advantage of Using Entry Cohorts Caseload Snapshots Versus Entry Cohorts1 (Handout #1) A limitation of traditional child welfare research is that the data used are often comprised of periodic snapshots of the caseload of children in care at a given point in time This information is important for providing basic management accountability; however, it does not accurately capture the experience of all children who come in contact with the child welfare system, and can be misleading when examining issues such as length of stay or placement moves in care Point in time estimates are biased They overrepresent children who have the worst experiences in care—that is, those who tend to stay in care for long periods Slide #6 illustrates this phenomenon: the xaxis represents a timeline from January 1, 2007, up to January 1, 2009, and the 10 numbered horizontal lines represent the duration in care of 10 children in foster care A point in time snapshot taken of the caseload on January 1, 2008, will only observe about half the children (line numbers 1, 2, 3, 5, and 10 from the top) The other children will not be detected by this snapshot Therefore, any mean or median measure of length of time spent in care will be biased by the children observed in the snapshot who have longer lengths of stay Placement stability estimates would be similarly biased Using snapshot data, staff may not recognize the opportunity to effectively focus resources by assuming that most children in the system are poorly served This discussion adapted from: Webster, D., Needell, B., & Wildfire, J (2002) Data are your friends: Child welfare agency self-evaluation in Los Angeles County with the Family to Family initiative Children and Youth Services Review, 24(6/7), 471-484 Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 Slide (continued) ACTIVITY: Referring to PPT slide #6, ask trainees to call out how many children would comprise the calendar year 2007 entry cohort (Answer: 7) Then ask trainees to call out how many children would comprise the calendar year 2008 entry cohort (Answer: 3) Fun Exercise with Medians and Means “How long children stay in care in your county?” is an important question that people often ask The following brief exercise illustrates the need to reflect on the approach taken to answer the question about length of stay ACTIVITY: Prior to class, trainer cuts out rows (make more as needed) from the following table and distributes one row each to student at the beginning of this exercise (The trainee given the “Bill Gates” row is asked to keep his/her identity secret) WORK TITLE UCB Ph.D., Consultant Administrative Assistant INCOME Roughly minimum wage (let’s say, $10 K) $20 K Administrative Assistant $20 K Social Worker I $40 K Social Worker I $40 K Social Worker II $50 K Social Worker II $50 K Supervisor $60 K Supervisor $60 K Branch Manager $75 K Branch Manager $75 K Bill Gates, CEO Microsoft $3 Billion Ask students (except for Bill Gates) to state their Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 10 works to improve public policy, increase training and raise public awareness on the issue It tracks its success by utilizing data on older youth in foster care and on exits from care http://www.cpyp.org Summarize points supporting why data is important for supervisors:  It is critical to know where your limited time would best be focused—information on all workers’ caseloads captured in CWS/CMS provides a more comprehensive picture of where a child welfare agency is performing well and where it needs improvement  Regularly updating staff in unit meetings— and celebrating any progress or success—can positively reinforce and motivate workers  Regular use of data and understanding relationships between the outcomes “tells the story” of how well children and families are being helped—whether supervisors are motivating staff in a unit meeting, communicating needs for policy and practice change to a child welfare director, or being “accountable” to relevant members of the community (e.g., judges, child advocates, legislators, board of supervisors) Break  Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 14 Slide 8 PART Using Outcome Data in Child Welfare Practice This part of the seminar will introduce the specific outcome measures tracked under the California Outcomes and Accountability System, the quarterly data report, and the UC Berkeley website The UC Berkeley website demonstration is a “live” online session that requires a high-speed Internet connection The California Quarterly Outcomes and Accountability County Data Report What is it, and what’s on it? The California Outcomes and Accountability quarterly report provides summary-level federal and state outcome measures that are intended to be the basis for ongoing county self-assessment of its performance over time Each county is expected to use data in the report for the self-assessment process—that is, to determine reasons for current performance as well as to help plan for measurable improvement The California Outcomes and Accountability quarterly report contains information on the following areas: CWS participation rates along with the three federal child welfare “domains” of safety, permanency, and well-being A condensed version of the report layout is included in trainees’ handout packets The most recent version of the complete quarterly report for all California counties and the state as a whole is available at http://cssr.berkeley.edu/ucb_childwelfare/ccfsr.aspx California Quarterly Outcomes & Accountability County Data Report (Trainee Handout #3) Child Welfare Services Participation Rates: provides data on the number, and number per 1,000 children in the county/state, for key child welfare indicators It is intended as background information to assist your county in analyzing your county’s performance by the outcome indicators Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 15 Safety Outcomes: designed to reflect the effectiveness of efforts to protect children from abuse/neglect by reporting instances of abuse and neglect at various stages of child welfare services and process, measures which reflect the frequency of social worker contact with children and the speed of face-to-face investigation of abuse/neglect allegations Permanency Outcomes: designed to reflect the number of foster care placements for each child, the length of time a child is in foster care, and the rate that children re-enter foster care after they have returned home or other permanent care arrangements have been made Child and Family Well-Being Outcomes: designed to reflect the degree to which children in foster care retain relationships with the family and extended communities with whom they were associated at the time of their removal from their parents The county data report is produced on a quarterly basis—that is, every three months Each quarter, a data extract from CWS/CMS is queried to create the reports Counties are allowed a two-month window after the close of each quarter to comply with CWS/CMS data entry requirements Thus, taking the 2007 Quarter extract from CWS/CMS as an example, information—about any activity within a child’s case history that occurred prior to June 30, 2007—could be entered into CWS/CMS up until August 31, 2007, and still be included in the 2007 Quarter data extract Slide Federal Versus State-Enhanced Measures Unlike point-in-time counts that contain information on children who are still in care at a given point in time, or exit cohorts that examine children leaving care and are potentially confounding by mixing long- and short-stayers in care, longitudinal data systems capture information on all children who enter foster care and follow them for the entire time they are in care Information on a cohort of children as they move through out-of-home care does not systematically exclude children from the study population if they leave care As Fred Wulczyn of the Chapin Hall Center for Children has observed, “if you were trying to evaluate a cancer therapy or HIV treatment— would you look only at patients who survived (or only at those who died) to determine the effectiveness of the intervention—or would you follow all patients from the time the treatment Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 16 started?” The answer is obvious Reliable information on children’s child welfare experiences requires following children from first entry to care through placements within and exits from the system, and, for some children, subsequent re-entries to care Supervisors, workers, and administrators need to evaluate performance and base decisions on the most comprehensive and accurate information possible The AB 636 “state enhanced” measures, therefore, use entry cohort analysis to supplement the federal standards and provide this information Many of the new measures adopted in 2006 by the Administration for Children and Families (ACF) were part of California’s enhanced measures adopted through the AB 636 legislation As mentioned above, California continues to monitor and track data regarding social worker timely response and monthly visits to children even though there is not a mandate to so Other measures enhanced by California include the wellbeing measures regarding sibling placements, least restrictive placements, ICWA/cultural considerations and the post foster care outcomes What the ¡ ! is a "Rolling Year" ?2 Use graphics on PowerPoint slides to explain this topic Use flip chart if further explanation is needed Part of this discussion based on: Hurley, D (2004) Completing the county self-assessment, Module 3: Analyzing child welfare outcome indicators California Social Work Education Center: Berkeley, California Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 17 Slide 10 Slide 11 Rolling Years Each quarter, most measures are available using the most recent data available for that measure using “ROLLING YEARS.” A rolling year examines an outcome for children in a year-long cohort, but the year being tracked will not always be equal to the January–December calendar year A rolling year can be April (of one year) to March 31 (of the following year); other rolling year periods can be July 1(of one year) to June 30 (of the following year), or October to September 30 The January to December 31 calendar year is also a rolling year There are Rolling Year Time Periods (corresponding with the quarterly data extracts from CWS/CMS):  (Trainee Handout #4) Slide 12    Q1: April 1–March 31 Q2: July 1–June 30 Q3: October 1September 30 Q4: January 1– December 31 (extract cut-off: April 1st) (extract cut-off: July 1st) (extract cut-off: October 1st) (extract cut-off: January 1st) The rolling year used depends on the outcome being examined For example, using 2009 Quarter data, UC Berkeley can look at first entries to care for children who enter July 1, 2008–June 30, 2009, reunifications within 12 months for children who entered July 1, 2007–June 30, 2008, and adoptions within 24 months for children who entered July 1, Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 18 2006–June 30, 2007 Slide 11 As new data become available each quarter with the most recent extract from CMS, these measures can be “rolled” forward to track a 365-day cohort up to the most recent study cut-off date For example, for 2009 Quarter 3, information on October 1, 2008, to September 30, 2009, will be available This gives users of the website the ability to monitor trends over time without the problems associated with quarterly measures (e.g., seasonal effects, very little data in small counties) Slide 12 Pop Quiz on Rolling Years (Trainee Handout #4) ACTIVITY: Pop Quiz Ask trainees to take minutes to answer the questions (listed below) on Handout #4 After minutes, ask trainees to call out correct answers to questions Review rolling years concept if necessary Pop Quiz on Rolling Years: When the 2009 Quarter data are available (i.e., last day of data: September 30, 2009), the next progressive quarter’s worth of information can be added in tracking outcome measures Question 1: In the 2009 Q3 data extract, what is the most recent time period used to measure first entries to care? Answer: The time period is October 1, 2008, to September 30, 2009 Question 2: In the 2009 Q3 data extract, what is the most recent time period used to measure the percent of children reunified within 12 months? Answer: The time period is October 1, 2007, to September 30, 2008, since this is the most recent cohort of children for whom each can be followed for Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 19 an entire 12 months to end up in the September 30, 2009, data extract Question 3: In the 2009 Q3 data extract, what is the most recent time period used to measure the percent of children adopted within 24 months? Answer: The time period is October 1, 2006, to September 30, 2007, since this is the most recent cohort of children for whom each can be followed for an entire 24 months to end up in the September 30, 2009, data extract Taking Your First Steps Toward Data Analysis UC Berkeley Dynamic Website Introduction The introduction of the UC Berkeley Child Welfare Dynamic Report System Website requires a high-speed Internet connection The trainer must be very familiar with the layout and information available on the UC Berkeley Child Welfare Dynamic Report System Website Screen shots are provided in the Trainee’s PowerPoint slides (slide numbers are indicated below) strictly as points of reference for organizing the training Slide 13 UC Berkeley Child Welfare Dynamic Report System Website All the data on the California Outcomes quarterly report are posted on the websites of the Center for Social Services Research at University of California, Berkeley (http://cssr.berkeley.edu/ucb_childwelfare/), or on the California Department of Social Services site (http://www.dss.cahwnet.gov/research/) In addition to the summary-level numbers provided on the county data reports, data on the UC Berkeley Child Welfare Dynamic Report System website include much more detailed information for each of the California Outcomes and Accountability measures (i.e., measures can be examined by age group, ethnicity, gender, and, in most cases, placement type) as well as for other outcomes Be sure to click on and read the methodology link for any outcome analysis that you are examining The Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 20 Slide 14 Slide 15 methodology description provides important detail about how the analysis was conducted The quarterly data report includes a number of tabbed pages which allow the user to compare the latest outcome data with an established baseline or another period of time The report also provides a unique view of the composite measures The composite viewer tab shows the outcome performance relative to the national standard The goal is to improve state performance on all measures There is also a tab which details the methodology for each measure and provides a simple explanation for each outcome indicator There are also tabs that show the past dates for the retrieval of past outcome data Finally, there are graphs that depict the outcome performance over time at a one-shot glance Other important features to explore on the UC Berkeley Child Welfare Dynamic Report System website:  All Available Outcomes by Specified County— provides all reports available on the UC Berkeley Report System website pertaining to a county that a user specifies Select your county from a drop-down menu for county-specific information With the dynamic site you can also choose different variables to review over time http://cssr.berkeley.edu/ucb_childwelfare/  Data Highlights—are one-page documents about statewide trends and developments in child welfare as interpreted from CWS/CMS data A useful exercise would be to substitute your own county data for the statewide data An index of available highlights is available at http://cssr.berkeley.edu/CWSCMSreports/Highli ghts/  Composite Viewer—shows the outcome Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 21 Slide 16 performance relative to the national standard for: Safety Measures: S1_1: S2_1: No Recurrence of Maltreatment No Maltreatment in Foster Care and Permanency Composites: Slide 17 Slide 18 Composite 1: Composite 2: Composite 3: Composite 4: Reunification Composite Adoption Composite Long-term Care Composite Placement Stability Composite The series of slides #16 through #20 depict how the individual measures of each permanency composite are weighted These slides correspond to the Trainee Supplement New Federal Measures Listed with National Standards For trainees who have great interest in this subject, the Trainee Supplement Federal Outcome Descriptors describes the computations for each outcome measure, as well as the rates for referrals, substantiations, entries, and ‘in care’, which are also found on the CSSR website Slide #21 shows the overall structure for the relationships among measures, components, composites, and federal goals Slide 19  Mailing List Link—for email updates from the UC Berkeley site You will be notified of changes, updates, and additions to the site To sign up for email updates go to: http://cssr.berkeley.edu/ucb_childwelfare/mailin fo.aspx  Presentations—Presentations by UC Berkeley child welfare researchers are available on the website and can provide supplemental information for understanding and reading data found on the website http://cssr.berkeley.edu/cwscmsreports/presentat Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 22 Slide 20 ions/ Static Site—Information is still available on the static website, although the new dynamic website is more user-friendly http://cssr.berkeley.edu/cwscmsreports/defaultSt atic.asp Slide 21 In conjunction with explaining the CWS/CMS dynamic website, county-specific data tools should also be discussed For example, Safe Measures reports or Business Objects reports are also useful tools for monitoring and assessing performance outcomes When all tools are used in conjunction, supervisors can make the link between the macro picture of the federal outcome measures and standards and state, county, unit and worker outcomes and performance It may be helpful to select one outcome area to demonstrate For example, a supervisor could compare “visitation with parents” with “reunification within 12 months” in their unit and in their county The data could assist the supervisor to address visitation and reunification issues with staff Break PART Outcome-Based Management—A Hands-On Exercise People learn best by doing This final portion of the seminar will give trainees the opportunity to apply some of information covered in Parts & and to begin to grapple with looking at outcome data, interpreting results, and formulating plans to positively affect targeted outcomes About 45 minutes should be spent working with the data; and the remaining 15 minutes can be used for reporting back Outcomes Based Management—A Hands-on Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 23 Slide 22 Exercise (Handouts #5-16) The trainer should break the trainees into groups of 5–7 people Each group will work together on interpreting the outcome data in their handout packet The trainer should circulate between the groups to answer questions and help keep them on task After forming the different groups, the trainer should underscore suggestions on what to think about as they review the data (these are listed on Handout #5) Not all the following points are applicable to this in-class exercise, but should be borne in mind whenever they work with outcome information in their county Things to keep in mind as you examine outcome data3:  Begin your analysis by determining which outcomes (and possibly for which subgroups) are “high priority” and “high performance” areas within your county  Examine trends in the outcomes over time (e.g., has the number and rate of entries changed over the past several years? Has the proportion of children reunified within 12 months changed?)  Slide 23 Also remember that sometimes data may not answer “why” in relation to practice questions For example, in the first round of the CSFR California failed to meet the required standards in the PIP regarding placement stability, among other outcomes The data was able to show there weren’t improved outcomes in those areas, however it did not tell the story of “why” that was so Data is needed to help focus the questions and discussion to drill-down to the “why” In the case of the first round CSFR, the Some of these points adapted from: Hurley, D (2004) Completing the county self-assessment, Module 3: Analyzing child welfare outcome indicators California Social Work Education Center: Berkeley, California Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 24 Trainee Handout #6 data outcomes have generated new questions  Analyze age and especially ethnic disparities (slides #23–28/Trainee Handouts 6–11)  Look at differences between placement types (particularly kinship versus non-kinship care) in slide #29/Trainee Handout 12  Examine the differences in exits to permanency (slide #30/ Trainee Handout 13) Slide 24 Trainee Handout #7  Comment on the difference between the national goal and the data for Measure C3.3: In Care Years of Longer (slide #31/Trainee Handout 14)  Note the measures for Permanency Composite 4: Placement Stability (slide #32/Trainee Handout 15)  Note the trend towards improvement for the percentage of all siblings placed together as reflected in slide #33/Trainee Handout 16 Slide 25 Trainee Handout #8 Slide 26  Consider the relationships between outcome measures Recalling the “cycle of experiences” discussion and handout, a change in one performance measure may have a relationship with other outcomes For example, diverting children from out-of-home care would decrease entry rates—but the children who still require placement may be more troubled, and less likely to be reunified, affecting reunification rates  Identify factors that may be influencing performance Some examples are: • Management information system issues (e.g., data cleanup, data entry, or interpretation issues that may have influenced performance measures • Staff and provider training needs • Case review system • Service array and agency collaborations • Recruitment, licensing and retention of Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 25 Trainee Handout #9 • foster and adoptive parents Quality assurance system  Identify any county policies, regulations, or practices that may contribute to your performance on a given indicator  Slide 27 Trainee Handout #10 Identify linkages between performance outcomes and practice processes  Think of other outcome information that you may want to have in order to get a more complete picture of performance (e.g., no reentry data is given in the handouts, but this information would be helpful to know with respect to children reunifying from care)  Conclude analysis by determining areas of strength and those in need of improvement Summary Points for Hands-on Exercise Handouts Slide 28 Trainee Handout #11  Black children are referred for maltreatment proportionally more than any other group Black children make up 6.2% of the total child population, but account for 14.9% of referrals (Slide #2/Trainee Handout #6)  Once referred, children are equally likely to have substantiation Across racial groups approximately 22–29% of children who are referred are substantiated Slide 29 Trainee Handout #12 Slide 30  Black children with open cases are the least likely to be served at home and the most likely to be in long term foster care Only % of the Black children with open cases are in the Family Maintenance Program, compared to 11.1% of White Children, and 13.4% of Hispanic children (Slide #24/Trainee Handout , for July 1, 2008)  Black children are the most likely to enter foster care Black children enter care at a rate of 11.2 per 1,000—compared to 3.0 per 1,000 for White children, and 3.7 per 1,000 for Hispanic Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 26 Trainee Handout #13 Children (Jan 1, 2007–Dec 31, 2007, data)  Black children are the most likely to be in foster care The rate of Black children in foster care is 29.1 per 1,000—compared to 5.2 per 1,000 for White children, and 6.0 per 1,000 for Hispanic Children (July 1, 2008, count day) Slide 31 Trainee Handout #14  Black children stay in care longer than children from other racial groups The median length of stay is 553 days for Black children, 432 days for White children, and 487 days for Hispanic children (Jan 1, 2001–Dec 31, 2006) Slide 32 Trainee Handout #15 Slide 33 Trainee Handout #16 Report Back and Session Wrap-Up The trainer should close the session by underscoring that managing for results necessitates creating a flow of information to support mid-course corrections and continuous improvements in outcomes Some helpful ideas to accomplish this are: Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 27    Recognize that evaluation is a process, not an isolated event or a particular report Regularly monitor progress (e.g., expand management team meetings monthly to include regular updates on outcomes, create specific data/outcomes tracking committee that regularly meets and reports to management team and all agency staff) Cultivate an office culture where everyone owns the outcomes: • Prominently display outcomes throughout the agency • Regularly update all staff on progress towards achieving outcomes Finally, a homework assignment is provided in their Trainee Content (Handout #18) This series of steps is meant to review some of the key points covered in the seminar, and to reinforce the notion that supervisors can take what they have learned back to the office and begin applying it immediately They can locate/download the latest copy of their county’s Quarterly Outcomes report, and refer to the UC Berkeley Child Welfare Dynamic Report System website for detailed underlying data (e.g., trends and relationships between outcome measures) Through the Dynamic Report System, they can track their agency's successes and areas needing improvement to help children and families more effectively Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 28 ... and families Ask for ways that might make using data and Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 outcomes measures more accessible for supervisors and... county self-assessment, Module 3: Analyzing child welfare outcome indicators California Social Work Education Center: Berkeley, California Supervisor Core | Managing for Results: Trainer’s Guide |... county performance with respect to the outcomes Key Concepts to Understand Child Welfare Supervisor Core | Managing for Results: Trainer’s Guide | Version 1.1, July 2009 Outcomes Before examining