Using the Ohio Scales for Assessment and Outcome Measurement in a Statewide System of Care Oklahoma Systems of Care (OSOC) Expansion Implications for Evaluation OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION From 2002-2011, using a combination of state funds and funds from Phase IV and Phase VI SAMHSA grants, Oklahoma’s wraparound program expanded from an initial counties to 56 This rapid expansion necessitated a state evaluation separate from the national evaluation and longitudinal study Not all sites would be included in the national evaluation’s longitudinal study Data from the national evaluation would cover a small percentage of youths in OSOC wraparound Quality improvement and decision support required 100% data collection from enrolled families A scalable solution would incorporate line staff as data collectors, so burden – in time and training – had to be limited OSOC Sites 2000-2011 Pawnee Ellis Payne Dewey Kingfisher Blaine 2002 2008 Custer Canadian Oklahoma Washita Beckham Caddo Grady 2003 2009 Harmon Greer Kiowa 2010 Okfuskee Haskell Hughes Pittsburg Latimer Leflore Garvin Comanche Jackson Coal Murray Pushmataha Tillman Cotton Carter Atoka Johnston Jefferson McCurtain Love 2011 Sequoyah Pontotoc Marshall 2005 Adair Muskogee McIntosh McClain Stephens 2004 Okmulgee 2007 Lincoln Seminole 2001 Roger Mills Mayes Wagoner Creek Logan Pottawatomie 2006 Rogers Noble Major Cleveland 2000 Garfield Ottawa Delaware Woodward Osage Craig Cherokee Woods Tulsa Harper Beaver Alfalfa Texas Washington Kay Grant Cimarron Nowata OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Bryan Choctaw OSOC 2001-2011 Families Served, Sites and Counties by Fiscal Year Sites in Operation Counties Served 1800 1667 1600 56 Families Served 1400 49 44 1200 56 50 40 1088 40 31 800 648 600 852 400 304 156 200 38 2001 2002 13 2003 30 26 20 20 20 10 13 10 34 31 762 483 60 50 53 1246 1000 1405 70 2004 2005 2006 Fiscal Year 2007 2008 2009 2010 2011 # of Sites and Counties Clients Served OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Development of State Evaluation OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Psychometric tool originally in use would not scale Costs and logistics related to credentialing requirements were prohibitive Evaluation committee of evaluators, staff, families and youth developed the state evaluation tool set – based on extracts from longitudinal study public domain instruments – in spring and summer of 2004 2003/2004 national conferences exposed evaluation team members to the Ohio Scales, in use by other Systems of Care communities State evaluation committee chose to build OSOC Assessments around the Ohio Scales in July 2004 Why the Ohio Scales? OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Originally designed as a statewide instrument Qualities appropriate to statewide implementations Parent—Administered at baseline with follow up at month intervals Self-administration No credentialing requirement Nominal cost Brevity Simple language and item content Easily understood scoring, interpretation of results Youth—Administered to youth aged 12 and older at baseline with follow up at month intervals Worker—Administered at baseline with follow up at months and then month intervals from baseline Good and growing research base Articles on flash drive Ohio Scales: Youth Problem Scale 10 11 12 13 14 15 16 17 18 19 20 Arguing with others Getting into fights Yelling, swearing, or screaming at others Fits of anger Refusing to things teachers or parents ask Causing trouble for no reason Using drugs or alcohol Breaking rules or breaking the law (out past curfew, stealing) Skipping school or classes Lying Can’t seem to sit still, having too much energy Hurting self (cutting or scratching self, taking pills) Talking or thinking about death Feeling worthless or useless Feeling lonely and having no friends Feeling anxious or fearful Worrying that something bad is going to happen Feeling sad or depressed Nightmares Eating problems Externalizing Subscale Delinquency Subscale Internalizing Subscale 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 Most of the Time All of the Time Often Instructions: Please rate the degree to which your child has experienced the following problems in the past 30 days OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Once or Twice Several Times (Copyright © January 2000, Benjamin M Ogles & Southern Consortium for Children) Not at All Youth Problem Scale 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 Produces a score of - 100 Ohio Scales: Youth Functioning Scale (Copyright © January 2000, Benjamin M Ogles & Southern Consortium for Children) Resiliency Subscale Caring for health needs and keeping good health habits (taking medicines or brushing teeth) 10 11 12 13 14 15 16 17 18 19 20 Controlling emotions and staying out of trouble Being motivated and finishing projects Participating in hobbies (baseball cards, coins, stamps, art) Participating in recreational activities (sports, swimming, bike riding) Completing household chores (cleaning room, other chores) Attending school and getting passing grades in school Learning skills that will be useful for future jobs Feeling good about self Thinking clearly and making good decisions Concentrating, paying attention, and completing tasks Earning money and learning how to use money wisely Doing things without supervision or restrictions Accepting responsibility for actions Ability to express feelings Doing Very Well Getting along with friends Getting along with family Dating or developing relationships with boyfriends or girlfriends Getting along with adults outside the family (teachers, principal) Keeping neat and clean, looking good OK Some Troubles Instructions: Please rate the degree to which your child’s problems affect his or her current ability in everyday activities Consider your child’s current level of functioning Quite a Few Troubles OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Extreme Troubles Youth Functioning Scale 0 0 1 1 2 2 3 3 4 4 4 0 0 0 0 0 0 0 1 1 1 1 1 1 1 2 2 2 2 2 2 2 3 3 3 3 3 3 3 4 4 4 4 4 4 4 Produces a score of - 80 Ohio Scales Guidelines from the Texas Study OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION 2003 study by the Texas Department of Mental Health and Mental Retardation used comparisons with CBCL and SDQ to validate the scales and to establish classification criteria for the 20-item Ohio Problems and Functioning scales: Problems Scale Score of 25 and above = critical impairment Score of 17 – 24 = borderline impairment Decrease of 11 or more points = clinically significant improvement Functioning Scale Score of 44 and below = critical impairment Score of 45 – 53 = borderline impairment Increase of or more points = clinically significant improvement Significant Improvement in Ohio Scales Problems and Functioning Scores OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Benchmark Value = 70-80% of 'Impaired' Youths Should Show Significant Improvement at Months (-11 points for Problems, +8 for Functioning) Site County County County County County County County County County County 10 County 11 County 12 # Youths with # Youths with % of Impaired Baseline and Impaired % Youth with # Showing Showing 6-Month Baseline Significant Significant Baseline Scale Assessments Scores Impairment Improvement Improvement 13 69% 78% 47 36 77% 26 72% 38 30 79% 20 67% 11 82% 67% 12 75% 67% 16 14 88% 64% 25 21 84% 13 62% 73 62 85% 35 56% 10 90% 56% 32 28 88% 15 54% 11 10 91% 50% 16 12 75% 50% FY 2012 Ohio Scales Clinically Significant Improvement by Impairment at Baseline OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Months Baseline Not Improved, 179, 32% Not Impaired, 154, 22% Impaired, 554, 79% Improved, 381, 68% OSOC Youth Data 2007 – 2011 Referrals to Wraparound from Primary Sources Ohio Scales Problems – Baseline 80% 60% Children and youth referred by Child Welfare and Juvenile Justice are more likely to score in the normal range than those referred by other sources Impaired 73% 71% OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Borderline Normal 71% 68% 56% 51% 40% 34% 26% 20% 18% 14% 18% 17% 16% 11% 12% 0% Source of Referral (# referrals) 14% 17% 12% OSOC Youth Data 2007 – 2011 Referrals to Wraparound from Primary Sources Ohio Scales Functioning - Baseline 80% Children and youth referred by Child Welfare and Juvenile Justice are more likely to score not impaired than those referred by other sources Impaired 62% 61% 60% OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Borderline 61% 50% 49% 47% Normal 40% 33% 30% 28% 24% 20% 23% 23% 19% 17% 22% 16% 22% 17% 0% Source of Referral (# referrals) OSOC Youth Data 2007 – 2011 Total Enrollments by Impairment at Baseline Only 11% of enrolled wraparound youths score in the normal range on both Problems and Functioning scales Just under half of enrolled youths are critically impaired on both scales These data provide evidence that our referral networks are functioning well and that OSOC is providing services to appropriate families OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Normal, 408, 11% Impaired - Both Function and Problems, 1761, 49% Borderline, 516, 15% Impaired Problems Only, 618, 17% Impaired Function Only, 284, 8% OSOC Youth Data 2007 – 2011 6-Month Improvement in Problems / Functioning by Gender, Caregiver Form Improved, Problems OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Improved, Functioning Females who are rated impaired at baseline show clinically significant improvement at a higher rate at six months than males on both scales % of Impaired Youths w/Significant Improvement 70% 60% 59% 56% 53% 52% 50% 40% 30% 20% 10% 0% Female (P=407, F=333) Male (P=943, F=800) Gender (P = Problems count; F = Functioning count) OSOC Youth Data 2007 – 2011 Significant Improvement, Baseline to Months by Age – Caregiver Form No improvement OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Improved 70% 63% 63% 61% 60% 50% 394 40% 279 39% 37% 194 37% 30% 20% 179 229 112 10% 0% - 11 12 - 14 Age Group 15 - 18 Problems Scale Caregiver vs Youth Responses, by Item (N = 1983) Only five items were rated more or equally problematic by youths as by their caregivers 1.7 1.0 1.0 0.8 1.0 1.5 1.6 1.3 1.6 1.1 0.7 0.6 0.6 0.5 0.5 0.5 0.6 0.6 0.9 1.2 1.2 1.4 1.6 1.1 1.0 1.7 2.0 2.2 2.3 2.2 1.8 1.9 1.9 2.3 2.4 2.1 1.3 1.2 Response Value ( 0-5) 2.0 Youth 2.6 2.9 Caregiver 2.5 3.0 OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION 0.0 Scale values: Problems Scale Item = = = = = = Not at All Once or Twice Several Times Often Most of the Time All of the Time Functioning Scale Caregiver vs Youth Responses, by Item (N = 1766) OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Functioning Scale Item 1.5 1.8 2.5 2.6 2.7 1.9 2.4 1.8 1.7 2.0 2.3 2.5 2.8 2.8 2.1 2.5 1.7 2.0 2.4 2.8 2.3 2.7 2.2 1.8 1.7 2.3 2.6 3.1 2.5 3.1 2.6 2.7 2.4 2.3 2.5 1.9 2.7 2.5 Response Value (0-4) 3.2 Scale values: = Extreme Troubles = Quite a few Troubles = Some Troubles = OK = Doing Very Well Youth 1.6 Caregiver OSOC Youth Data 2007 – 2011 High Externalizing/High Internalizing and Improved High Externalizing 100% High Internalizing 80% 67% 61% 45% 88% 72% 65% 84% 86% 73% 68% 70% 61% 74% 65% 59% 37% 26% 0% 72% 68% 44% 40% 20% Improved Internalizing 95% 88% 60% Improved Externalizing OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION 29% 28% 70% 71% 58% 45% OSOC Youth Data 2007 – 2011 Impaired Youths with More than School Days Missed at Baseline Over School Days Missed OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Under School Days Missed No Clinically Significant Improvement Baseline to Months 100% Significant Improvement 100% 87% 80% 74% 63% 59% 60% 41% 40% 80% 80% 61% 60% 38% 39% 40% 26% 20% 20% 0% 0% to 11 (50) 12 to 14 (49) 15 to 18 (24) Age Group (# Youths) 20% 13% to 11 (83) 12 to 14 (69) 15 to 18 (50) OSOC WFI 2008 Pilot Study Ohio Scales Change by Caregiver WFI Score OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION o In 2008-09, OSOC piloted the WFI 4.0 with sites and wraparound facilitators The results were compared to the improvements in the Ohio Scales o Generally, as WFI ratings fell, so too did Ohio Scales improvement Caregiver WFI Avg Functioning Change 20 92.4 90.1 80 88.7 88.3 83.8 79.5 16 73.1 15 WFI Score 13 60 50.7 10 40 12 13 11 8 8 20 4 Point Change in Scale (Abs Value) 100 Avg Problem Change Future Plans for the Ohio Scales OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Add borderline cases to the outcomes and appropriateness analyses Explore the use of the Ohio Scales as a pre-referral team screener, to provide additional information for the team that staffs each referral In response to research, recommend that the age range for the Youth Form of the Ohio Scales be extended down to 9-year-olds Add usability features to the on-line data system to improve the ability of families and staff to use and interpret Ohio Scales results Resources OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Ash, S., & Weis, R (2009) Recovery Among Youths Referred to Outpatient Psychotherapy: Reliable Change, Clinical Significance, and Predictors of Outcome Child and Adolescent Social Work Journal 26, 399-413 Davis, T., Gavazzi, S., Scheer, S., Kwon, I., Lammers, A., Fristad, M., & Uppal, R (2011): Measuring Family Caregiver Perceptions of Support in Caring for Children and Youth With Mental Health Concerns, Journal of Social Service Research, 37:5, 500-515 Ogles, B M., Carlson, B Hatfield, D R., & Karpenkov, V V (2008) Models of case mix adjustment for Ohio mental health consumer outcomes among children and adolescents Administration and Policy in Mental Health and Mental Health Services Research, 35, 295-304 Dowell, K A & Ogles, B M (2008) The Ohio Scales Youth Form: Expansion and Validation of a Self-Report Outcome Measure for Young Children Journal of Child and Family Studies, 17, 291-305 Turchik, J., Karpenkov, V.V., & Ogles, B M (2007) Further Evidence of the Utility and Validity of a Measure of Outcome for Children and Adolescents Journal of Emotional and Behavioral Disorders, 15, 119-128 Carlston, D L & Ogles, B M (2006) The impact of items and anchors on parent-child reports of problem behavior The Child & Adolescent Social Work Journal, 27, 24-37 Ogles, B M., Carlston, D L., Hatfield, D.R., Melendez, G., Dowell, K., and Fields, S A (2006) The role of fidelity and feedback in the wraparound approach Journal of Child and Family Studies, 15, 114-128 Ogles, B M., Dowell, K., Hatfield, D, Melendez, G., & Carlston, D (2004) The Ohio Scales In M E Maruish (Ed.), The use of psychological testing for treatment planning and outcome assessment (3rd ed., Vol 2) (pp 275-304) Hillsdale, New Jersey: Lawrence Earlbaum Ogles, B M., Melendez, G., Davis, D C., & Lunnen, K M (2001) The Ohio Scales: Practical Outcome Assessment Journal of Child and Family Studies, 10, 199-212 Texas Department of Mental Health Mental Retardation (2003) Validation and norms for the Ohio Scales among children served by the Texas Department of Mental Health and Mental Retardation http://www.dshs.state.tx.us/mhprograms/RDMCAtrag.shtm E-TEAM OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION The Educational Training, Evaluation, Assessment, and Measurement (E-TEAM) Department of the University of Oklahoma is a full service department with senior researchers, data analysts, technical writers, and data base developers and managers representing decades of experience in all phases of research data processing E-TEAM also has several years’ experience as evaluation manager for the State of Oklahoma Systems of Care (OSOC) grant community, implementing its national and local evaluations as well as special OSOC collaborative studies with child welfare and juvenile justice Belinda Biscoe, Ph.D., Assistant Vice President, Public and Community Services, at the University of Oklahoma, College of Continuing Education, is the Director of E-TEAM E-TEAM has staff with extensive experience in tribal projects in conjunction with the American Indian Institute (also located at the University of Oklahoma) E-TEAM also has bi-lingual staff to facilitate communication with Spanish-speaking customers and subjects The Oklahoma Systems of Care E-TEAM evaluation staff: Melinda Anthony, Family Interviewer José Diaz, Data Analyst Kelly Phillips, Research Assistant Sharon Strait, Research Assistant Geneva Strech, Senior Research Associate Stephen Strech, Youth Interviewer John Vetter, Senior Research Associate Lisa White, Research Associate ... comparisons with CBCL and SDQ to validate the scales and to establish classification criteria for the 20-item Ohio Problems and Functioning scales: Problems Scale Score of 25 and above = critical... Change Future Plans for the Ohio Scales OKLAHOMA STATEWIDE SYSTEMS OF CARE EVALUATION Add borderline cases to the outcomes and appropriateness analyses Explore the use of the Ohio Scales as a pre-referral... screener, to provide additional information for the team that staffs each referral In response to research, recommend that the age range for the Youth Form of the Ohio Scales be extended down to 9-year-olds