Outcomes.USF.performance monitoring overview

Introduction This is a brief introduction to the latest pieces that we are sending to you The main focus of the material is on the development of a strong performance measurement/monitoring and continuous quality improvement system We emphasize this because we have found it to be an important component of effective service systems It is also our experience that within states that have had class action lawsuits, the inclusion of a strong performance measurement and monitoring system has been one of the most effective components for bringing about change, and for guiding that change In the material that we are sending, we have emphasized that a performance measurement system should be based on the overall outcomes that a system is seeking to achieve, and on the theory of what will lead to the achievement of those goals (we call this a “theory of change” approach) Basically we see two main outcomes, and have included figures to illustrate each: an increase in access to care for children in need of mental health services; and an increase in effectiveness of care for children who receive mental health services Within the goal on improved effectiveness, the primary emphasis of our focus is not on symptom reduction but on measures of community functioning This is consistent with the Rosie D ruling, and the emphasis within Medicaid on improving overall functioning It is also consistent with the recommendations of the President’s New Freedom Commission on Mental Health, which emphasized a recovery and resilience model designed to enhance community functioning rather than a more narrow model targeted at the goal of symptom reduction (See Figure 1) We encourage you to think of the development of a performance measurement system as a staged process The system has to begin by developing capacity for identifying effective practices and outcomes, and then to benchmark effective practices, next to create and evaluate supports for reaching those benchmarks, and finally to create mechanisms for re-evaluating benchmarks Over time, the system should become more efficient at delivering effective services; thus benchmarks should evolve over time to reflect system maturation From this perspective, performance monitoring is a way to teach persons about system expectations for performance, monitor system progress, and to evolve the system to continuously pursue better outcomes Figure We are therefore providing the following: ♦ Two figures that provide an overview visualization of levels of monitoring; ♦ A summary of general principles of outcome assessment; ♦ A summary of measures of child outcomes and youth and family satisfaction with recommendations; ♦ A more detailed description of what is being currently done in different domains that we consider to be state of the art Overall, we make the following recommendations: Whatever is done should include transparency and public presentations of findings, regular and frequent review by key decision-makers, feedback loops to those who have provided the data, and culturally and linguistically competent measures and data collection procedures; The major focus should be level of success in achieving the desired outcomes: increased access to care and access in a timely manner; increased effectiveness of care resulting in better functioning in the community, and child and family satisfaction with care; The child measures that seem to be most closely aligned with the goals of increasing community functioning, and have both the best psychometric data and field experience are the CAFAS and the CANS-MH, and the MHSIP measures of child and family satisfaction (YSS and YSSF); It is also essential to collect data on a regular basis on the process of providing services, with a focus on those processes that are believed to be most essential for achieving the desired outcomes; Overall, it is recommended that information be collected, at a minimum, on: ♦ The screening process with regard to its timeliness, its adherence to standards, the information it provides on characteristics of children and families, and its role in providing access to care for those in need; ♦ The treatment planning process, using a measure that involves direct observation such as the Wraparound Observation Form; ♦ The overall treatment process, using an in-depth approach such as Service Testing and System of Care Practice Review, a measure that looks at the whole process such as the Wraparound Fidelity Index, and through gathering data, much as Hawaii does, on the implementation of the recommendations of the treatment planning process; ♦ The actual services provided and the cost of the services; ♦ Periodic semi-structured interviews with key stakeholders from the mental health and other related systems, and also with parent leaders, to secure their feedback on how well the system is functioning and what might be done to improve it; ♦ Child outcomes, as measured by the CANS-MH or the CAFAS, with overall data also being disaggregated to determine the degree of change for children with different constellations of problems, and different backgrounds; ♦ Child and family satisfaction using measures such as the YSS and YSSF and also incorporating within the Service Testing or SOCPR some less structured questions on the child and family experience with treatment; ♦ System outcomes, such as number and percentage of children being served in out-of-home care, and percentage of children in need who are receiving home and community-based services Figure Performance monitoring is an essential element of system reform and continuous quality improvement Planful identification of key data points and feedback loops that inform practice and decision making can be powerful instrument for system change Such data can be used to highlight meaningful progress and areas for improvement at the agency and system levels, as well as supervision tools and a means of monitoring the progress of individual children and families This seems to be particularly effective when the expectation is that the results of the performance monitoring will be presented publicly Public dissemination of system or agency ‘report cards’ have been used as system reform tools in Hawaii, where regional directors are required to present the results for their region each quarter to a meeting of key stakeholders including state-level administrators, legislators and staff, and family members, and quarterly reports are posted on the agency website for the use of consumers, key decision makers and the general public alike Likewise, the state of Indiana posts consumer satisfaction data for all of its community mental health centers for review and use in health care decisions The transparency and power of such methods of accountability extends beyond the behavioral health sector – for example, in his administration as New York Mayor, Giuliani required city agencies to post a set of relevant indicators on a weekly basis (Giuliani, 2002) Recognizing the both the need for information on which to base decisions, as well the need for parsimony, it is suggested that performance monitoring systems be premised on the principles of relevance, timeliness and appropriateness Relevant systems answer the questions, ‘What worked?’, ‘What didn’t work?’, ‘What could be better?’ Successful outcomes should be defined at the appropriate level, but should originate from the point of view that children and families are experiencing improvements in their well being and are satisfied with the services and supports they receive While Massachusetts currently collects some system level indicators, such as readmissions to acute settings (Commonwealth of Massachusetts, 2004), and child and family level indicators, such as change scores on the Child and Adolescent Functional Assessment Scales (Hodges, 1994), it is recommended that the current system monitoring efforts be expanded to include indicators that reflect the processes and outcomes at the child and family level, the agency or provider level and at the system level Determining the key activities at each of these three levels, monitoring these activities, establishing rapid and ongoing feedback loops for each activity and making this information easily available to all stakeholders is essential to system reform efforts In this section, key activities at each level are identified and suggestions for indicators are proposed While service systems are inherently dynamic, discussion herein will be discussed in terms of three levels; child and family level, the agency and / or program level, and, the service system level Within each of these levels, there are multiple points at which data should be collected and inform processes and decisions internal to these domains Ideally, information also flows between the levels Child and family level – just as a key principle of service and support delivery is that services are family driven and child focused, development or reform of performance monitoring systems should begin with the examination of key points in service and support delivery Screening and comprehensive assessment– Most children and families first encounter the system at the point of screening, and so it is critical that data be collected on how well the screening function is being carried out, as well as on child and family characteristics These data inform activity at the service system level, the agency and within the child and family level (See Table below) Those children, youth and families that ‘screen in’ must participate in a comprehensive assessment process that is ecological, encompassing all life domain areas, and inclusive of child and family strengths to be used in service and support planning In Michigan, for example (Wotring, personal communication, 2006), the Child and Adolescent Functional Assessment Scales (CAFAS) is used as one element of eligibility criteria for services and is also used in the determination of the most appropriate level of care, e.g., clinic based or intensive home based services (Hodges & Wotring, 2000) The CAFAS is then collected quarterly while the child is enrolled in services and at discharge These data are used in service planning and monitoring at the child and family level They are also used in supervision of direct care staff, at the agency level At the agency level, data are also used to assess whether children and families are being served at the appropriate level of care If, for example, children served in outpatient services were not improving over time, the plans of these children would be revisited, to determine a more appropriate type or intensity of services At the system level, these data are also aggregated One example of how these data are used was in the selection of evidence based practices that match the characteristics of the children and families being served Table Activity Screening & comprehensive assessment Data element(s) Child and family characteristics & functioning Child and family characteristics & functioning Child and family characteristics & functioning Level Informed/ Level of aggregation System level Agency & direct care staff level Child & Family level Examples of data use • • • • • Planning service mix that addresses needs and characteristics of the population Eligibility determination Ensuring staff characteristics, staff development, and agency policy are appropriate for children and families served (cultural competence); Eligibility determination Needs and strengths are identified for further assessment and use in service planning Treatment, service and support planning, and service and support implementation – The activity areas of treatment / support planning and service and support implementation are difficult to disentangle once an initial plan has been developed because they are joined in an iterative cycle of continuous planning, service delivery, monitoring and feedback For this reason, the two areas of activity are discussed in a single section On completion of the comprehensive assessment and outcomes monitoring tool, e.g., CAFAS or CANS, the needs and strengths of the child and family identified in the assessment guide the service and support planning process, as well as inform other levels within the service system (See Table2) At this level, emphasis of data should focus on the quality and inclusiveness of the planning process Quality should include measures of whether identified services address the needs and strengths of children and families across life domain areas, are inclusive of professional and informal supports that touch the family, and are consistent with principles of family-driven and culturally competent care Of equal importance is the collection of the intensity and duration of identified services and interventions to meet the needs and build on strengths While most states collect information on the type, intensity and duration of services, if only for billing purposes, few states systematically capture information on the process of service planning Two notable examples of states that collect and use these data are Hawaii and Arizona [Please refer to treatment planning information sent under separate cover.] Hawaii uses multiple data sources to monitor service planning and delivery One approach to better understanding service delivery is in Hawaii’s requirement that providers submit monthly reports that 1) document the service type, intensity and duration, 2) include what it calls ‘core practice elements’, i.e., the nature of the intervention, e.g., cognitive behavior approach, 3) documents progress toward treatment or service goals, and 4) provides the child and family specific outcomes of services In tandem with functioning and level of care data gleaned through quarterly completion of the CAFAS and CALOCUS, care managers1 (a.k.a case managers / care coordinators) use service data in planning and service delivery for individual child and families Likewise, their supervisors have access to these data for individual supervision and staff development purposes These data are also aggregated and reviewed at the Branch (regional) and state levels to identify effective services for expansion and to provide corrective feedback to provider agencies Hawaii also uses a qualitative approach to understanding service planning and practice, called service testing (Foster & Groves, need cite) This process uses case level data, i.e., the services and processes that support individual children and families, to provide agency and system level feedback on system performance The service testing approach used by Hawaii uses teams of state employees and Parent Partners (caregivers of children with EBD) that conduct interviews with caregivers, youth, providers and state employees These data are then used to provide feedback at the agency level to identify strengths and weaknesses in their work with children and families, and identify agency processes and policies that promote or inhibit improvement in child and family functioning These data are also aggregated for use by state level committees to assess system performance and provide feedback to the executive level staff The use of methods like service testing and system of care practice review (SOCPR) is time intensive and relatively costly but seems to be well worth it—it appears to be one of the most powerful Hawaii’s case managers are state employees Private providers, for or non profit, contract with the state to provide therapeutic and supportive services mechanisms for genuinely determining how well the system is functioning, and how well young people are doing, and for identifying areas in need of improvement Specific measures that address the service planning and / or delivery process used elsewhere, although not on a statewide level, include the Wraparound Fidelity Index (WFI, Bruns, Burchard, Suter, & Force, 2005), the Wraparound Observation Form (WOF, Epstein et al, Epstein M H., (1998) and the System of Care Practice Review (SOCPR, Hernandez, Worthington & Davis, 2005) Measures like the WFI are completed by direct care staff, caregivers and youth, and are brief and simple to complete, and would be relatively easy to incorporate into practice to provide rapid feedback to those making service and support decisions, the multidisciplinary child and family team These data can then be used to guide subsequent planning and activities, in supervision of direct care staff and to identify system barriers to seamless service provision Measures such as the WOF and the SOCPR provide detailed information on staff behaviors and system factors that facilitate or impede provision of appropriate services in the context of the child and family team process and service delivery Administration of theses measures for a sample of youth and families, provides rich information that can guide advocacy for change at the system level and in supervision Further, both measures are in the public domain and can be incorporated into and enhance existing quality assurance processes or be used to frame current monitoring In our work with local communities in Florida, practitioners and families are very receptive to feedback generated by these two measures, because strengths are identified, the areas for improvement identified are specific in nature, feedback is provided in the context of quality assurance, and enough detail is provided to generate plans for coaching and supervision The combination of quantitative and qualitative data collection within a single measure, means data can be aggregated for use at the policy level, for example, to advocate when service gaps are identified or when cross system funding or training is called for These more intensive data collection procedures, such as service testing, SOCPR, and WOF, can be used for a sample of children and families—they need not be used for all children Table Activity Data element(s) Treatment, service and support planning Service type, intensity and duration Staff behavior; Service type, intensity and duration Level Informed/ Level of aggregation System level Agency & direct care staff level Examples of data use • • • • Service or treatment goals and objectives; Service type, intensity and duration Child & Family level • Identify gaps in the service array In combination with characteristics and functioning, can inform financing, e.g., capitation and risk adjustment Clinical supervision, staff development and training- Ensure staff have the skills to plan services across life domain areas Barrier busting - identify agency and system barriers to successful service implementation, e.g., eligibility criteria that prohibit or impeded access to needed services Monitoring pace of goal attainment in order to adjust service delivery to better to meet the needs of children and families Child and family outcomes and satisfaction – Child and family outcomes and satisfaction are the drivers of effective performance measurement systems As described above outcomes monitoring is an integral and ongoing part of the assessment, planning and service delivery process Beginning at the time of initial screening, regular administration and use of a tool such as the CANS or the CAFAS, that can be used both to inform practice and to assess impact is vital An important companion to these measures are surveys of child and family satisfaction, which can begin as early as the first quarterly assessment period (See Table 3) The Mental Health Statistics Improvement Project (MHSIP, 2001) advocates for the use of measures that address the domains of access, participation in treatment, cultural sensitivity, satisfaction with services and perceived outcomes These caregiver and youth version of these brief measures are easy to complete and provide quick feedback at the individual staff member, program, agency and system levels We are attaching another document that summarizes some of the available measures of child functioning We recommend the use of either the CANS or CAFAS Table Activity Outcome and satisfaction monitoring Data element(s) Level Informed/ Level of aggregation System level Examples of data use • Identify system barriers for advocacy Agency level • • Child & Family Level • Provider report cards (Indiana) Assess whether families perceive progress, receipt of appropriate and culturally competent services Children & families have access, range, options, & continuity of services Individual child and family ‘report’ cards, which can be used by families to understand their progress and talk to other persons involved with their families, e.g., therapists, teachers, etc Increased satisfaction with services • • Staff, program and agency levels – As suggested in the introductory section, monitoring and feedback loops need to be adopted at all levels of the system for optimal system performance The types of information gathered at this level included staff characteristics and behaviors, supervision and coaching, agency culture, administrative supports and the quality of leadership Provision of culturally and linguistically appropriate services depends, at least in part, on the recruitment and retention of staff that mirror the ethnic composition of the eligible population Clearly, staff qualifications and practices also effect quality of care, so it is important for supervisors to have information to assess staff performance relative to espoused values and principles, that staff have and are able to competently demonstrate needed skills, and that supervisors have the ability to provide coaching, corrective action for their staff (See Table 4.) Agency culture and policy also has a relationship to child and family improvement Access to care is contingent on the child and family’s ability to get to needed services in a timely way Areas to monitor might include the amount of time from a family’s first contact to the time of the first appointment, whether agency hours of operation facilitate or impede a family’s ability to get to their appointments, the length of agency waiting lists by service type, and investigation into ‘no show’ rates Other agency practice and policy that effect service provision include policies on ‘comp’ or flexible work schedules and so that staff can meet with families and their informal supports at convenient times Easy access to and availability of flexible funding to provide supports to families that may fall outside of the traditionally funded services or items, is another important 10 aspect of service provision that is governed by agency policy Information technology, or lack thereof, also has an effect on staff members’ ability to access and share information with families and others involved in the families’ lives Table Level System level Domain Service mix and capacity Agency Level Supervision & training Agency Level Human Resources Key processes Contracting tied to • adoption and implementation of EBPs Data based tools that • guide supervision across life domain areas, focus on ameliorating key needs and building on strengths of children and families served Staff recruitment and • retention Measures Ongoing review of eligible population characteristics to guide selection of appropriate EBPs Incorporation of measures into CQI and aggregation of data at the individual staff level, program / unit level and agency level to staff training and development Development of plans that encourage identification and selection of staff that reflect the cultural and linguistic characteristics of the eligible population System level – Performance monitoring at the system level informs policy and regulatory functions, financing, service mix and capacity, and human resource support and development Data sources that inform performance monitoring at this level include aggregated child and family characteristics, outcome and satisfaction data, treatment and support planning data, agency level and administrative datasets that contain billing and service utilization data (See Tabsle 5) The design of these systems should allow for generation of standardized reports, as well as adhoc reports that can be easily prepared to address emerging policy issues As is true for all levels in a performance monitoring system, it is not enough that the data be collected, but that a rapid feedback capacity be incorporated so that information can be shared with system level stakeholders, family members who use services, as well as to stakeholders including at the regional and agency levels The final essential element of useful performance monitoring is that the information reported is easily understood and actionable 11 Table Level System level Domain Policy & regulatory Key processes Ensuring access & equity • • Finance Establishment of braided or blended funding mechanisms • Service mix & capacity Ensure appropriate community based & residential services are available to meet needs of the eligible population • • • Human resources Ensure state of the art training and supports are available at the agency and practitioner level • Measures Dissemination of information & referral that are culturally and linguistically appropriate to the eligible population; Review of data to identify health care disparities Alignment of agency policy and eligibility criteria to allow consumers seamless movement throughout the system Review of characteristics and service use of youth in residential treatment settings to ensure appropriateness Review of community based service capacity relative to available prevalence data Identification of special populations in need of additional supports Dedication of training resources to purchase of EBP curriculum, capacity for certifying trainers in EBPs, number of agencies with staff trained 12 References Bruns, E.J., Burchard, J.D., Suter, J.C., & Force, M.D (2005) Measuring fidelity within community treatments for children and families In Epstein, M., Duchnowski, A., & Kutash, K (Eds.) Outcomes for Children and Youth with Emotional and Behavioral Disorders and their Families, vol Austin, TX: Pro-ED Childs, C (2004) Fiscal years 2005-2007 State Mental Health Plan Boston, MA: Commonwealth of Massachusetts Executive Office of Health and Human Services, Department of Mental Health Epstein M H., Jayanthi, M., McKelvey, J., Frankenberry, E., Hardy, R., Dennis, K., et al (1998) Reliability of the wraparound observation form: An instrument to measure the wraparound process Journal of Child and Family Studies, 7(2),161-170 Foster & Groves Guiliani, R.W (2002) Leadership New York: Hyperion Hernandez, M., Worthington, J., & Davis, C.S (2005) Measuring the fidelity of service planning and delivery to system of care principles: The system of care practice review (SOCPR) (Making children’s mental health services successful series, 233-1.) Tampa, FL: University of South Florida, The Louis de la Parte Florida Mental Health Institute Hodges, K (1990, 1994 revision) Child and Adolescent Functional Assessment Scale Ypsilanti, MI: Eastern Michigan University, Department of Psychology Hodges, K & Wotring, J (2000) Client typology based on functioning across domains using the CAFAS: Implications for service planning Journal of Behavioral Health Services and Research, 27, 257-270 Mental Health Statistics Improvement Program (2001) Youth services survey www.mhsip.org 13 ...Figure We are therefore providing the following: ♦ Two figures that provide an overview visualization of levels of monitoring; ♦ A summary of general principles of outcome assessment; ♦ A summary... and a means of monitoring the progress of individual children and families This seems to be particularly effective when the expectation is that the results of the performance monitoring will... service delivery, monitoring and feedback For this reason, the two areas of activity are discussed in a single section On completion of the comprehensive assessment and outcomes monitoring tool,