USING ORGANIZATIONAL CHANGE SURVEY (OCS) TO EVALUATE CYFAR EFFECTIVENESS IN ADDRESSING COMMUNITY NEEDS LESSONS FROM A FOUR YEAR STUDY IN ILLINOIS

USING ORGANIZATIONAL CHANGE SURVEY (OCS) TO EVALUATE CYFAR EFFECTIVENESS IN ADDRESSING COMMUNITY NEEDS: LESSONS FROM A FOUR YEAR STUDY IN ILLINOIS Al Zwilling1 and Andre N Mbassa2 University of Illinois Extension1 and University of Illinois2 Abstract Documenting the current state of and changes in states’ abilities to develop and sustain effective programs for targeted groups is paramount to the successful implementation of many community projects This presentation provides guidelines to help CYFAR projects understand how to assess organizational change overtime and how to utilize findings in making program modifications throughout the course of a CYFAR project The University of Illinois Extension utilized the Organizational Change Survey (OCS) to establish general system baseline for measuring changes of the organization and changes in the communities as part of its state strengthening process Cross sectional primary data was collected from U of I Extension staff working with youth, families and communities, two local community advisory groups and task forces of one of the advisory groups over a four year period Initial findings indicated that this instrument can be sensitive to organizational and community change The instrument was used to establish a baseline for measuring Extension’s commitment to developing and implementing the project’s common vision and strategic plan and assess the progress over time There appears to be a link between offering staff development on specific topics (mini grant program) and organizational change Increased participation in training did not always reflect a change in the organization Rather, the quality of training and its relevance to the staff needs contributed to an increase in programming Project support and motivation in using communication technologies and internet resources enhanced staff effectiveness in conducting programming as well as extending their expertise to other community needs Introduction The objective of the CYFAR Program is to improve state extension system’s capacity to support community-based programs for children, youth and families at risk Over a four year period the University of Illinois Extension utilized the Organizational Change Survey (OCS) developed at the University of Arizona as a measure to evaluate and improve its community programs The goal of CYFAR at the state level is to “build capacity among vulnerable youth, families, communities and extension staff to pragmatically and effectively address problems related to youth in risky environments” (Peeples, Zwilling, Wiley & Spelke, 2000) At the community level the project has specific emphasis toward at risk issues while building staff development and capacity to address these issues The Knox County Healthy Youth Alliance (HYA) and Task Forces (TF) work towards reducing teen pregnancy, youth crime, school dropout and teen use of alcohol, tobacco and other drugs The HYA goals were for Knox County schools, communities and families to implement a comprehensive community based prevention education strategy that included: utilization of comprehensive health curriculums for grades K-12 in the schools; community training in understanding youth development, risk and protective factors, and effective community planning and evaluation The Prep Plus Advisory Committee (PPAC) in St Clair County was designed to provide a series of intensive personal and professional development opportunities for pregnant and parenting teens who have demonstrated their desire to achieve The second goal of this group was to develop the skills of group members in identifying and analyzing community issues, strategic planning and proposal writing Research on organizational change identifies several organizational indicators of readiness This section identifies these indicators as another way to examine if and how they have contributed to organizational change Change can vary in complexity from the introduction of relatively simple processes into small workgroups to transforming an entire organization Cummings and Worley (2005) organize the activities of managing change into five categories These categories are motivating change, creating a vision, developing political support, managing the transition, and sustaining the momentum Napier et al (1998) described eleven work principles for creating the capacity for change within institutions of higher education These work principles include leadership imperative, compelling futures, process by design, data gathering as a creative act, team learning, trust and relationships, changing frames of reference, large group intervention, action learning, governance for action and linking change and resource allocation As one examines these indicators of readiness, one can find their presence within the groups being discussed Research findings show that OCS instrument can be sensitive to organizational and community change Results indicate that over time, Extension staff at the state and local communities had a clear vision of the task assigned, and the training and support provided to each of the groups is contributing towards achieving the goals of the organization It is worth noting that community coalitions’ vision and strategic plan for CYFAR initiative was clear to its members at the start of the project while CYFAR vision evolved over a longer period of time for Extension staff at the state level In the area of training, community coalition members received more training on programs for at risk audiences compared to Extension staff at the state level The Technology Mini Grant process engaged over 122 Extension staff However, the level of utilization of CYFAR focused resources such as CYFERnet remained low for all groups indicating a need for providing incentives for participants in using this source of information Methods The Survey • The Organizational Change Survey comprises six components that cover topics ranging from implementation of common vision to training opportunities, collaboration within and outside of Extension, diversity of Extension staff, program sustainability and respondent characteristics • The survey was administered to Extension staff from across the state once in 2001, 2002, 2003, and 2005 • The survey was adapted and administered to both Healthy Youth Alliance and Task Force members in Knox County at four points in time and to Prep Plus Advisory Committee group in St Clair County over a period of two years • The OCS instrument consisted of two types of questions (multiple choice and text open end questions) and two scales (rating scales and agreement scales) • The 5-point Likert scale ranging from 1-5: 1- “Strongly Disagree” and 5- “Strongly Agree”, a 5-point agreement scale ranging from 1-5: 1“Too Little” and 5- “Too Much” and a rating scale ranging from 1-4: 1- “Poor” and 4- “Excellent” were used Data Collection and Analysis • Cross sectional primary data was collected from U of I Extension staff, HYA, TF and PPAC members • On-line and paper versions of the instrument were used to collect data • On-line data were recorded in a Microsoft Excel spread sheet and imported into SPSS statistical software • Paper versions of the instrument were mailed to participants from the two communities and responses were returned to the research team on campus • Paper survey was manually coded and data entered directly into the Data Entry program of SPSS • SPSS statistical software was used for data analysis • Data analysis included frequency distributions, descriptive statistics, and crosstabulations Results and Discussions • Extension Commitment And Staff Time Assignment To CYFAR Programming Figure presents the data regarding proportion of assignment devoted to children at risk Statistics show that at the state level over the four years of study, more respondents reported working zero to quarter time with at risk audiences With the exception of the first year participants, less than 40% of respondents devoted 1/2 to full time working with children at risk The trend was somewhat reversed at the community level In Knox County, over 50% of the HYA and TF participants devoted 1/2 to full time to programming for CYFAR Similar trend was observed with the Prep Plus Advisory Committee in St Clair County where all committee members in year devoted 1/2 to full time working with at risk audiences compare to less than 50% in year In assessing the state vision for CYFAR program, less than 30% or respondents in year and respectively agreed or strongly agreed that the state extension had articulated a clear vision for CYFAR program (Figure 2) Apparently, the vision was still evolving in the first two years and with the introduction of the University of Illinois Extension Strategic Agenda in 2003, the percentage of respondent agreeing or strongly agreeing more than double in the third and fourth year The Strategic Agenda identified and communicated to all extension staff a limited set of priorities including catering to at risk audiences in the state that Extension can pursue successfully to improve its programming, strengthen its organization, and make it more agile to pursue engagement in a rapidly changing environment At the community level, an even greater percentage of respondents (40% for HYA and TF) agreed or strongly agreed in the first year that their coalition had a clear vision and strategic plan for carrying out programming for at risk audiences This percentage increased rapidly to 100% the third and fourth year respectively Similar trend was observed in St Clair County • Assessment of Participants Knowledge and Expertise in Delivering CYFAR Programming Respondents assessed their knowledge and practice relative to principles of positive youth development, risk and resiliency factors, developing, implementing and evaluating programs, and policy and legislation as it affects youth at risk Table 1, and summarize the findings for all three organizations Less than 50% of all respondents at the state level rated themselves good or excellent on all six topics under evaluation, with the rating for knowledge in risk and resiliency posting the lowest score (10%) in the first year These scores greatly improved the second and third year for all six topics and remain substantially high the fourth year for principles of youth development, risk and resiliency factor and program evaluation This improvement in knowledge and expertise is the results of a series of trainings delivered at CYFAR conferences and other seminars in these topics The goals of the first statewide conference were to increase the participants understanding of and appreciation for the importance of working with diverse audiences and establishing U of I Extension’s strategic plan for working with children, youth and families in risky environments The goals of the second year conference were to increase the knowledge of the research that supports youth, family and community programs, recognize the importance of designing comprehensive programs for youth, examine previous and current CYFAR programs in practice and how to evaluate community programs No conference the third year, only the implementation of the Technology Mini Grant process The fourth year conference goals were to improve internal cooperation between teams and become aware of positive youth development collaboration and resiliency research The final year conference goals were to review the goals of the state strengthening process and see how the goals were achieved, increase participant understanding of longitudinal evaluation and select models of effective programming and evaluation for use in participant’s communities In addition to the annual Illinois Conferences, CYFAR offered a number of statewide professional development opportunities for Extension staff These topics included risk and resilience, helping youth increase their understanding of diversity and sexual health issues, teen pregnancy prevention and utilizing electronic tools for program development, implementation and evaluation In Knox county HYA project, 57% of participant rated themselves good or excellent in principles of youth development and knowledge of risk and resiliency factor in the first year (Table 2) The survey showed that participants rating improved over time for three areas only namely principles of youth development, risk and resiliency and programming for youth at risk Less than 50% of all participants rated themselves good or better in the remaining three areas evaluated Similar trend was observed in St Clair County where more than 50% of participants involved with Prep Plus project rated themselves good or excellent in principles of youth development, risk and resiliency and programming for youth at risk (Table 3) • Capacity Building Through Staff Development and Training Tables 4, and present data on extension staff participation in ten training topics At the state level, between 28% and 76% of respondents reported training participation, with half or more taking advantage of six of the ten topics in the first year (Table 4) Participants at the training were asked to rate the sufficiency of these trainings from 1=too little, 3=adequate, to 5=too much The mean ratings in the first year ranged from 2.11 to 2.79 Subsequent year’s results showed a remarkable improvement in training participation and rating, indicating that participants were interested in training and needed more as they progress in their work with at risk audiences Also, Extension perceived the need and provided necessary assistance to enhance the organization’s performance More than half of respondents were trained in all ten topics in the second year, in eight topics the third year and nine topics the fourth year respectively The mean rating also improved ranging between 2.12 to 3.09 the second year, 2.39 to 3.05 the third year and 2.24 to 3.00 the fourth year showing Extension’s commitment to providing adequate staff development A similar trend was observed in Knox County as shown by the results compiled and presented in Table While more than half of participants attended training in seven or more topics throughout the life of the project, their rating of training was even better with a mean sufficiency rating of or higher for five topics in year and four topics in year respectively In St Clair County, the majority of PPAC participants who received some form of training where trained in four or less topics (Table 6) Over a two year period, project participants at various trainings posted a mean sufficiency rating of for training on research-based information on youth growth, development and comprehensive health only • Access to Internet, Financial and Other Technical Resources Technology affects changes and the internet has virtually transformed society The OCS survey assessed Extension professionals’ use of internet for work relevant to children, youth, and families at-risk by measuring the frequency with which respondents accessed the CYFERnet web page and other websites, and the reasons for using internet technology Respondents reported using both CYFERnet and other websites with varying frequencies A greater proportion of participants used non-Extension websites and other state and national Extension websites compare to using CYFERnet, despite the various trainings and awareness campaigns in the value of using CYFERnet for programming The low usage of CYFERnet suggests that Illinois staff may need more training and encouragement to use this powerful tool The most common ways the internet is used throughout the four years of study are to locate background information for a news article or presentation, locate information or fact sheets to share with program staff and/or participants, and to locate information to develop new programs or modify existing ones (Table 7) Although 71% of respondents in year acknowledged using internet services to communicate with a colleague within the state, this percentage dropped substantially the following years Less than half of Knox County participants in year and used internet service to accomplish the seven tasks listed in Table However there was a substantial improvement in year and year where more than 50% of them reported using internet services for all tasks except participating in an online tutorial on interactive learning experience In St Clair County, internet resources were used at time period one mostly to locate background information for a news article or presentation (67% of respondents said yes), and to locate information or facts sheets to share with program staff and/or participants (67%) Whereas at time 2, Table shows that internet services were utilized to communicate with colleagues within the state (83%), communicate with a colleague in another state (67%), locate information to develop new programs or modify existing programs (67), and to locate information for news article or presentation Figure presents responses to the question: Our state extension service has allocated financial resources specifically for work in CYFAR The charts show that in year an year 2, less than 20% of participants agreed to receiving financial resources to carry out programs for at risk children However this percentage rose dramatically to 50% in year and 59% in year This may stem from the fact that during year 3, CYFAR offered minigrants for interdisciplinary groups to utilize technology in the development and delivery of programs to reach youth and families in risky environments This process seemed to have engaged the largest number of Illinois Extension staff of any of the CYFAR interventions over the course of the project Over 122 different individuals were named in the proposals This process proved to be successful, in that all groups completed their technology projects, shared their accomplishments at the 2004 statewide conference, and have sustained their projects Conclusions • Sharing a common vision and strategic plan is crucial in designing activities to accomplish a goal While knowledge of the vision grew, after four years some participants were still unsure or disagreed that Illinois has a clear vision for CYFAR Extension should provide more information to its personnel on the state vision as it fits into the national agenda • Training in areas relevant to CYFAR programming will ensure the success of the program Extension services should emphasize the need and encourage staff involved with at risk audiences to participate at seminars and workshops For instance, training on project evaluation should be promoted since knowledge in evaluation guidelines could assist participants in focusing their programs on appropriate activities • CYFERnet is a valuable internet resource with useful information on children, youth and families at risk that Extension staff at state and community levels should utilize Extension should devise a means to promote the use of this source of information One could integrate the CYFERnet resource into all professional development for youth, family and community staff offered through U of I of Extension References Cummings, T G., and Worley, C G (1993) Organization Development and Change (6th ed.) Cincinnati, OH: South-Western College Publishing Napier, R., Sidle, C C., Sanaghan, P., and Reed, W S (1998) Metamorphosis: Creating the Capacity for Change, National Association of College and University Business Officers, January, pp 18-28 Peterson, D.J., Roebuck, J.C, and Betts, S.C (2003) Iowa Cooperative Extension’s Capacity to Support Programs for Children, Youth and Families at Risk: Results of the Third Round of the Organizational Change Survey Arizona Youth, Family and Communities The University of Arizona Tucson, Arizona Peeples, G.G., Zwilling, A., Wiley, A., and Spelke, K (2000) Building Extension Capacity and Adding Strengths in Local Communities in Illinois State Strengthening proposal funded by USDA Urbana, IL: University of Illinois Table 1: Sate Staff Rating of Knowledge and Expertise Relative to Working with At-Risk Youth (%) Topic Poor 36 Year Good/Excel Fair 81 19 64 26 56 Programming for youth at risk 26 52 22 Obtaining resources and future 39 13 45 48 Principles of positive youth Year Good/Excel Fair 16 48 Poor Year Good/Excel Fair 100 Year Poor Good/Excel Fair Poor 82 18 10 18 26 100 0 65 33 79 21 99 76 22 48 61 39 100 0 39 45 16 16 39 64 36 100 0 39 49 12 13 39 64 36 100 0 29 57 14 development and comprehensive health Risk and resiliency factor funds to support programs for youth at risk Evaluating social programs for youth at risk Policy and legislation as it affects youth at risk Table 2: Rating of Knowledge and Expertise Relative to Working with At-Risk Youth (Knox HYA/TF) Topic Year Good/Excel Fair Poor Year Good/Excel Fair Year Poor Good/Excel Fair Poor Year Good/Exce Fair Poor 15 57 29 14 52 26 22 92 l 85 comprehensive health Risk and resiliency factor 57 43 57 39 83 17 92 Programming for youth at 29 57 14 57 26 17 84 8 69 31 risk Obtaining resources and 43 14 43 26 52 22 25 75 38 54 43 29 28 35 52 13 33 58 39 61 14 57 19 36 44 30 25 68 15 85 Principles of positive youth development and future funds to support programs for youth at risk Evaluating social programs for youth at risk Policy and legislation as it affects youth at risk Table 3: PPAC Rating of Knowledge and Expertise Relative to Working with At-Risk Youth (%) Topic Year Good/Excel Fair 10 Poor Year Good/Excel Fair Poor Principles of positive youth 83 17 100 0 comprehensive health Risk and resiliency factor 67 33 50 50 Programming for youth at risk 75 25 100 0 Obtaining resources and future 33 67 33 67 50 42 17 83 Policy and legislation as it affects 42 42 16 33 50 17 development and funds to support programs for youth at risk Evaluating social programs for youth at risk youth at risk Table 4: Training Received in Various Topics and Sufficiency Ratings (State Data) Topic Year Year %Yes MSR* %Ye MS Finding resources to strengthen, expand and 52 2.43 s 65 institutionalize programs for youth at risk Principles of collaboration (initiating, 55 2.62 56 11 Year %Ye MS Year %Ye MSR R 2.55 s 53 R 2.39 s 53 2.24 2.95 60 2.88 58 2.47 facilitating, maintaining, evaluating) The impact of your own values, perceptions, 71 2.68 71 2.67 63 2.75 72 2.66 audiences Empowering members of the community to 35 2.11 55 2.84 44 2.71 53 2.34 identify and address the needs of youth at risk Research-based information on youth growth, 74 2.79 68 3.09 72 2.94 86 2.76 development and comprehensive health Recruiting and preparing volunteers to work 33 2.45 53 2.13 60 2.59 54 2.27 with diverse audiences Designing and implementing an evaluation 28 2.25 64 2.14 76 2.53 62 2.41 29 76 55 2.29 2.76 2.64 59 82 72 2.12 2.38 2.62 40 89 74 2.48 3.05 2.92 46 89 81 2.25 3.00 2.95 skills and prejudices on serving diverse process How to facilitate organizational change Use of computers Use of electronic communication *MSR: Mean Sufficiency Rating Range of Responses: 1= too little, = adequate, = too much Table 5: Training Received in Various Topics and Sufficiency Ratings (Knox HYA/TF) Topic Year Finding resources to strengthen, expand and institutionalize programs for youth at risk 12 Year Year Year %Ye MSR* %Ye MSR %Ye MSR %Ye MSR s 67 1.67 s 71 2.73 s 100 2.83 s 62 2.92 Principles of collaboration (initiating, facilitating, 67 1.83 61 2.81 100 3.33 77 3.31 50 1.17 70 2.91 100 3.08 77 3.00 33 1.00 38 2.52 92 2.75 54 2.85 50 1.67 64 3.00 100 3.00 69 3.23 20 0.40 52 2.61 100 3.08 46 2.69 Designing and implementing an evaluation process 50 1.00 67 2.57 92 2.50 46 2.92 How to facilitate organizational change 50 1.00 59 2.41 92 3.00 54 2.77 Use of computers 40 1.40 71 2.91 92 2.67 46 2.62 Use of electronic communication 50 1.17 43 2.48 92 2.83 62 3.00 maintaining, evaluating) The impact of your own values, perceptions, skills and prejudices on serving diverse audiences Empowering members of the community to identify and address the needs of youth at risk Research-based information on youth growth, development and comprehensive health Recruiting and preparing volunteers to work with diverse audiences *MSR: Mean Sufficiency Rating Range of Responses: 1= too little, = adequate, = too much Table 6: Training Received in Various Topics and Sufficiency Ratings (PPAC) Topic Finding resources to strengthen, expand and Year %Yes MSR* 58 2.83 Year %Yes MSR 17 2.50 institutionalize programs for youth at risk Principles of collaboration (initiating, facilitating, 42 67 maintaining, evaluating) 13 2.42 2.50 The impact of your own values, perceptions, skills 58 2.92 83 2.83 and prejudices on serving diverse audiences Empowering members of the community to 33 2.08 17 2.33 identify and address the needs of youth at risk Research-based information on youth growth, 42 2.83 33 3.00 development and comprehensive health Recruiting and preparing volunteers to work with 00 1.83 33 2.33 diverse audiences Designing and implementing an evaluation process How to facilitate organizational change Use of computers Use of electronic communication 33 42 58 58 2.33 2.25 2.50 2.33 33 40 40 50 2.67 2.60 2.20 2.50 *MSR: Mean Sufficiency Rating Range of Responses: 1= too little, = adequate, = too much Table 7: In the Past 12 Months, I Have Accessed Internet Resources To: (State) Use Year Year Year Year Communicate with a colleague within my state Yes 71% No 29% Yes 48% No 52% Yes 49% No 51% Yes 53% No 47% Communicate with a colleague in another state 42% 58% 30% 70% 32% 68% 36% 64% Locate background information for a news article or 58% 42% 53% 47% 59% 41% 67% 33% presentation 14 Locate information of fact sheets to share with 71% 29% 71% 29% 66% 34% 72% 28% 68% 32% 53% 47% 66% 34% 72% 28% Locate information on program evaluation 42% 58% 40% 60% 39% 61% 49% 51% Participate in an online tutorial or interactive learning 26% 74% 28% 72% 33% 67% 37% 63% program staff and/or participants Locate information to develop new programs or modify existing programs experience Table 8: In the Past 12 Months, I Have Accessed Internet Resources To: (HYA/TF) Use Year Year Year Year Communicate with a colleague within my state Yes 14% No 86% Yes 48% No 52% Yes 67% No 33% Yes 77% No 33% Communicate with a colleague in another state 14% 86% 26% 74% 50% 50% 38% 62% 15 Locate background information for a news article or 43% 57% 57% 43% 83% 17% 92% 8% 28% 72% 43% 57% 83% 17% 69% 31% 43% 57% 52% 48% 92% 8% 85% 15% Locate information on program evaluation 43% 57% 26% 74% 67% 33% 46% 54% Participate in an online tutorial or interactive learning 14% 86% 35% 65% 8% 92% 31% 69% presentation Locate information of fact sheets to share with program staff and/or participants Locate information to develop new programs or modify existing programs experience Table 9: In the Past 12 Months, I Have Accessed Internet Resources To: (PPAC) 16 Use Year Year Communicate with a colleague within my state Yes 50% No 50% Yes 83% No 17% Communicate with a colleague in another state 33% 67% 67% 33% Locate background information for a news article or presentation 67% 33% 67% 33% Locate information of fact sheets to share with program staff and/or 67% 33% 50% 50% Locate information to develop new programs or modify existing programs 58% 42% 67% 33% Locate information on program evaluation 50% 50% 33% 67% Participate in an online tutorial or interactive learning experience 25% 75% 17% 83% participants 17 Figure 4: Research: Effective Change Management/Capacity Leadership Imperative Motivating Change • Creating Readiness • Overcoming Resistance Compelling Futures Creating a Vision Process by Design • Energizing Commitment • Describing Desired Future Data Gathering as a Creative Act Political Support Team Learning • Assessing Power Base • Influencing Stakeholder Trust/Relationships Managing Transition Changing Frames of Reference • Project Management Large Group Intervention Sustaining Momentum • Resources, Support System • New Skills, Reinforcing Action Learning Cummings and Worley, 2005 Governance for Action Linking Change and Resource Allocation Napier, Sidle, Sanaghan, & Reed, 1998 18 ... appears to be a link between offering staff development on specific topics (mini grant program) and organizational change Increased participation in training did not always reflect a change in. .. organizational indicators of readiness This section identifies these indicators as another way to examine if and how they have contributed to organizational change Change can vary in complexity from. .. The mean ratings in the first year ranged from 2.11 to 2.79 Subsequent year? ??s results showed a remarkable improvement in training participation and rating, indicating that participants were interested