EVALUATION 331 Figure 12.4. Organization Development Evaluation Model 10. Administer Longitudinal Evalutation 1. Do Assessment 7. Analyze Results 8. Present Results 9. Make Revisions 2. Determine OD Needs 3a. Determine Level of Evaluation 3b. Design Intervention 4b. Develop Evaluation Instrument 4a. Develop Intervention 5b. Administer Formative Evaluation 5a. Implement Intervention 6. Administer Summative Evaluation 18_962384 ch12.qxd 2/3/05 12:18 AM Page 331 IMPORTANCE OF EVALUATION If you don’t know where you are going, any path will get you there. And if you don’t know where you are, you’ll never know where you are going. The process of evaluating is truly about determining whether the path you are on has been the most efficiently executed and effective path for the system you are in. In turn, this will influence how you move forward into the future. The funda- mental premise is self-awareness. A system of any size—from one person to a whole organization—will be most amenable to change when it has all relevant data. The more that the people in a client system are able to understand them- selves, the greater the likelihood that they will take ownership in determining how to best move forward. Within this context, the key change agents are more able to influence the sys- tem with data that demonstrate success and failure. Action learning focuses on important outcomes—What was planned? What happened? What was learned? What will be done next time? One critical step to creating learning is addressed in the question: What happened? Success and failure are concepts critical to cre- ating a learning organization, and this is where the consultant as intervener comes into the picture. By providing the system with avenues for having a bet- ter understanding of itself through rigorous evaluation, the system will begin to change its thinking paradigm. When the system begins to change its thinking paradigm, the learning has begun. There is no other more powerful way to cre- ate change than through learning rooted in the shifting of how we think. This learning is what will lead to new and sustainable organizational habits that result in long-term health and prosperity. WHERE EVALUATION FITS IN THE INTERACTION WITH CLIENTS As indicated earlier, it is important to establish the evaluation processes during the pre-launch phase. In this way, what is important for the client, the consul- tant, the participants, and the process can be established before the process is begun. With a plan in place, continuous improvement in the OD process, based on identified evaluative feedback, can occur. Evaluation can occur at three points in the OD process: during the process (formative), at the conclusion of the process (summative), or at some time after the process has ended (longitudinal). The most important of these times is during the OD process—formative evaluation. To discover that something could have been done differently or that the process did not create the intended 332 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION 18_962384 ch12.qxd 2/3/05 12:18 AM Page 332 outcomes after the process is over does not allow for adjustments or modifica- tions along the way. WHO SHOULD DO THE EVALUATION? This decision, too, should occur during the pre-launch phase. There are three common answers to the question of who should perform the evaluation: the client organization, the consultant, or a third party. Many factors will influence where the responsibility for evaluation will rest. Given the preference for for- mative evaluation expressed earlier, the responsibility for evaluation will fall equally with the consultant and the client. Regular, ongoing evaluation will pro- vide feedback continuously, if so designed, and will allow the consultant and the client to partner to maximize the potential in the OD process. For summa- tive and longitudinal evaluation, there are pros and cons for each potential eval- uator. These are summarized in Exhibit 12.1. KEY MEASURES IN OD When taking a systems approach to measuring in OD, we advocate the inclu- sion and integration of six key dimensions: performance, morale, emotion, crit- ical customer service behaviors, organization climate, and personal characteristics (Cady & Lewis, 2002). Each component falls into one of two categories of mea- surement, soft or hard. Soft measures are perceptual in nature. That is, the scor- ing of soft measures is based on an individual’s judgment or opinion. For example, an employee is asked to respond in a survey to the statement, “I am satisfied with my job.” The employee chooses a number on a seven-point Likert scale ranging from strongly disagree with the statement (equal to 1) to strongly agree with the statement (equal to 7) (Rea & Parker, 1997). Hard measures are observable and can be objectively counted (that is, not based on opinion), often based on behavior or units of some kind. For example, the number of units sold or made is a hard measure. Taking a systems approach to evaluation is important in encouraging mem- bers of an organization to understand how their actions and behaviors (soft measures) influence important bottom-line outcomes (hard measures). A sys- tem model can be used to provide meaningful ongoing feedback at all levels, to diagnose departmental or organization-wide performance issues, and to iden- tify appropriate interventions that will positively impact performance. We will now review the six key measurement dimensions mentioned earlier that should be considered when taking a systems approach to evaluation. EVALUATION 333 18_962384 ch12.qxd 2/3/05 12:18 AM Page 333 334 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION Exhibit 12.1. Advantages and Disadvantages to Possible Evaluators Advantages Disadvantages Client Organization • Not an additional cost to the organization • Will be present to the organization long after the specific OD process is over • Knows the culture of the organization better than an outside party can • Knows what is important to the organization and, thus, what needs to be evaluated • Because of responsibility for participating in planning the OD process, can build evaluation into pre-launch phase • May be too closely associated with the organization to see what is necessary in the evaluation • May fear that the outcome of the evaluation will cause them to lose credibility if they were responsible for the contract • May not have time to do evaluation effectively because of other projects • May not have the necessary competencies to carry out the evaluation Consultant • Knows the OD process and its intended outcomes as well • Should have the necessary competencies to carry out the evaluation • Because of the responsibility for partici- pating in planning the OD process, can build evaluation into the pre-launch phase • Gives the consultant the opportunity to modify processes used or provide adequate compensation for mistakes made • Adds to the project‘s costs • Unless a long-term involvement is envisioned, the consultant may not be around to complete the evaluation • May not be viewed as objective as there is an incentive to make the evaluation look good (for affirmation of client‘s choice) or for additional needs to surface (for additional work) • May create a sense of client‘s dependence on the consultant Third Party • Viewed as more objective because not involved in any other way in the OD process • Usually more expert because of the ability to specialize in evaluation • Has nothing to lose or gain in the outcome of the evaluation • Clearly, an additional cost • A third party may not want to take on such a limited component of a project • Is not integral to the project, so may not know either the process or the outcomes to be assessed • Is probably not in a position to influence improvements in the process or in im- proving the ODs consultants processes • Guarantees that the evaluation will not be formative Measurement Dimension 1—Profitability and the Bottom Line When discussing the bottom line, many people think of profitability first. In this chapter, profitability is part of a broader definition of performance that is based on a balanced approach to assessing the bottom line. There are three perfor- mance measures, the first of which is profitability. Profit is one measure of per- formance that has dominated throughout the history of economic ventures, and it remains a central focus today. However, there is growing recognition that 18_962384 ch12.qxd 2/3/05 12:18 AM Page 334 profit is not the only indicator of the real bottom line (for example, Kaplan & Norton, 1996). For non-profit organizations, consider cash flow as a measure that replaces profitability for a for-profit organization. There are other measures, along with profit, that provide a more balanced approach to assessing the bottom line. That is why the word performance is used in place of bottom line. In addition to profit, there are internal and exter- nal performance measures, such as customer retention, organizational learning, new products developed, and many more. Measurement Dimension 2—Morale Measures of morale are soft by nature. That is, asking the employees or cus- tomers their opinions regarding a variety of issues assesses each measure. Mea- sures of morale address attitudes and intentions from both the employee and customer perspective. An attitude is a bi-polar affect regarding some target (for example, I like it or I don’t like it). Attitudes focus on such things as job satis- faction for employees and product/service satisfaction for customers. Intentions are cognitive choices made with regard to pending action or lack of action (for example, I will do this or I will do that or I will do nothing). Intentions focus on such things as intention-to-leave a job or intention-to-return to buy a product/ service again. As mentioned earlier, these measures tend to focus on a degree of agreement or disagreement on a Likert scale. Measurement Dimension 3—Emotion Emotions are similar yet distinct from morale. Emotions encompass glad, sad, mad, afraid, and ashamed. There are also the emotions of excited, peace, and numb. These emotions tend to be related to discrete events and are reactionary in nature. The reason that they are considered similar to morale is that they are known to influence morale directly. For example, a failed performance by an employee (for example, a lost sale) can lead to anger and guilt. These emotions can subsequently lead to an exaggerated low level of job satisfaction and a heightened level of intention-to-leave. Over time, these levels of satisfaction and intention-to-leave can return to the original levels prior to the failed perfor- mance. That is, emotions generally have a temporary influence on morale, not a permanent one. Measurement Dimension 4— Critical Customer Service Behaviors Customers have certain expectations when receiving products and services. There are all sorts of support required and questions to be asked. In the process, the customer can define a set of employees’ behaviors that add value to the product or service. These behaviors are applicable across industries, yet must EVALUATION 335 18_962384 ch12.qxd 2/3/05 12:18 AM Page 335 be customized to fit a particular industry (service or product). There are five critical employee behaviors that can lead to higher levels of customer satisfac- tion and retention: response time, personalized attention, professionalism, expertise, and comforting. Measurement Dimension 5—Organization Climate There is a considerable amount of research and numbers of survey measures that are directed toward a better understanding of those factors that directly influence task performance. These task-dependent measures fall into three variables: job climate factors, teaming behaviors, and leadership behaviors. Job climate factors are directly related to the work being done and include envi- ronmental support, personal capabilities, and reward systems. Teaming behav- iors are the critical behaviors team members should exhibit that lead to a high-performing team. Teaming behavior measures include the following: antic- ipates and fulfills team needs; manages own personal contribution; displays a positive attitude; and effectively communicates. Leadership behaviors are those critical behaviors that leaders should exhibit to create a high-performing team. Leadership behaviors include the following: supporting, informing, monitoring, recognizing and rewarding, empowering, and clarifying. Measurement Dimension 6—Personal Characteristics There are two types of personal characteristics, each of which contains differ- ent measures. First, there are biographical characteristics that are considered hard measures and are usually found in most personnel files. Biographical char- acteristics include such things as age, race, gender, tenure, and function. The second type of personal characteristic is personality. Personality refers to more stable behavioral traits that people exhibit. For example, there are a variety of personality models, such as the Myers-Briggs Type Indicator (MBTI), the Big Five (extraversion, emotional stability, agreeableness, conscientiousness, and openness to experience), and other non-dimensional measures, such as locus- of-control and pessimism. While these tend to be very popular in use, their validity is suspect. TYPES OF EVALUATION There are many ways to perform an appropriate evaluation. In planning an OD evaluation, it is important that an agreement be reached in the beginning about what is important and how it will be determined, whether the outcome has 336 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION 18_962384 ch12.qxd 2/3/05 12:18 AM Page 336 been achieved or not. As described earlier in this chapter, it is important to decide who will conduct the evaluation. Outcome measures must be identified. A time frame—formative, summative, and longitudinal—must be determined. And then the appropriate tool or approach must be chosen. Formal/Informal A formal approach to evaluation will be decided among the parties involved in the OD process. Informally, however, all of the parties involved will be con- stantly looking at ways to improve their roles in the OD process. Sometimes this may be apparent, as when a consultant might simply ask a client contact how things are going. Often, however, the informal evaluation process will not be apparent. It may consist of continually observing how people are responding and participating, checking on available statistics that might reflect on the desired outcome measures, and even watching stock prices. The concern with informal evaluation is that the results are not shared and are not systematic. As a result, partnering between consultant and client loses synergy, and results that surface are biased by happenstance observations rather than systematic ones. It also makes great sense in summative and longitudinal evaluation to repeat whatever was done in the assessment stage. Since that process was successful in identifying the focus for the OD process undertaken, it makes sense to use the same approach in evaluating the outcomes of that process. Thus, the recommendation of the authors is that a shared, systematic per- spective be selected from among the many formal modes of evaluation that are available. There are many ways in which such approaches could be classified. We have chosen to classify them as quantitative (also called hard) and qualita- tive (also called soft). These will be detailed further in this section. Quantitative A preference still exists in business to have quantifiable data, and many of the traditional approaches to evaluation in OD rely on quantifiable data. These data come from a variety of sources—surveys (both simple and complex), measur- able outcomes (such as discussed earlier in this chapter—both in terms of meet- ing pre-specified goals and pre-process and post-process comparisons), and benchmarking (especially with competitors) are the primary approaches used. Surveys These can be as simple as a basic satisfaction survey (often referred to as a happy sheet) to a somewhat more complicated feedback form (for example, an assessment of how meetings are going) to multi-rater feedback to a very com- plex culture survey of the organization. EVALUATION 337 18_962384 ch12.qxd 2/3/05 12:18 AM Page 337 The basic satisfaction survey is probably the easiest to construct and use and, thus, is frequently used. Items to be measured might include participants’ satis- faction with the OD process (activities, materials, consultant, facilities, and so on). Sample items are presented in Exhibit 12.2. A somewhat more complicated survey, although based on the same con- cepts as the satisfaction survey, is one that focuses on a specific function within the organization—for example, how well executive team meetings are going, level of customer satisfaction, product innovation, and so on. See Exhibit 12.3 for an excerpt from a survey frequently used by one of the authors. Multi-rater assessment is another survey form that is often used when the OD focus is on an individual or a small group of individuals. This is often referred to as 360-degree feedback. When completed in a survey form, it pro- vides quantitative data. There are many issues with this type of approach, espe- cially when used inappropriately to determine salary and promotion potential (see McLean, 1997). A prime concern with the use of such an instrument is that research shows that different groups of reviewers provide different feed- back, naturally, as people are observed by different groups of people in differ- ent contexts. Finally, the most complex form of survey is the climate (or culture) survey. This is usually a comprehensive review of the factors in the organization that affect the climate (or culture). A controversial issue that frequently surfaces with surveys is whether to use a customized survey (developed by the consultant or a team in the client organization) or a commercial, standardized survey. See Exhibit 12.4 for a comparison of the two approaches. 338 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION Exhibit 12.2. Sample Items from an OD Process Satisfaction Survey Evaluation of Team Building OD Process In responding to the following questions, circle the number that corresponds with the following scale: 5 = Very Satisfied; 4 = Satisfied; 3 = Neutral; 2 = Dissatisfied; 1 = Very Dissatisfied 1. How well the need for team building 54321 was originally identified 2. How appropriate the retreat 54321 facilities were 3. How effective the team-building 54321 activities at the retreat were 18_962384 ch12.qxd 2/3/05 12:18 AM Page 338 EVALUATION 339 Exhibit 12.3. Survey to Determine Satisfaction with a Team Meeting Process Team Meeting Assessment Circle the number to the right of each item that matches your assessment of how well the team is currently functioning. Use the following scale: 5 = Strongly Agree; 4 = Agree; 3 = Neutral; 2 = Disagree; 1 = Strongly Disagree Planning 1. The right people are on the team. 54321 2. An agenda is distributed before 54321 meetings. 3. Supporting papers are distributed 54321 before meetings. 4. The agenda items are timed. 54321 5. The agenda contains expected 54321 outcomes. 6. Visual aids are available, including flip 54321 charts, markers, tape, sticky notes, etc. 7. Room arrangement maximizes 54321 interaction. 8. Participants bring distributed 54321 materials to meetings. 9. Presenters are prepared when 54321 scheduled. 10. Agendas adequately reflect meetings. 54321 Doing 11. Meetings start on time. 54321 12. Everyone is present on time. 54321 13. Meetings focus on improvement 54321 rather than providing information. 14. Minutes are taken at all meetings. 54321 15. Telephone calls do not interrupt 54321 meetings. 16. Discussion is focused on outcomes; 54321 it does not ramble. 17. Each team member fully participates 54321 when competent to do so. 18. Team members do not dominate 54321 discussion. 19. Each agenda item ends with a 54321 clearly developed action step. 20. Blaming does not occur among 54321 team members. Practicing Organization Development, 2nd Ed Copyright © 2005 by John Wiley & Sons, Inc. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com 18_962384 ch12.qxd 2/3/05 12:18 AM Page 339 340 PRACTICING ORGANIZATION DEVELOPMENT, 2ND EDITION Exhibit 12.3. Continued Doing 21. Team members do not interrupt 54321 each other. 22. I feel free to express my opinions 54321 without fear of reprisals. 23. Decisions are made on a 54321 consensus basis. 24. All team members accept responsibility 54321 for the quality of team meetings, providing feedback during the meeting on ways to improve meetings. 25. Risk taking during meetings is 54321 encouraged. 26. Meetings end on time. 54321 27. Meetings end early if the agenda 54321 is completed. 28. The group leader uses good 54321 facilitation skills. 29. Conflict is encouraged. 54321 30. Conflict is handled in a healthy way, 54321 focusing on facts and “I” statements. 31. Team members listen to each other. 54321 32. Team members listen to new ideas 54321 without responding with “Yes, but.” 33. Team members build on one 54321 another’s new ideas. 34. The agenda consists of items at an 54321 appropriate level, delegating detail to appropriate employees. 35. Team members trust each other. 54321 36. Team members respect each other. 54321 37. Agenda items for the next meeting 54321 are identified before the end of each meeting. 38. Appropriate problem-solving tools 54321 are used during meetings. 39. Sufficient resources are provided 54321 for the team’s action steps to be carried out. 40. The next meeting date is set 54321 prior to adjournment. Practicing Organization Development, 2nd Ed Copyright © 2005 by John Wiley & Sons, Inc. Reproduced by permission of Pfeiffer, an Imprint of Wiley. www.pfeiffer.com 18_962384 ch12.qxd 2/3/05 12:18 AM Page 340 . of the real bottom line (for example, Kaplan & Norton, 1996). For non-profit organizations, consider cash flow as a measure that replaces profitability for a for- profit organization. There are. important bottom-line outcomes (hard measures). A sys- tem model can be used to provide meaningful ongoing feedback at all levels, to diagnose departmental or organization- wide performance issues,. a heightened level of intention-to-leave. Over time, these levels of satisfaction and intention-to-leave can return to the original levels prior to the failed perfor- mance. That is, emotions generally