Developing Objectives for Projects 75 Application objectives are critical because they describe the expected outcomes in the intermediate area—between the learning of new tasks and procedures and the delivery of the impact of this learning. Appli- cation and implementation objectives describe how things should be or the desired state of the workplace once the project solution has been implemented. They provide a basis for evaluating on-the-job changes and performance. Impact Objectives Every project should drive one or more business impact measures. Impact objectives indicate key business measures that should improve as the application and implementation objectives are achieved. The following are typical impact objectives. When the project is completed . . . • System downtime should be reduced from three hours per month to no more than two hours per month in six months. • Incidents should decrease by 20 percent within the next calendar year. • The average number of product defects should decrease from 214 to 153 per month in the Midwest region. • The customer satisfaction index should rise by 2 percent during the next calendar year. • Sales expenses for all titles at Proof Publishing Company should decrease by 10 percent in the fourth quarter. • The average number of new accounts opened at Great Western Bank should increase from 300 to 350 per month in six months. • The shelter costs per homeless person should be below $70,000 in two years. • There should be an across-the-board reduction in overtime for front- of-house managers at Tasty Time restaurants in the third quarter of this year. • Employee complaints should be reduced from an average of three per month to an average of one per month at Guarantee Insurance headquarters. • The companywide employee engagement index should rise by one point during the next calendar year. 76 ACHIEVING BUSINESS ALIGNMENT WITH THE PROJECT • There should be a 10 percent increase in Pharmaceuticals Inc. brand awareness among physicians during the next two years. • The dropout rate for high school students in the Barett County system should decrease by 5 percent within three years. Impact objectives are critical to measuring business performance because they define the ultimate expected outcome from the project. They describe the business unit performance that should result from the project. Above all, impact objectives emphasize achievement of the bottom-line results that key client groups expect and demand. ROI Objectives The fifth level of objectives for projects represents the acceptable return on investment, the monetary impact. Objectives at this level define the expected payoff from investing in the project. An ROI objective is typically expressed as an acceptable ROI percentage, which is expressed as annual monetary benefits minus cost, divided by the actual cost, and multiplied by one hundred. A 0 percent ROI indicates a breakeven project. A 50 percent ROI indicates recapture of the project cost and an additional 50 percent ‘‘earnings’’ (50 cents for every dollar invested). For some projects, such as the purchase of a new company, a new building, or major equipment, the ROI objective is large relative to the ROI of other expenditures. However, the calculation is the same for both. For many organizations, the ROI objective for a project is set slightly higher than the ROI expected from other ‘‘routine investments’’ because of the relative newness of applying the ROI concept to the types of projects described in this book. For example, if the expected ROI from the purchase of a new company is 20 percent, the ROI from a new advertising project might be around 25 percent. The important point is that the ROI objective should be established up front and in discussions with the project sponsor. Excluding the ROI objective leaves stakeholders questioning the economic success of a project. If a project reaps a 25 percent ROI, is that successful? Not if the objective was a 50 percent ROI. FINAL THOUGHTS This chapter outlines the beginning point in the ROI methodology, show- ing how a project can be structured from the outset, with detailed needs Final Thoughts 77 identified, ultimately leading to project objectives at five levels. This level of detail ensures that the project is aligned with business needs and remains results-focused throughout the process. Without this analysis, the project runs the risk of failing to deliver the value that it should, or not being in alignment with one or more business objectives. The outputs of the analysis are objectives, which provide a focus for project designers, developers, and implementers, as well as participants and users who must make the project successful. Issues surrounding data collection are discussed in the next three chapters. Collecting reaction data and perceived value data will be covered first. Chapter 5 Measuring Reaction and Learning When the initial analysis is completed and the project is positioned for success, implementation occurs. During implementation, feedback is collected from participants involved in the project. Their reactions and learning with regard to the project provide indications of its potential for success. This chapter focuses on the measurement of reaction and learning. Collecting these data at the beginning of the project corresponds to the first operational phase of the ROI methodology. Participant feedback can furnish powerful information to use in making adjustments and measur- ing success. Measuring learning is an important part of the evaluation process, especially when a project is intended to change behaviors or processes on the job. Participant knowledge of what to do and how to do it is critical to a project’s success. This chapter outlines the most common approaches to collecting data and explores ways to use the information for maximum value. THEIMPORTANCEOFREACTION It is difficult to imagine a project being conducted without the collection of feedback from those involved in the project, or at least from the sponsor. Collecting reaction and perceived value data serves several purposes. Participant feedback is critical to understanding how well a project serves the customer and the potential of the project to meet the identified business needs. 79 Project Management ROI: A Step-by-Step Guide for Measuring the Impact and ROI for Projects Jack J. Phillips, Wayne Brantley, and Patricia Pulliam Phillips Copyright © 2012 John Wiley & Sons, Inc. 80 MEASURING REACTION AND LEARNING Customer Satisfaction Reaction and perceived value are customer satisfaction measures for the project. Without sustained, favorable reactions, the project may not suc- ceed. The individuals who have a direct role in the project are immediately affected by it and often have to change processes or procedures or make other job adjustments in response to the project’s initiation. Participant feedback on preferences is critical to making adjustments and changes in the project as it unfolds. The feedback of project supporters is also important because this group will be in a position to influence the project’s continuation and development. The sponsors, who approve budgets, allocate resources, and ultimately live with the project’s success or failure, must be completely satisfied with the project—and their overall satisfaction must be verified early and often. Immediate Adjustments A project can quickly go astray. A project can be mismatched to the solution from the beginning, so getting feedback early in the process is necessary to allow for immediate adjustments. This can help prevent misunderstandings, miscommunications, and, more important, misap- propriations. Collecting and using reaction data promptly can enable an improperly designed project to be altered before more serious problems arise. Predictive Capability Reaction data can be used to predict the success of a project using analytical techniques. Project participants are asked to estimate the effectiveness of the project’s application and, in some cases, the resulting value of that application. The reaction data thus become a forecast (forecasting is described in detail in Chapter 12). Figure 5.1 shows the correlation between reactive feedback and application data. Countless studies have been conducted to verify this correlation. In this analysis, the reaction measures are taken as the project is introduced, and the success of the implementation is later judged using The Importance of Learning 81 Reaction Application Figure 5.1 Correlations between reaction and application. the same scale, e.g., a 1 to 5 rating. When significant positive correlations are present, reaction measures can have predictive capability. Some reaction measures shown to have predictive capability are • The project is relevant to my work. • The project is necessary. • The project is important to my success. • The project is important to the success of this organization. • I intend to make this project successful. • I would recommend this project. Measures such as these consistently lead to strong positive correlations and consequently represent more powerful feedback than typical mea- sures of satisfaction with the project. Some organizations collect these or similar reaction measures for every project or program initiated. THEIMPORTANCEOFLEARNING Several key principles illustrate the importance of measuring learning during the course of a project. Each of these in itself is sufficient to justify the measurement of learning; collectively, they provide an indication of the full range of benefits that result from measuring the changes in skills, knowledge, and other qualities that occur during a project. 82 MEASURING REACTION AND LEARNING The Importance of Intellectual Capital Intellectual capital has become an important concept as organizations have progressed from agricultural to industrial to knowledge-based sys- tems. Intellectual capital is what the organization knows, and it can be classified in a variety of ways for measurement purposes. Figure 5.2 illustrates one categorization of intellectual capital, showing intellectual capital as a combination of human capital, renewable capital, structural capital, and relationship capital. 1 As projects are implemented, they focus on increasing one or more of these major elements of intellectual capital. For some organizations, intellectual capital translates directly into suc- cess, in the form of rewards by the stock market. Up to 80 percent of the market value of some high-technology firms is attributed to intellectual capital. This demonstrates the value of measuring learning in projects aimed at improving intellectual capital. The Learning Organization In the past two decades, organizations have experienced a rapid transformation of competitive global markets as a result of economic changes. Organizations must learn new ways to serve customers and use Human capital Renewable capital Structural capital Intellectual capital Relationship capital Individual capital: Personal expertise and experience, and the ability to transform it into new, shared knowledge New knowledge: Created, transferred, and communicated by many people Intellectual properties: Patents, licenses, etc. Marketable innovations: Products, services, and technology Work processes: Institutionalized knowledge in the form of procedures, policies, process technologies, etc. Documentation: Databases, records, and knowledge documents of various forms Networks: Resources for information and influence Customers: Particularly those most innovative in their industries Figure 5.2 Categories of intellectual capital. The Importance of Learning 83 innovations and technology to enhance their efficiency, to restructure, reorganize, and execute their functions globally. In response to this need for a change in strategy, the concept of the learning organization evolved. This concept requires organizations to use learning proactively in an integrated way and to support growth for individuals, teams, and entire organizations. With the new focus on creating learning organizations—where countless activities and processes are in place to promote continuous learning—measurement has become an even more important issue. The Compliance Issue Organizations face an increasing number of regulations with which they must routinely comply. These regulations involve all aspects of business and are considered essential by governing bodies to protect customers, investors, and the environment. Employees must have a certain amount of knowledge about the regulations to maintain compliance. Consequently, an organization must measure t he extent of employee learning and understanding with regard to regulations to ensure that compliance is not a problem. Some projects are compliance driven. For example, one large banking organization had to implement a major project to ensure that its employees were all familiar with money laundering regulations. This project was precipitated by the bank’s’ continuing failure to comply with the regulations. The problem appeared to be a lack of knowledge of the rules. When projects such as this are initiated, learning must be measured. The Use and Development of Competencies A competency model describes a particular combination of knowledge, skills, and characteristics necessary to perform a role in an organiza- tion. The use of competencies and competency models has dramatically increased in recent years. In the struggle for a competitive advantage, many organizations have focused on people as the key to success. Com- petency models are used to ensure that employees do the right things, clarifying and articulating what is required for effective performance. Competency models help organizations align behavior and skills with the strategic direction of the company. 84 MEASURING REACTION AND LEARNING The Role of Learning in Projects When projects involve new equipment, processes, and technology, the human factor is critical to project success. Whether an organization is restructuring or adding new systems, employees must learn how to work in the new environment, and this requires the development of new knowledge and skills. Simple tasks and procedures do not automatically come with new processes. Instead, complex environments, procedures, and tools m ust be used in an intelligent way to reap the desired benefits for the organization. Employees must learn in different ways—not just in a formal classroom environment, but through technology-based learning and on-the-job practice. Project leaders and managers serve as coaches or mentors in some projects. In a few cases, learning coaches or on-the-job trainers are used with projects to ensure that learning is transferred to the job and is implemented as planned. Project team members don’t always fully understand what they must do. Although the chain of impact can be broken at any level, a common place for such a break is at Level 2, learning and confidence. Employees simply may not know what to do or how to do it properly. When the application and implementation does not go smoothly, project managers can determine if a learning deficiency is the problem, and if so, they may be able to eliminate it. In other words, learning measurement is necessary to contribute to leaders’ understanding of why employees are, or are not, performing the way they should. SOURCES OF DATA Possible sources of reaction and learning data concerning the success of a project can be grouped into distinct categories. Project Participants The most widely used data source for project evaluation is the partici- pants, those directly involved in the project. These ‘‘users’’ must take the skills and knowledge they acquired in the project or process and apply them on the job. They also may be asked to explain the potential impact of that application. Participants are a rich source of data for almost every aspect of a project. They are by far the most credible source of reaction and perceived value data. Sources of Data 85 Participant Managers Another key source of data is the individuals who directly lead the par- ticipants. They have a vested interest in the project and are often in a position to observe the participants as they attempt to apply the knowl- edge and skills acquired in the project. Consequently, they can report on the successes associated with the project as well as the difficulties and problems. Although managerial input is usually most valuable as a source of reaction and perceived value data, it is also useful for other levels of evaluation. Other Employees When entire teams are involved in the implementation of the project, all employees can provide useful information about the perceived changes prompted by the project. Input from this group is pertinent only to issues directly related to their work. Although data from this group can be helpful and instructive, it is sometimes not elicited because of the potential for introducing inaccuracies to the feedback process. Data collection should be restricted to those team members capable of providing meaningful insight into the value of the project. Internal or External Customers The individuals who serve as internal customers of the project are another source of reaction, perceived value, and other types of data. In some situations, internal or external customers provide input on perceived changes linked to the project. This source of data is more appropriate for projects directly affecting the customers. They report on how the project has influenced (or will influence) their work or the service they receive. Although this group may be somewhat limited in their knowledge of the scope of a project, their perceptions can be a source of valuable data that may indicate a direction for change in the project. Project Leaders and Team Members The project leader and project team may also provide input on the success of the project. This input may be based on on-the-job observations during the course of the project and after its completion. Data from this source [...]... Rewarding Project Leadership Team Climate Project Execution Just for Me Necessary New Information Efficient Recommend to Others Facilities/Environment Easy/Difficult Appropriate Practical Topics for Learning Measures 87 Project participants must be assured that the project has the necessary commitment Issues important to project management and the organization sponsoring the project include project leadership,... support the project The culture in our work group does not support the project We have no opportunity to use the project skills, knowledge, or information We have no time to implement the project Technology was not available for the project Our systems and processes did not support the project Resources are not available to implement the project My job changed and this no longer applies The project is... satisfaction with, the project More important, these data provide evidence relating to the potential success of the project Reaction data should be used to: • • • • • • Identify the strengths and weaknesses of the project and make adjustments Evaluate project team members Evaluate the quality and content of planned improvements Develop norms and standards Link with follow-up data Market future projects based... which a project is implemented provides critical data about its success, and about factors that can contribute to greater success as the project is fully implemented The Level 1 and Level 2 measures occur during a project s early stages, when more attention and direct focus are placed on the participants’ direct involvement in the project Measuring application and implementation occurs after the project. .. application and implementation data is the follow-up session A follow-up session is an intended regrouping of the project team to assess the project status while continuing to move the project forward Follow-up sessions provide the project team with a forum for discussing the effectiveness of the project, and they are also ideal for discussing application barriers and developing strategies to overcome them... level checks the progress of the project against the objectives Objectives are critical to a project, and they should be established at each level Fundamentally, the measurement of learning reveals the extent of knowledge and skill acquisition in relation to the project This is a critical element, particularly for knowledge-based projects, new technology applications, and projects designed to build competencies... necessary For some projects, it is the most critical data set because it provides an understanding of the degree to which successful project implementation occurs, and of the barriers and enablers that influence success Value of Information As briefly discussed in Chapter 3, the value of information increases as progress is made through the chain of impact, from reaction (Level 1) to Project Management ROI:... reengineering project As part of the project, participants were given the assignment of designing and testing a basic system A project team manager observed participants as they checked out the system; then the manager carefully completed the same design and compared his results with those of the participants These comparisons and the performance of the designs provided an evaluation of the project and... of moving the project forward through participants’ on-the-job use of knowledge about the project Essentially, this measure reflects the degree to which the project is implemented by those who are charged with its success This is the first step in transitioning to a new state, behavior, or process Understanding the success of the transition requires measuring application and implementation Project Focus... a task or project by a set date A summary of the results of the completed assignment provides further evidence of the project s success The action plan is the most common type of follow-up assignment With this approach, participants are required to develop action plans as part of the project Action plans contain the detailed steps necessary to accomplish specific objectives related to the project The . well a project serves the customer and the potential of the project to meet the identified business needs. 79 Project Management ROI: A Step-by-Step Guide for Measuring the Impact and ROI for Projects. Measures 87 Project participants must be assured that the project has the nec- essary commitment. Issues important to project management and the organization sponsoring the project include project. percent ROI indicates a breakeven project. A 50 percent ROI indicates recapture of the project cost and an additional 50 percent ‘‘earnings’’ (50 cents for every dollar invested). For some projects,