CSFs measure the end result, usually as seen through the eyes of both the customer and the contractor. KPIs and metrics generally measure the quality of the processes used to achieve the end results and accomplish the CSFs. KPIs and metrics are internal measures and can be reviewed on a periodic basis throughout the life cycle of a project. Some people believe that CSFs are the same as metrics and KPIs and in the confusion try to track them. CSFs are usu- ally broad categories and difficult to track, whereas metrics and KPIs are more specific and are, therefore, more appropriate for measurement and then report- ing through means such as dashboards. CSFs are often interim steps between the definition of success and the establishment of metrics.
Metric measurements can be too costly. Even if all of the stakeholders are in agreement on the CSFs, the cost of measuring the metrics to support the CSFs can be prohibitive and the benefits achieved may not support the cost.
Unless the measurements pay for themselves in supporting the CSFs, overly expensive or useless measurements should not be made. Stakeholders must believe that the correct metrics were chosen and that the measurements accu- rately portray the true status. It is important to understand that some metrics for success cannot instantaneously determine the success of a project. True mea- surements may not be able to be made until well after the project is completed.
Projects often fail because the project manager and the stakeholders cannot agree on the CSFs and then may end up selecting useless metrics that cannot provide meaningful data. It is not uncommon for the stake- holders to believe that having the fewest scope changes is a CSF, whereas the project manager believes that following the change control process rig- idly is the true CSF regardless of the number of scope changes. Some causes of failure include:
◾ Improper understanding of the meaning of the CSFs.
◾ Failing to believe in the value of the CSFs.
◾ Each stakeholder is working toward his/her own definitions of the CSFs.
◾ Stakeholders refuse to come to an agreement on the correct CSFs for the project at hand. This can occur during the project as well as at the onset.
◾ Failure to understand the gap between the actual performance and the CSFs.
◾ Believing that the measurement costs for the metrics to support the CSFs are too costly.
◾ Believing that it is a waste of productive time measuring the metrics needed to support the CSFs.
3.10 METRICS AND THE PMO
Previously we stated that you cannot effectively manage a project without having measurement and metrics capable of providing you with complete information. Project managers do not always possess expertise in selecting
the correct metrics, KPIs, and CSFs. This is where the PMO can be of assis- tance. With regard to metrics, the PMO can:
◾ Assist each project team with the development of project-based metrics
◾ Recognize that some metrics may be project or client specific
◾ Maintain a metrics library
◾ Develop a metric/KPI template for the library
◾ Recognize that metric and KPI improvements must evolve over time and that updating is necessary
As part of the PMO’s responsibility as the guardian of all project manage- ment intellectual property, the PMO will coordinate the efforts for the updat- ing of the metrics. A periodic review of the effectiveness of each metric/KPI is essential because metrics have life cycles. As stated previously, the metrics can remain as is, be updated, or be retired from service. Metrics can become irrel- evant without supporting data. Overseeing metrics and KPIs can be expensive and difficult. Creating unnecessary metrics should be avoided.
The PMO may have the responsibility of debriefing the project teams in order to capture lessons learned and best practices. A typical debriefing pyramid is shown in Figure 3-6. The PMO will look at metrics related to the project, use of organizational process assets such the EPM systems, the business units, and perhaps even corporate strategic metrics.
Figure 3-6 The Postmortem Pyramid
Performance Analysis
Time Cost Quality Scope
Business Metrics
Customer
Project Management
Metrics
Success Metrics Business Opportunity Mission
Business Unit Metrics
Metrics for Organizational
Process Assets Project- Specific Metrics Corporate
Objectives
Project Objectives Corporate Success Metrics
87
3.10 METRICS AND THE PMO
At the same time, the PMO may evaluate all of the metrics used to see if the metrics should be part of the metrics library and if the cost of using the metric (i.e., measurement) was worth the effort. Figure 3-7 show a typical way that the metrics may be evaluated. Below the risk boundary or the par- ity line parity line, the value exceeds the cost and the risks of using the met- ric are acceptable. Above the parity line, the measurement cost may exceed the value of using the metric and careful risk analysis is necessary.
If the metrics are deemed to be of value, then the metrics can be classi- fied in the metrics library the same way that we often classify best practices.
A typical example is shown in Figure 3-8. For companies that are reason- able immature in project management, the focus is on promoting the use of metrics but perhaps only a few. Companies that are reasonably mature in project management build and maintain metric and KPI libraries.
In determining the best possible metrics for a project, the PMO may find it necessary to perform metrics benchmarking. Two critical factors must be considered:
◾ The project management maturity level of your organization as well as the company against which you are benchmarking
◾ The project management maturity level of the stakeholders There are also misconceptions that must be considered such as:
◾ Metrics that work well for one company may not work well for another company if the use of the metrics is based upon in-house practices.
◾ Identifying the metrics is easy. Using them is difficult.
◾ Some metrics may be more a rough guide than a precise benchmark.
Figure 3-7 Metric Cost versus Value
High
Unacceptable Risk
Cost of Measurement
Low
Acceptable Risk
High Low
Value of Metric Usage
Figure 3-8 Best Practices Classification
Effectiveness and Efficiency
Satisfaction Future
Business
CONTINUOUS IMPROVEMENTS PROJ.
MANAGEMENT TECHNOLOGY
Technology Advances
Project Management USING ST
TE-OF-THE-AR A T
TECHNOLOGY
ENGAGEMENTS
INTERF ACING METHODS FOR SUCCESSFULL
Y WINNING NEW BUSINESS COMP
ANY
-WIDE BUSINESS
PERFORMANCE IMPOR TANT
INITIA TIVES
BUSINESS APPLICA
TIONS
CUSTOMER INTERF
ACING, REPOR
T PREP
ARA TION,
AND ST
ANDARD PRACTICES
BEST PRACTICES
LIBRARY
Since most PMOs are overhead rather than direct labors charges, it may be necessary for the PMO to establish their own metrics to show the PMO’s contribution to the success of the company. Typical ROI metrics that the PMO use include:
◾ Percent of projects using/following the EPM system/framework
◾ Ratio of project manager to total project staff
◾ Customer satisfaction ratings
◾ Year-over-year throughput
◾ Percent of projects at risk or in trouble
◾ Number of projects per headcount (staffing tolerance for projects)
◾ Ways to improve faster closure
◾ Percent of scope changes per project
◾ Percentage of projects completed on time
◾ Percentage of projects completed within budget
It is important to understand that metrics management is an essential component of knowledge management and involvement by the PMO is essential. It is very difficult to improve processes and work flow without gathering metrics and storing the results for traceability.
89