Metrics That Matter

With many IT leaders forced to justify their group’s economic value, it’s more critical than ever to present executive managers with the right measurements for IT’s contributions. But most IT organizations fall short of this objective.

Although the economy has recovered somewhat from the triple whammy of the 9/11 attack, corporate scandals, and dot-com bust of the early 2000s, many IT organizations have not seen a return to the go-go years of the 1990s in which technology investments were easy to come by—thanks to a basic corporate belief in the enabling power of IT. Today’s executives no longer take it as an article of faith that technology investments bring real value and that IT groups are both effective and efficient; they now demand proof. From many IT leaders’ viewpoints, this has meant not only stagnant budgets, but increasing demands for both new initiatives meant to provide competitive advantage and efforts to satisfy new post-scandal regulatory requirements such as Sarbanes-Oxley. Worse for some, IT budgets have been treated as expensive cost centers that need to be continually scrutinized based on some arbitrary formula, such as basing IT spending on a fixed percentage of revenues.

Certainly, there are inefficient IT departments that spend money on dubious projects. But there are also many IT departments that are effective and contribute real value to the enterprise—efficiency and value that can be imperiled by a bean counter’s approach to funding. But to demonstrate that efficiency and value, both IT and executive management need to rethink how they measure IT value. Far too often, consultants say, IT leaders drown executives in operational data such as help-desk resolution times and network uptime—data that is meaningless to the corporate strategy and cements IT’s reputation as being little more than a janitorial service for technical systems.

“Many CIOs report these to the business side, and all they get are blank stares,” said Mark Lutchen, a partner at the consultancy PricewaterhouseCoopers.

CIOs and other IT leaders need to rethink how they measure and demonstrate their value to the enterprise by using metrics that matter. Although few companies actually do this, consultants agree on a basic framework that IT can take to deliver what executives need. The specifics will vary from company to company, but the principles are consistent. The most fundamental principle: “Metrics give you the opportunity to change behavior,” said Lutchen. So they should be chosen in concert with the most critical desired behaviors to be maintained and changed.

Along these lines, “make sure the metrics have a business component to them,” said Scott Holland, a senior business advisor at The Hackett Group.

Alan Houghton, a principal consultant at PA Consulting Group, also suggested, “Think of what you do as a service. That creates a mind-set of what’s important.”

MEASURE EFFECTIVENESS AND EFFICIENCY
It is easier to measure efficiency than effectiveness because efficiency can be based on hard data such as dollars spent, time taken, and functions delivered. Effectiveness, on the other hand, involves some subjective interpretation (such as customer satisfaction) in addition to specific data points (such as staff reductions achieved or extra income generated). But you should measure both attributes nonetheless.

Efficiency is important, but it’s essentially a defensive metric, meant to show you are not wasting resources and thus justify operational investments. “CIOs need to take the first step and clean their own houses first,” said PricewaterhouseCoopers’ Lutchen, before expecting to get extra money for value-creating initiatives.

But if they demonstrate only efficiency, they play into the bean-counter mentality that all that matters is extracting more efficiency from the system. That’s an easy road to continued cuts, said consultant Richard Chang, CEO of Richard Chang Associates. At many companies, “this cost focus has led to the sub-optimization of IT,” Chang said. It can even lead to the eventual outsourcing of IT. Demonstrating effective value creation is critical to getting more funding and keeping a seat at the executive table in which strategic efforts are the focus. In such a case, even if the more generic IT operations are squeezed or outsourced, IT remains as a value creator for the company.

Understanding this issue, “some organizations separate the operations costs from the strategic costs,” said Bernard “Bud” Mathaisel, CIO of the IT outsourcer Achievo and former CIO of Ford Motor and Walt Disney & Co. This helps separate efficiency concerns (operations costs) from effectiveness concerns (strategic costs) and makes it easier to map the corporate strategy to the appropriate IT metrics.

THREE BASKETS OF METRICS
IT should measure three types of attributes in what is essentially a modified form of the Balanced Scorecard approach to measuring performance and change management. Those three attributes are: strategic value, project management effectiveness, and operational effectiveness. Ironically, while the first two matter the most to executives in most cases, IT typically focuses on the third area, which executives only care about if the IT department has a history of failure and thus needs to be closely monitored on the basics. An effective IT department would only report operational metrics to the executive team if there’s a significant failure (along with an explanation derived from real data) or, on an occasional business, as a high-level trend report to demonstrate continuous improvement or retention rates of valuable staff that presumably are required of all departments.

When Sam Lamonica became CIO of the construction firm Rudolph & Sletten, the IT systems were a mess, with the department under tight scrutiny over basic operations. Lamonica diligently reported operational performance to the executives as he fixed the underlying systems and processes. When, after a few months, the executives saw that the operations were no longer troubled, they asked him to stop such reports, now confident that their new CIO knew what he was doing. The boardroom focus on IT changed to taking advantage of it strategically, which is what the focus should be. Lamonica, of course, continued to gather such metrics for internal IT use to monitor efficiency and quickly identify any incipient deviations.

Of the two remaining areas where executives should focus, the easier basket to construct metrics for is project management effectiveness, which should cover quality, scope, and milestones, said Frank Modruson, CIO of the consultancy Accenture. Recommended metrics include schedule adherence, budget adherence, functional delivery requirements specifications, and—the least often measured—return on investment for several years after deployment.

This last metric lets the business derive the real value of the investment, not just gauge the specific project management efficiency as the others do. After all, effectively implementing a project that doesn’t return the projected value means the business didn’t gain what it expected, and that knowledge is needed to improve the up-front ROI analysis for future projects. Yet most companies—at least 85 percent—don’t bother assessing their projected ROI after deployment. So they have no idea whether their efforts were truly effective in that they served actual business needs, said Holland of the Hackett Group. Instead, they just move on to the next project or jump immediately into making enhancements, Chang said.

CIOs should measure a project for at least several years after deployment to see if it continues to deliver as promised over its full life cycle, recommended PricewaterhouseCoopers’ Lutchen. After that period, the technology usually becomes part of the operational fabric, and it becomes hard to discern its specific value apart from those of other systems in use. But CIOs should do more than just look at individual projects’ efficiency and effectiveness; they should do trend analysis to see if either changes over time to identify improvements, flaws, or opportunities. For example, such analysis may reveal that project management efficiency remains high, but that the effective value is inconsistent, indicating that the justification process is flawed.

The third area—strategic value—is the hardest to measure because many factors beyond IT are involved, such as the degree of a business unit’s effective use of the technology, market changes that may affect initial value assumptions, or actions by competitors that neutralize any initial advantage. “After all, there is no such thing as an IT project; it’s a business project in which IT is important,” said Lutchen.

That doesn’t mean the CIO should ignore this issue or point fingers. Instead, the CIO can and should take the project value trend analysis from the second basket and use that as part of the IT’s strategic value metrics. These metrics require historical data such as trend information from which legitimate projections attuned to projected goals can be created, said Accenture’s Modruson. “It’s about comparing what you want to do versus where you are,” he explained.

That also means understanding that both IT initiatives and the business itself go through cycles, assessing the budget, operations, and ROI trends accordingly, said Achievo’s Mathaisel.

Treating IT as a portfolio of services is an effective way to track strategic value, said PA’s Houghton, especially if the executives reassess every few years the strategic goals and how the portfolio supports them.

And IT should be part of the overall portfolio that is measured and assessed, noted Chang. “You need common measures for the C-suite, so they’re all in the boat together.”

DETAILS GET IN THE WAY
It’s tempting to collect all sorts of data to demonstrate your value or success. But you should collect only what really matters and then deliver appropriate subsets or high-level rollups as appropriate to the recipient’s needs, recommended Anthony Perrone, a consultant at J.C. Jones & Associates. For example, a network administrator will want to see network traffic rates over time, while the CIO may care only to see when basic thresholds were not met, and the executive team needs to see all such operational data reduced to one number of overall IT operational efficiency compared to past performance.

A good rule of thumb is that there should be less than a half-dozen key metrics provided to executives, said Hackett Group’s Holland. If they need the details, provide the drill-down capabilities, but don’t make them part of the standard report.

GETTING THE RIGHT FINANCIAL RESOURCES
Most IT departments don’t have the financial skills to create effective project management and value measurements. Larger organizations’ IT groups should bring in such financial experts, having in effect a CFO or controller of IT, recommended PricewaterhouseCoopers’ Lutchen. CIOs at smaller companies should partner with the CFO to create such metrics together. This has the added benefit of aligning budget values with IT values and can help a bean-counter CFO see beyond the immediate numbers just as it can help an operationally inclined CIO see beyond the speeds and feeds.

The result should be metrics that aren’t just about raw cost, according to Holland. “It’s important to understand the makeup of that cost,” he said. A good way to do that is to separate the status-quo operational costs from the costs of adding value through new initiatives (including the added operational costs they will incur). Also, it’s best to avoid cost-metrics-based spending per employee or per dollar of revenue—these say nothing about what you actually need. For budget trending, a better measurement—if used with other, more strategically oriented metrics—is IT cost per end user, Holland said. To illustrate the fallacy of looking only at the bottom-line cost, Hackett’s research found that world-class IT organizations now spend $9,024 per end user, 7 percent more than the $8,485 spent by typical companies.

The bottom line for determining the IT metrics that matter for the business is thinking in a business context, said Houghton: “At the end of the day, you need to think in terms of that business value.” And then define and present a manageable set of metrics aligned to that value.

Sources

Richard Chang, CEO, Richard Chang Associates; contact via Jill Hennigan, jhennigan@rca4results.com, (949) 727-7477 ext. 265.

Scott Holland, senior business advisor, The Hackett Group; contact via Gary Baker, gbaker@thehackettgroup.com, (718) 523-6353.

Alan Houghton, principal consultant, PA Consulting Group; contact via Devon Jaffier, devon.jaffier@paconsulting.com, (212) 973-5939.

Sam Lamonica, CIO, Rudolph & Sletten Construction, sam_lamonica@rsconst.com, (650) 216-3701.

Mark Lutchen, partner, PricewaterhouseCoopers; contact via Cecile Rakover, crakover@lakpr.com, (212) 329-1427.

Bernard “Bud” Mathaisel, CIO, Achievo, bmathaisel@yahoo.com, (925) 876-4190.

Frank Modruson, CIO, Accenture, frank.b.modruson@accenture.com or contact via Ed Trepasso, ed.trapasso@accenture.com, (917) 452-3555.

Anthony Perrone, consultant, J.C. Jones & Associates; contact via Rich Kiley, rkiley@robertscomm.com, (585) 246-0260.

Galen Gruman frequently writes about enterprise software strategies for InfoWorld, CIO, and other publications.